RF szn2 and h1 nerf

Hi everyone.
This topic have a place to be.
Let’s put all of the suggestions and complaints about recent h1 nerf with h2 pump aka bug fix.

3 Likes

I have done some analysis comparing all of the portal options across Haunt 1 and Haunt 2.

The latest chart and numbers are visible in the link below.
https://aavegotchistats.com/portalrarity

Haunt 2 has lower minimums and higher maximums and appears flatter than haunt 1, initially this concerned me, but after calculating the variance and standard deviation values, haunt 1 actually has a slightly higher variance and a slightly higher standard deviation.

I don’t think we have enough data at this stage to say whether there is a statistical balance issue between haunt 1 and haunt 2 yet, those very high haunt 2 portals could just be outliers.

Although I am still sceptical of this just being a case of outliers after seeing there is different code for interpreting the random number from Chainlink VRF for calculating traits for haunt 1 gotchis vs haunt 2 gotchis.

I need more data to say conclusively, at this stage I will continue to monitor the data as more haunt 2 portals are opened.

2 Likes

So, we’ve seen that

  • The code change from H1 to H2 does not affect BRS
  • The collateral change gives a SLIGHT advantage to H1
  • The results of looking at all 10 aavegotchis in all the opened portals, show no statistical bias in average BRS nor in BRS spread.

H1 is already advantaged in XP and Kinship, there is no “nerf”, and no action needs to be taken.

MAYBE, if someone really wants to push this, we could find out whether player selection of non-top BRS in Haaunt 1 is driving the observed skew in high-end gotchis. I know that when H1 dropped, I had a list of names I wanted to grab as quickly as possible, so I was opening and summoning quickly. I may have overlooked a BRS gem… maybe?

Is there a way someone can crunch the data to show what % of the time the top BRS gotchi was summoned from an open portal?

3 Likes

Thanks to discord user eitri, we have a very thorough analysis on the code and its implications. I am pasting his entire statement below.

I’m very new here, but as a coder I got drawn into this as an interesting puzzle. My day job is Javascript, so I’m not a Solidity expert and I could easily have made some wrong assumptions or mistakes in logic, but if I have then hopefully someone can point them out and we can learn more as a result. I know that the actual Aavegotchi team are certain to be looking at this in future when they have time, but maybe this will help others in the community who want to do their own investigations, given it’s such an active discussion topic right now.

I may regret jumping in here as I see there are strong feelings on many topics - I’m sure all with very good reasons. I’ve only been looking at the code - and I’ve spent enough time on it I might as well post it I guess :wink: Apologies for length, I’m trying to be concise, honest.

(BTW - hi! Aavegotchi seems so cool :))

Curiously, other than the well-known low/high H1 trait split, the only other skew I could find in the code was a very slight one towards higher BRS in H1.

Here’s what I found:

  1. The baseRarityScore function (https://github.com/aavegotchi/aavegotchi-contracts/blob/master/contracts/Aavegotchi/libraries/LibAavegotchi.sol#L281) calculates the BRS by simply adding up each trait’s difference from 50. This is where the final ‘normal’ distribution shape comes from: a sum of random numbers. Therefore, the underlying trait number-generation is the place to look.

  2. The toNumericTraits function (https://github.com/aavegotchi/aavegotchi-contracts/blob/master/contracts/Aavegotchi/libraries/LibAavegotchi.sol#L65) generates the trait numbers (in range 0-99) from an original Chainlink large random number. It does this differently depending on the haunt.

  3. I found only two code differences between haunts in toNumericTraits: A) the code change that has been focused on so far, from “value /= 2” to “value = value - 100”, and B) ‘_modifiers’ which is the collateral/spirit force modifier, e.g. ETH = AGG+1, USDT = AGG-1: different haunts have different available collaterals and so a different set of modifiers.

Each trait is meant to be a random number between 0-99. I.e. each possible number is equally likely, and if you plot them all that would be a uniform distribution (a flat line). The H1 code bug (A) meant that actually, there was a ~30% chance of numbers between 0-49 and ~70% chance 50-99, which was confirmed nicely in reality by a graph of trait values that someone helpfully posted earlier (the one shaped like a step).

Looking at the toNumericTraits H1 code, this confirms the 30/70 distribution: assuming I’m understanding the code correctly, this is what it does:

  • slice off a piece of the original chainlink random number to get a number between 0-255.
  • if that value is from 0-99, then we use that as the trait value.
  • if that value is from 100-199, then we divide it by two (solidity rounds down the result) to get a value between 50-99, and use that value. (This is the bug that was fixed)
  • if that value is from 200-255, then we fall back to another way of deriving a new random number from the chainlink one, to get a number between 0-99, and use that value. (I’ll assume that this approach results in a nice uniform random number - the details don’t actually matter as the same approach is used for all haunts.)
  • finally, apply any spirit force modifier to the trait value (+1 or -1 for a single non-eye trait per gotchi)

So random numbers between 0-99 and 200-255 generate trait values between 0-99, but 100-199 generates trait values between 50-99. Of the 156 correctly generated, 78 result in low traits and 78 in high traits, but of the 100 incorrectly generated all result in high traits. That’s 78 low and 178 high, out of 256 total: 78/256 = 30.46% which matches the graph posted of observed traits.

The fixed code changes only this:

  • if that value is from 100-199, then we subtract 100 to get a value between 0-99, and use that value.

This results in a nice uniform (flat) distribution, so that all trait values between 0-99 are equally likely.

The interesting thing about the above, is that the bug does not itself affect BRS of the population overall. It changes which individual gotchis end up with what traits when the algorithm runs, but on a large scale the total numbers/distribution of traits is the same. To see this, consider the affected range: a random number in the range 100-199.

  • In Haunt 1, this number is divided by 2 (rounded down) to get the trait value. 100 → 50, 101 → 50, 102 → 51, … 197 → 98, 198 → 99, 199 → 99.
  • In Haunt 2, we subtract 100 to get the trait value. 100 → 0, 101 → 1, 102 → 2, … 197 → 97, 198 → 98, 199 → 99.

Then, looking at the lowest and highest 1% i.e. trait values 0 and 99:

  • Haunt 1 has two original random numbers that produce it: 198 → 99 and 199 → 99.
  • Haunt 2 has two original random numbers that produce it: 100 → 0, 199 → 99.

Both haunts get the same number of rare trait values, they’re just given to different gotchis in each case.

As far as I can tell, the changed code cannot have changed the resulting BRS distribution.

The last piece is the spirit force modifier, which actually can cause a difference in BRS between haunts, though I would think it would be too small to make a significant difference in practice. Nevertheless for completeness:

Notice that the collateral modifier effects are not balanced: in both haunts, there are more collaterals with a +1 effect than -1.

Therefore on average, in Haunt 1 there’s a +0.111 average effect on a single trait score from collateral, and in Haunt 2 it’s stronger, at +0.428.

In Haunt 2, trait scores are uniformly distributed between 0-99, i.e. there are an equal number of gotchis with high traits vs low traits, so half will benefit from the collateral modifier (+0.428) on average and half will lose - net result, no effect on the average BRS. (There would also be another visible effect: effectively the trait distribution is being slightly shifted to the right, but the boundaries that we call ‘rare’, ‘mythical’ etc are not. So e.g. this would reduce numbers of low mythical non-eye traits and equally increase high mythical: I don’t know how large a sample size you’d need to be able to see this in practice though.)

In Haunt 1 however, the trait scores are not uniformly distributed due to the bug, i.e. out of (say) 100 gotchis, 30 have a low trait score and 70 have high. So if there’s a small positive collateral modifier (+0.111) on average, that will be bad for only 30 gotchis’s BRS and good for 70 gotchies, resulting in a net increase in the average BRS. By a very very tiny amount!

TLDR: I looked through the code to figure out if it could cause the problems we’ve been seeing in the stats between Haunt 1 and Haunt 2. I saw how it produced the H1 low/high trait distribution, that matched observed traits. I couldn’t find any code that would affect the BRS, except for minor effects from collaterals. The only BRS bias I could find in the H1 code was actually towards a slight (probably unnoticeable) positive BRS. I hope some of you find these workings useful, either to help you inspect the code yourself or find errors in my assumptions/understanding/logic :slight_smile:

Further work that would be helpful but I do not have the skills for: a statistician (or research/data scientist) could look at the on-chain gotchis generated by all opened portals and see if the observed traits mathematically match the expected distribution with reasonable certainty e.g. with a t-test. I know that getting statistics right is hard, so I’m not going to try that myself.

2 Likes

I’ve also done some work to simulate rarities and trait distributions using the function that was used to generate numeric traits. This data is taken from 100,000 simulations of each haunt.

Haunt 1:

Haunt 2:

https://github.com/pakim249CAL/gotchi-rarity-test

1 Like

HAHA.

no nerf, no need to do anything, this dude has spoken.

The denial will continue from H2 owners, and the suffering for H1 buyers won’t be any less.

Nothing changes the fact this implementation was flawed enough to not be able to ever bunch H1 and H2 in an equal and fair competition. You simply can’t.

Continuing to argue in favor of that, especially from those who rallied for rookie boards, just makes me bearish on the community.

Everyone out for their own interests.

I’m making available the full dataset for all gotchis that were available in all portals from haunt 1 and haunt 2 which anyone else can build off of and analyze.

Data: File on MEGA

Source code:

https://github.com/pakim249CAL/portal-analytics

Special thanks to @jarrod. I used his queries from aavegotchistats.com to build mine.

image

different curves for a competition based on curves but sure, there is no nerf and no action needs to be taken.

Please stop trying to shut down a legitimate issue.

1 Like

Look at the shape of those curves. Don’t make me get out my protractor now guys. I haven’t spoken much but the denial is incredible.

2 Likes

The shapes, or the gap between white and blue curves? I think the gap is caused by fewer H2 portals being opened so far, no? The white (should) rise to meet the blue when they have an equal number of open portals.

Or are you looking at the tails somehow? I can’t see enough detail to judge anything in the tails by eye from that picture.

Edit: There are two separate issues:

  1. a potential mathematical problem that led to an inherently stronger H2. i.e. it’s easier to summon higher brs from H2 portals.

  2. the effects of dilution on previous haunts when a new one is released. How do we keep competitive gotchis competitive?

Both are important issues. The first can be proven to exist, if it does, mathematically. Identifying the problem would make the solution a lot clearer, if not obvious.

The second is a longterm strategic and philosophical discussion that needs to be solved regardless of any formula-based accidental H1 nerf.

2 Likes

I genuinely don’t see what you are talking about. The center is about the same. There may be slightly longer tails on H2 (which you can’t tell from that chart), which no one has been able to prove to be anything other than luck or selection bias.

There is no reason, based on the code we can all see, that the BRS is significantly biased between the haaunts (except the minor effect of collaterals, favoring H1). Showing an inherent bias is a necessary criterion for making any kind of correction/compensation. Otherwise you’re complaining about player behavior and/or RNGs.

1 Like

With less portals opened, H2 has 9/10 of top BRS, and is also over-represented in the top 50 and top 100.

I am not worried as time has continued to be on my side with highlighting just how OP H2 is when compared to H1 at producing highest-range BRS (what the RF paradigm rewards most).

Maybe when the top 45 or 50 BRS spots are exclusively H2… Maybe after a few haunts there’s not a single H1 gotchi in the top 100. I don’t know when or how the denial ends for some of you but I can continue waiting.