Balanced bonuses estimation

[Duplode]

Ever since ZakStunts started having multi-car races, the balanced bonuses question has been a matter of interest to pipsqueaks and track designers alike: which set of bonuses would give all cars even chances of winning? This post presents a what I believe to be a sensible, comprehensive way of answering that question. A list of bonuses estimated for most cars featured in ZakStunts can be found towards the end of the post. Before that, a few words about my methods and assumptions.


A couple pre-season rule reviews ago, I suggested figuring out balanced bonuses by comparing pairs of cars, sorting the races they were involved according to the ratios of the bonus multipliers (or the advantages, which is how I'm calling the differences of the multipliers' logarithms, and which are easier to work with as they are on a linear scale), and finding the point at which the winning car changes. While that idea is basically sound, there were a few obstacles when it came to applying it systematically:

• Figuring out which races are relevant for each pair of cars. (Two years ago, I picked those in which at least one car of the pair reached the top 6. However, that might discard quite a lot of useful information, specially for the last two seasons, with close bonuses and single-car podiums.)
• Reliably estimating the switch point between the cars. (Eyeballing only takes you so far, specially when there is significant overlap between the winning ranges of the cars, or with custom cars for which not many data points are available.)
• Distilling the pairwise estimates into a single, overall estimate of the bonus for each car. (With 33 cars used in ZakStunts since 2008, we might have up to 32 different pairwise estimates for each car!)

For #1, after playing a bit with the available data I have settled on picking matches (that is, comparisons between cars available for a specific race) in which at least one of the cars either reached 120% of the winning time or had at least 10% of the submitted replays. That seems to give a good balance between preserving information and excluding accidental results.

For #2, a simple and well-established solution is logistic regression: using the advantages and the record of wins and losses for a pair of cars, we can fit a model that estimates the winning probability for the given advantage. For instance, here is an updated version of the Carrera versus Lancia chart from my older post, with a fitted logistic curve:



The midpoint of the curve, the point at which the model predicts the cars have the same chance of winning, is at an advantage of +0.056, which would amount to a +5.4% relative bonus of the Carrera over the Lancia.

(Side note: while the logistic model has the great advantage of simplicity, there are other reasons to regard it as a reasonable functional form for our problem. In particular, the winning probability as a function of the advantage should in any case be a sigmoid, as it must go to either 100% or 0% as the bonuses grow apart from each other, and also must be symmetrical around the midpoint.)

Obstacle #3 was arguably the decisive one. Ideally, we'd want to use the full wealth of pairwise comparisons to reach an overall value for each car, but do so in a principled way. After trying a few ad hoc approaches with limited success, I learned there is a standard way of taking in the pairwise models all at once: the Bradley-Terry model. It is similar in a few ways to an Elo ranking but without evolution in time (so the abilities of the players are assumed to remain constant), and is commonly used to estimate the abilities of players or teams within a tournament. In our case, the cars are the "players", and each pairwise meeting in a race is a match. The bonus advantages, then, can be incorporated in the same way that, in more typical uses of Bradley-Terry, home advantage for sports teams is handled: as an added term to the car "ability" scores.

With those pieces in place, fitting a Bradley-Terry model to the ZakStunts data is straightforward, giving us abilities for each of the cars, which are displayed with error bars on the chart below:



Up to a scaling factor, these abilities correspond to the intrinsic advantages for each car, and so they can be easily converted to balanced bonuses. You'll note I have excluded a few of the ZakStunts cars:

• Acura, GTO, Indy and Vette, due to the notorious problem of powergear leading to too much variability from one track to the next. (For a rough idea of where they stand relative to the others, see the Magic Lamp experiment.)
• Xylocaine, which, besides also being a powergear car, was never competitive in its one ZakStunts season so far.
• Speedgate, whose driving technique evolved so much that it is questionable how representative its historical results are.

It might be prudent to take the estimates of cars which only had one or two seasons so far with a grain of salt, as the error bars in the chart above suggest. (Uncertainty is specially high for the 911 Turbo because the seasons in which it was competitive were at the height of the far-apart-bonuses era, which tends to make its matches less informative.)


With the i's dotted and the t's crossed, here are the balanced bonus estimates:



These percentages should not be taken as definitive: the methodology might be further refined, and there will always be more races to add, with different scenarios to try out the cars. In any case, they are an auspicious start! 


One direction that might be explored in future work is adding more predictors to the model. The Bradley-Terry framework allows bringing in extra factors that might be additional sources of advantage or disadvantage for the cars. We might collect data on track-specific (e.g. the proportion of non-asphalt elements) or even replay-specific (e.g. how much of a lap is spent accelerating) variables, incorporate them to the model and see if they interact with the car abilities. That might be one way to quantify situational advantages of, for instance, off-road cars, or even powergear cars.

Both the data and the (very unpolished!) R code used to fit the model and obtain the bonuses are attached, in case you want to reproduce the calculations or tinker with them.

[Overdrijf]

Wow, very interesting data. Not all of these results immediately make sense in my head, but the methodology seems very thorough. And it's kind of poetic that the "basically developed to be my ideal car" BMW ends up so close to my favorite original car in the Jaguar. I might go and suggest an NTT race between the two.

[Argammon]

Great Work!

I think it would be cool to make all cars start with their balanced bonuses in January 2025. The scoreboard would surely look interesting.