In my previous thread (http://forum.stunts.hu/index.php?topic=3826.0) on pipsqueak performance modelling, I reported optimistically on modelling lap times over repeated attempts with a gamma distribution (https://en.wikipedia.org/wiki/Gamma_distribution). Back then, all I had for empirical evidence was a set of lap times driven in a minimal four-corners track. To better understand the model and the problem space, it would make sense to gather data on more realistic conditions. Soon enough, the perfect opportunity came up with Alpine, R4K's October race (http://www.raceforkicks.com/index.php?page=race&race=2021-10): great car, great (and not too difficult) track, and a real OWOOT NoRH race going on.
From October 11th to the deadline day, I had one (occasionally two) NoRH session on Alpine every day, and recorded all of my valid lap times. I tried my best to keep a consistent driving style, going for the most effective reproducible racing line I knew of, and not giving up on laps unless I crashed or left the track. The following set of box plots show what my lap times on each session looked like (for a closer look at the data, you can look at the attached CSV file):
(https://i.imgur.com/b6IbrM4.png)
I would divide my racing in three main stages:
- On sessions 1 to 5, I was still learning both car and track. In particular, I got under 83 seconds after figuring out a good line through the second and third corners, and under 82 seconds by understanding how the second fast chicane should be approached.
- From sessions 6 to 14, there was largely stability, with perhaps some very slow improvement, culminating with a 81.00 on session 14.
- On sessions 15 to 18, there was modest but marked improvement, which I attribute to either realising I could be a touch more aggressive with my lines upon rewatching the laps I had posted to R4K, or to being in better shape by no longer doing late night sessions. I drove my final R4K time of 80.55 early on session 15. After that, I only managed one more lap under 81 seconds: an 80.75 on session 18.
With the lap times at hand, the next step is attempting gamma fits on stretches in which I had consistent performance. A good place to start could be sessions 15-18, in which I was perhaps closer to my best. Here is a first attempt:
Fitting of the distribution ' gamma3 ' by maximum likelihood
Parameters :
estimate Std. Error
shape 5.5682015 1.79934095
scale 0.3520632 0.07724456
thres 80.2265133 0.24903711
Loglikelihood: -115.9133 AIC: 237.8267 BIC: 245.612
Correlation matrix:
shape scale thres
shape 1.0000000 -0.9492797 -0.9041436
scale -0.9492797 1.0000000 0.7526068
thres -0.9041436 0.7526068 1.0000000A recap on what the gamma parameters mean in our context:
- thres, short for threshold, is the predicted ideal time, the one the model assumes it is impossible to go below.
- shape indicates how hard it is to improve as you get closer to the ideal time. It is associated with track length and track/line difficulty.
- scale reflects how consistently the pipsqueak manages to drive. Smaller values make for a narrower gamma curve, squeezed closer to the ideal time.
While the diagnostic plots show the gamma fit is pretty good, the parameter values are all over the place, as the following bootstrap (https://en.wikipedia.org/wiki/Bootstrapping_(statistics)) confidence intervals indicate:
Nonparametric bootstrap medians and 95% percentile CI
Median 2.5% 97.5%
shape 5.2026795 1.965190 10.9580942
scale 0.3649339 0.222748 0.6145803
thres 80.2609610 79.615182 81.0369308
Ultimately, is is a bit much to try fitting those three parameters at once with the available amount of data; there is too much wiggle room. From the three parameters, the one we are in a better position to estimate by other means is
thres. The sum of the best sectors over the fifteen laps I had saved for posting to R4K is 80.05; that being so, 80.00 is a reasonable, if conservative, ideal lap estimate. Fixing
thres to 80.00 results in the following gamma fit over sessions 15-18:
Fitting of the distribution ' gamma3 ' by maximum likelihood
Parameters :
estimate Std. Error
shape 7.0919246 0.98618134
scale 0.3083916 0.04444141
Fixed parameters:
value
thres 80
Loglikelihood: -116.1671 AIC: 236.3341 BIC: 241.5244
Correlation matrix:
shape scale
shape 1.0000000 -0.9650934
scale -0.9650934 1.0000000 (https://i.imgur.com/B3K8LvE.png)
The 95% confidence intervals look a fair bit tamer now:
Nonparametric bootstrap medians and 95% percentile CI
Median 2.5% 97.5%
shape 7.2104854 5.4284800 10.1456150
scale 0.3034129 0.2069173 0.4084368While
thres is the easiest parameter to estimate through different means, the one it would be arguably more interesting to keep fixed is
shape, as it is supposed to be the one most closely associated to the nature of the track. Given the more believable estimate of
shape we have just obtained for sessions 15-18, it is worth trying to fits for other sets of sessions with
shape fixed at 7.1. Here is a fit for sessions 6-9...
Fitting of the distribution ' gamma3 ' by maximum likelihood
Parameters :
estimate Std. Error
scale 0.4312479 0.0320152
thres 80.1501134 0.1939291
Fixed parameters:
value
shape 7.1
Loglikelihood: -142.3025 AIC: 288.605 BIC: 293.6915
Correlation matrix:
scale thres
scale 1.0000000 -0.8532844
thres -0.8532844 1.0000000(https://i.imgur.com/GorkjIS.png)
Nonparametric bootstrap medians and 95% percentile CI
Median 2.5% 97.5%
scale 0.4276147 0.3703891 0.4924684
thres 80.1722529 79.8464586 80.5086957... and 10-14:
Fitting of the distribution ' gamma3 ' by maximum likelihood
Parameters :
estimate Std. Error
scale 0.3832479 0.02554952
thres 80.0632570 0.15585001
Fixed parameters:
value
shape 7.1
Loglikelihood: -171.2663 AIC: 346.5326 BIC: 352.1242
Correlation matrix:
scale thres
scale 1.0000000 -0.8591264
thres -0.8591264 1.0000000(https://i.imgur.com/RxxVe5S.png)
Nonparametric bootstrap medians and 95% percentile CI
Median 2.5% 97.5%
scale 0.3800133 0.3260546 0.4442169
thres 80.0765083 79.7532830 80.3593560Keeping the
shape fixed, the fits point to a noticeable reduction of the
scale parameter (suggesting more confident and consistent driving) across the sets of sessions. The difference in
thres (which might be ascribed to refinements of the driving line) is too small to be clearly meaningful.
Parameter estimation for my proposed model is challenging: pining down those three significantly correlated values without huge amounts of lap time data is nontrivial, and if it can be tricky to extrapolate from one day to the next, let alone doing so across different pipsqueaks or driving lines... On the flip side, the results of the experiment do suggest the gamma distribution is a reasonable model for lap times, and that it is worth it to keep pursuing this idea.
This is a very thorough experiment and your findings are very interesting. I appreciate you sharing them with us and also your participation in the race. It sounds as though your NoRH attempts were similar to mine. If you plan to run an experiment like this in the future and need additional data, let me know, I may be able to help.
Quote from: KyLiE on October 29, 2021, 03:34:09 PM
This is a very thorough experiment and your findings are very interesting. I appreciate you sharing them with us and also your participation in the race. It sounds as though your NoRH attempts were similar to mine. If you plan to run an experiment like this in the future and need additional data, let me know, I may be able to help.
Thanks! On future experiments: if you ever wake up feeling like recording all your NoRH laps for a race (or permanent competition challenge, etc.), I'll be all too happy to analyse the data :) (That goes for everyone else, too! ;))
I would be glad to contribute too. I normally don't post more than a few replays per month because when it starts to get difficult, there's a point when I know that most of my attempts will be worse than my best replay so far, but if this will help for a study, I'll definitely do it.
We have a couple of special events coming in and I'm not sure if they will be ideal for this, precisely because they are special, but if it helps, then count on me!
While I'd love to have extra datasets, do note this is probably only worth doing if the track is one that makes you feel at home (that's why Alpine was so great!); so if you feel the next few tracks aren't a good fit there is no need to rush or force your way through the ordeal. That said, if you are in the mood and the track fits, go for it! :D
If you are going to do it, I suggest racing with a fixed number of attempts (successful or not) per day. That makes the sessions easier to plan, as they will take roughly the same time, helps with comparing results from one day to the next, and removes quite a bit of the psychological pressure of chasing improvements.
Quote from: Cas on October 30, 2021, 02:54:50 AM
We have a couple of special events coming in [...]
Sounds interesting! ;)