Unstable replays at ZakStunts

[Cas]

If it weren't that different computers and set ups can result in different time-lines, I would say first load from Options and if that fails, hand to the pipsqueak the responsibility of providing instructions that work in replaying successfully, but this might not work for everyone. I'm therefore willing to support any educated decision the community makes here and if tied, I'll just pick a choice

While all this research helps much more to understand how things work than to prevent or fix unstable replays, I still think it's immensely valuable and I think we should continue adding to it all we have and I agree that it things get confusing, we can just split the thread. In fact, my opinion is this should go to a wiki article as well!  There are many points here Duplode makes that I hadn't considered at all.

Adding more to the research... I had not considered these "checkpoints", although I did know about how rewinding actually fast-forwards. It looks like Duplode's work can help us see ahead when a replay could result in instability and take special care during RH to ensure it won't fail. While Duplode took a very different angle to look at the problem from the one I did, both seem to be very compatible... especially about how rendering time seems to be the primary culprit here.

I will make some predictions about things we may observe in the future if things are the way they're looking to me:
- We'll find that replays with no PG can be unstable. The faster the speed, the more likely. PG replays are more likely to be unstable only because they are so fast
- We'll confirm that turning while at high speed increases the chance of instability and that a replay may be unstable because of fast speed, yet not noticeably unstable because the car isn't turning that much, so the result is practically the same
- We'll encounter more unstable replays in the future coming from low-cycle emulation than from high-cycle emulation, although all can be, because at high-cycle, the rendering time is shorter

My understanding about what causes this in the first place is that:
- Stunts uses an asynchronous loop, like when you have a ship you can move with your keys while enemies move around. They have to keep moving even if you don't. This results in a non-deterministic physics/world, but would compensate if the math were infinitely precise
- Stunts reads the input in a loop after rendering and thus, rendering time creates a time offset from the beginning of the loop to when the action turns into effect. Reading the input means from the keyboard while recording, or from the replay file while replaying.
- While fast-forwarding does not do rendering, there may exist some small actions before input reading in the loop, so while extremely unlikely, fast-forwarding can also produce a difference from system to system. This is completely theoretical and I don't expect we'll ever see it happening
- Increments (deltas) that are added at each iteration of the loop to calculate new positions and speeds have a limited precision. The adding of these increments is the core of the creation of instability. I suspect that Stunts tries to compensate the time offset when rendering by removing time from these increments, but because the precision has a limit, information is lost, causing an error that accumulates with time. On faster driving, increments are larger and so are errors. If vector directions vary slowly and smoothly, these errors may not be as noticeable, but they exist

My recommendations would be:
- When recording a replay, use fast emulation. If you're going to drive a fast car or use PG, make it even higher or maybe reduce the rendering details
- On high speed turns, while building a RH replay, take a look at what replaying looks like and how it compares to fulling fast forwarding to have a chance to fix problems before working more on an already unstable replay

[Duplode]

Expanding a bit on what I've said yesterday about option #4, it is a peculiar rule that kind of feels like an unstable equilibrium:

• On the one hand, if everyone is aware of the details of the rule and plays it safe, ensuring replays loads as complete from Options, it effectively reduces to #3. Then again, it might be a touch too optimistic to assume full awareness of the details, and personally I still lean towards being as lenient as we reasonable can in such cases.
• On the other hand, if there are replays that don't load as complete from Options, whether those replays will be ruled as valid or not will still depend on the specifics of the admin's computer/setup. It remains a big improvement over the status quo, as a much smaller amount of replays will be affected, and there's clarity about what's going on and which replays will be at risk. Nonetheless, there might be a case for taking that load from the admin's shoulders by relaxing it all the way to #5: accept any reasonable evidence of completion, asking the pipsqueak to provide it if necessary, or setting up a procedure to get extra eyes (and computers) looking for a successful timeline.

So, while at the moment I feel like #4 is a good enough compromise, #3 and #5 are defensible options too.


A couple brief reports on tests with alternative setups:

• I tested the game on 86box (a fork of the PCem emulator @Friker told us about, which I found easier to get working here) and FreeDOS, using a 100MHz 486 as my primary choice of emulated processor (the configuration file is attached for reference). The results are consistent with what we have seen so far: the fast-forward timelines remain the same, and switching to a slower processor can affect and/or remove divergent timelines. Unsurprisingly, the divergent timelines often are different from what get with DOSBox. For instance, Overdrijf's ZCT232 replay crashes at the side of a ramp at 34.15 on the F1 camera, which is what dreadnaut originally saw back in November 2020. Also, Alan's final ZCT244 replay, which doesn't play to end at all on DOSBox for me, successfully plays to the end on F2.
• On DOSBox, changing the emulated video device (and the corresponding graphics driver in Stunts' SETUP) also affects divergent timelines. For instance, with CGA, Hercules or Tandy graphics Overdrijf's ZCT232 replay crashes on the left bridge wall at 31.25 when played on F1, instead of completing the lap as on VGA. Similar things also happen on 86box.

@Cas I believe the only thing that might be reasonable to ask drivers to do is ensuring replays load as complete from Options (which would be mandatory with option #3, and would remove any risk of disqualification with #4). Requiring changes to DOSBox or Stunts settings makes the game less accessible, and is not guaranteed to help.

[Cas]

@Duplode, of course I wouldn't request pipsqueaks to have to go through so much struggle or make them responsible for so many safety measures. These are recommendations in case you're worried that your replay may be unstable and you feel you want to make sure you're not working on something that will later bring problems. Something for people to do when they want it, not as a request.

For instance, I imagine maybe some of us have DOSBox configured by default at low cycles and changing this is a very simple thing you have to only do once and might help, so why not?  Not that it is required.

[alanrotoi]

I bet for if the replay can be run with F1, F2, F3 or F4 view then accept it. 100%
Then we could talk about dosbox simulation and number of cycles too.

[Argammon]

I vote for proposal 4. Accept a replay if it loads correctly from options or plays correctly with any camera view using normal and/or double speed. For example, if the replay plays correctly only with the F2 camera using double speed that is ok.

Proposal 5 is what we should aim for in the future. However, I feel more research is necessary to find a way that the verification process is feasible, safe and cannot be exploited.

[Overdrijf]

My heart is still in an option 5 place, but considering the administrative arguments I think option 4 has a good case for it too. I would like as many replays as possible to be valid. I don't think anyone here can really purposefully and meaningfully abuse this effect to get faster times (the example given earlier seems to be a way to get back on track after a calculation error caused a crash, the actual intended timeline is the one in which you didn't crash, and you might not have crashed using a different PC or other settings) and even if they can, isn't that just a new frontier of replay handling skills?

If we have to choose one single way of verifying (I don't really see why, but if we have to) the revelations in this topic make me say that I prefer the loading/fast forwarding method over the replay method, since it seems to be the correctly calculated line, it's more likely to match what the player drove and it always seems to have a stable result, while which branch the regular play function takes can be unpredictable.

I can respect any choice made of course, and I will support @dreadnaut's right as the main competition administrator to draw whichever line he sees fit.

[Cas]

I think option 5 makes sense if the "steps" are simple, straightforward, and work for everyone, such as "run it at double speed" or "play it with F2 up to 1:35 and then with F4" as more complicated example, but not if verification depends on a video or only works for some. Problem is you may not be able to tell if a procedure works for all, even after you've tried and it worked for, say, 5 people so far. For this reason, I'm more inclined towards 4.

However, same as Overdrijf says, and like I mentioned earlier, I'll support what the community decides and I too think Dreadnaut has the primary vote here.

[Argammon]

I suggest that we (or better dreadnaut/duplode) make a decision soon. For all I know, ZCT262 allows for power-gear laps. I have no idea whether the power-gear laps are actually faster (including bonuses) than ordinary laps, but it is possible. So, it would be nice to get this settled soon.

I made sure my ZCT261 replay played to the end on my system but was a bit nerveous nonetheless because I was not 100% sure it would also do so on dreadnaut's. 

[Friker]

I am all for option 4 (maybe with a check of three independent persons).

Due to recent findings I would strongly prefer to add that a replay plays correctly from 0:00.00 or 0:00.05 is valid.

Option 5 seems to me pretty vague - I would be OK if it passes but only with the independent check.

[Friker]

Here goes the bigger post I promised on Wednesday.
...

Duplode, thank you for your insights. I would like to reply to it but I am quietly waiting for a rule change. Then we can continue with a discussion about these findings.

[dreadnaut]

Option #4 does seem an acceptable tradeoff between accepting as many replay as possible, and having a repeatable process to validate replays. I don't think there is a problem with "works on my machine": differences in hardware/cycles create (or collapse) the split between fast-forward vs divergent timelines.

Option #4 would accept all replays with a single type of 'timeline split'. A rejected replay would require part normal-speed play, part fast-forward, which would be more complex to identify and validate for all involved.

Rules would change from:

A replay must be a complete lap of the circuit, and should play correctly when loaded in Stunts, from "Options → Load replay". Please verify your replay before uploading it, as unplayable replays will be disqualified.

to:

A replay must be a complete lap of the circuit. Please validate your replay before uploading it, starting from "Options → Load replay". A replay is valid if it shows a non-crash frame when loaded, or if it plays from start to finish at normal speed. Invalid replays will be disqualified.

[alanrotoi]

A replay must be a complete lap of the circuit. Please validate your replay before uploading it, starting from "Options → Load replay". A replay is valid if it shows a non-crash frame when loaded, or if it plays from start to finish at normal speed. Invalid replays will be disqualified.

[/quote]

I'm sorry I can't tell what changed. For example, this new rule would change something in the previous results if it were implemented before?

[Daniel3D]

Yes, they all load not crashed if I'm not mistaken.
It would be good practice to rewind a bit to check if the finish line has been crossed.
But this is the most player friendly version of the rule as can be (in my opinion)

[alanrotoi]

Very good then!

[Duplode]

@-dreadnaut While I also find #4 an acceptable tradeoff, it's worth noting it might be slightly harsher than that. More specifically:

I don't think there is a problem with "works on my machine": differences in hardware/cycles create (or collapse) the split between fast-forward vs divergent timelines.

There are (at least) two different manifestations of "works on my machine":

• Replays which complete successfully on the fast-forward timeline and diverge on some machines but not on others. Frieshansen's withdrawn ZCT251 laps are an example. These will be accepted by #4.
• Replays which don't complete in fast-forward, and which diverges on different ways depending on the machine. Examples include Overdrijf's ZCT232 replay (plays okay for me, didn't for you) and Alan's final, 1:14.15, ZCT244 replay (played okay for you, doesn't for me). These still are at risk under #4.

The new wording of the rule looks fine 👍 Is it worth it to mention playing with multiple cameras, or would that be excessive detail for that page? (Edit: another plausible reason not to mention it is that, even if validation comes down to cameras, the pipsqueak will see the lap being complete on F1, and so changing cameras isn't actionable advice for them.)

@Friker Sure! Be it in this thread or in another, we'll carry on the investigation By the way, I still need to condense the information here fot the Wiki.