Unstable replays at ZakStunts

[Daniel3D]

Go to frame 1:46:00, press the 0.05 forward button until frame 1:48:00, and press play. The car finishes in 2:20:00 without penalty time.

This is a crash for me at 1:47:95. 

Can you provide a GIF?

PG can differ for everyone....
It's the butterfly effect..

[Argammon]

I made a GIF for @Friker. Here it is: (Ending 3)

EdIt: It is quite slow so you have to wait a bit before it starts. Sorry!

You cannot view this attachment.

[Daniel3D]

Can you do it for the other ending as well??

[Overdrijf]

Ah, that's what you meant by making a replay with three different endings. You replay handled to get the alternatie ending, played to where the regular run had finished to preserve that ending, and then drove the car to the finish.

By the way, I don't think I've ever seen such a good impression of the Blues Brothers claiming a parking spot as Friker's accidental driving there.

[Argammon]

Ok, here is more research. Please note that I am not doing this to prove Duplode's method is bad, but to gain a better understanding of how the engine works. 

I have created a replay that shows a crash both with the play button and Duplode's method. However, I actually finished the lap, please see the gif below. The replay is also attached.
If I can do this on purpose, it can also happen by accident when using replay handling.

You cannot view this attachment.

You cannot view this attachment.

Edit: Another relevant question is whether there are replays that show as a crash with Duplode's method but play correctly when using the play button. I did not test it but think that is the case.

[Friker]

If I can do this on purpose, it can also happen by accident when using replay handling

What knowledge do you have when you can do this on purpose? This actually reminded me of Renato Biker.. He produced magic carpets seemingly out of nowhere..

I probably messed up something - for me fast-forwarding is producing the same result as playing.. :-/ I somehow assumed that (fast-)rewind and fast-forward are the same things.. Can someone confirm that they compute car state in different manner (something like - rewind recomputes car state from the beginning without graphics included, fast-forward computes "next frame" - same as with Play button)?

Despite recent findings I still think that only suitable validation methods (accessible and quick) are 1. load a replay, rewind to the beginning, press play 2. load a replay, rewind 1.05s, see if car crosses the finish line next frame (probably continue ride with input from replay and see if the evaluation screen shows up is more precise?) 3. repldump. If you produce a replay in which you have to rewind some, play some, then continue to finish then bear the consequence that other cannot validate it.

Also - to stay on-topic - is there more testing to do regarding choosing validation process for ZakStunts? I feel that we (especially Argammon) showed that we can screw the game engine in multiple ways and there is no true interpretation of what is the intended way (for me personally - I would disregard any graphics and use repldump/rewind functionality only but I understand that graphical presentation is also a part of the game). So - can we finalize some options what validation process to choose or do you think that we can come up with more insights on this topic with more testing (what specifically do we want to achieve?)?

[Daniel3D]

I probably messed up something - for me fast-forwarding is producing the same result as playing.. :-/

Playing with this bug is very dependable on the setup used. So, don't be dismayed if you can't reproduce the anomaly...

[Argammon]

I probably messed up something - for me fast-forwarding is producing the same result as playing.. :-/

Playing with this bug is very dependable on the setup used. So, don't be dismayed if you can't reproduce the anomaly...

Maybe or maybe not. @Friker: If you post a gif of what you did, I can tell if you made a "mistake". I also agree that 1) together with 2) are currently our best bet. That way, we would be able to verify most but not all replays.

At the risk of stating the obvious, let me elaborate a bit on what I found (using my system). After frame 1.46.00, the car in Friker's replay is following 3 different paths simultaneously. By using the replay controls, you can access and drive each path separately. For example:

- If you finish path 1 (the one that currently crashes into the bridge), the car reaches the finishing line when you click the replay button, but Duplode's method fails.
- If you finish path 2, the replay button crashes the car, but Duplode's method works.
- If you finish path 3, both the replay button and Duplode's method are not suitable to verify the replay. 
- To fix the replay, you have to start driving a bit before 1:46:00 and the anomaly will likely vanish.


I hope that made sense! 

[Friker]

I probably messed up something - for me fast-forwarding is producing the same result as playing.. :-/

Playing with this bug is very dependable on the setup used. So, don't be dismayed if you can't reproduce the anomaly...

Maybe or maybe not. @Friker: If you post a gif of what you did, I can tell if you made a "mistake".

Sorry, I do not know how to easily create a gif so you have captured video from dosbox as an attachement.
I am also attaching some other of my crashed replays for others who might be interested.

[Overdrijf]

- To fix the replay, you have to start driving a bit before 1:46:00 and the anomaly will likely vanish.

The tricky thing in this situation is that at this point Friker is doing some very precise sliding to end up on the road rather than in the water. Starting a bit before 1:46 he'd be likely to replay handle himself into almost the same set of key presses and almost the same position. So... could that make the same anomaly appear?

[Duplode]

Two quick comments on some of the things @Argammon has noticed:

• Even in the presence of "butterfly effect" finishes with the timelines giving different lap times, we can carry on verifying lap times like we've always done, through the automated replay length and completion checks in the site. That makes sense, in particular, because the twenty 00 bytes at the end of a complete replay file provide evidence that the lap time reported by the site was the one seen by the pipsqueak before saving the replay.
• While it's indeed possible to intentionally finish unstable replays in such a way that they will appear incomplete when loaded from the Options menu, my working hypothesis is that in practice such a thing will happen far less often than unstable replays which show as complete. I am currently reviewing examples of unstable replays so that, among other things, I can present some extra evidence on that.

[Daniel3D]

If the game loads the end state of a replay consistent when loading from the options menu, regardless of who checks it. The it is good practice to check before submitting if that happens. Those kind of situations are rare and because they are easy to verify for the player.

[Duplode]

I'd like to share a few more interesting results on unstable replays, from investigations that were being prepared over the last few days.

Firstly, @Erik Barros (thanks a lot!) has kindly ran tests on some of the replays using his bona fide Pentium 266, running Windows 98:

• Alan's deleted ZCT260 lap: loads properly from Options, crashes under the u/d cork when played from the beginning, as described here.
• Frieshansen's 1:17.50 at ZCT251 from upthread: loads properly from Options, crashes when played from the beginning.
• Frieshansen's 1:17.45 from upthread: loads properly from Options, veers off-track by the final u/d cork when played from the beginning, which is a different divergence from what had been described here! Much like in Argammon's experiments, there's also a third possible timeline, reached by rewinding to a specific spot, in which the car flies high in the sky instead of completing the lap.

These results confirm that instability is not just an emulator issue, but also affects the game on period-appropriate hardware. Additionally, the novel timelines with Frieshansen's 1:17.45 further underline that it's anybody's guess whether and how instability will be observed on any specific system.

Secondly, I have reviewed GTO, Vette and Indy winning replays from the Competition Archive in search of further examples of unstable replays, and found 13 of them. The replays themselves and a spreadsheet listing them are attached below. With the caveat that this is what I've seen on my system, and your mileage may vary, here are some notes:

• The breadth of examples makes it clear unstable replays are not a new phenomenon, nor it is restricted to specific pipsqueaks. Whenever there are lots of powerslides, there is a nonzero chance of instability -- from a driver's perspective, that's all there is to it.
• While thirteen replays might not look like a lot (we have north of 200 plausible candidates in the Archive), keep in mind these only include winning replays, and there is no reason for instability not to happen further down the scoreboard. It's not surprising that in the last 36 months (ZCT225 to ZCT260) we've had three instability incidents, out of fourteen races with a PG car on the podium and seven full PG podiums.
• Almost all of those thirteen replays load properly from the Options menu, and diverge when played from the beginning. The sole, glorious exception is Gutix's ACT13 replay, which does the opposite. While this observation supports my hypothesis that most unstable replays in competition will pass my proposed validation, I find @Friker 's option #2 from this post (keeping playing-from-the-beginning verification as a fail-safe for replays which don't pass it) a very sensible one.
• The DOSBox cycles limit below which laps have the same outcome when played from the beginning as they do upon loading from Options changes per replay. While a large share of the thirteen replays here do not diverge at ~2200 cycles, that is not low enough for e.g. Alain's famous Eger replay, whose threshold here was ~1500.

I plan to do further analysis on these replays, as well as on the other ones shared on this thread, including running repldump on them all -- the working hypothesis being that repldump output always matches the load-from-Options outcome.

[dreadnaut]

This investigation is going quite deep, nice!

I suspected replays would crash on original hardware, and completing successfully on low-cycles DOSBox might be the simulation artefact here.

I have one more question: can we generate an unstable replay without using the "forward" button of the replay controls?

My hypothesis is that the bug (or butterfly) hides in the non-graphic play loop. If that's the case, rewinding a replay would not cause any issues, but advancing during a powergear stretch could make the car behave differently.

[KyLiE]

There has been a lot of discussion on this topic, which has been quite interesting to read.  Personally, I don't have an issue with the current rule.  I always ensure that my replays play back correctly when loaded from the options menu before posting them, regardless if I used power gear or not.  We've experienced unstable replays at Race For Kicks as well and have a similar rule for replay playability:

All replays must be playable continuously from start to finish when loaded in Stunts by selecting "Load replay" from the options menu. Playback at either standard or double speed is acceptable provided switching between the two speeds is not required for the replay to play back successfully.

Specifically, Duplode's replay for Joe's track in the permanent competition will play back successfully at double speed, but not at standard speed.  We also added an icon to easily identify replays like this.

I can also perform testing on legacy hardware, if necessary.  I have a 486 and a Pentium III system that run DOS natively, so let me know if running tests on either of these would be of any help.