Unstable replays at ZakStunts

[Argammon]

This is the end of the third Friker.

You cannot view this attachment.

[Daniel3D]

I need to write a Stunts based alternative universes story.

A time travel story?
Why skid always wins?
In the story style of wreck it Ralph 

[Overdrijf]

This is the end of the third Friker.

You cannot view this attachment.

A day at the beach, not bad.

But to save this topic from my own derailing attempt: great detective work! So it looks like the offending frame is not itself a near crash, rather just a powergear and steering frame? So it looks like Friker isn't doing anything we might want to ban or even anything we can point at to say "we should all not do that, and our replays will be fine".

[Friker]

Hello with my daily update.. So.. I went down the line of testing with PCem as it is marketed as the most faithful simulator of late 80s/early 90s PC systems. I tried Windows 95 with Pentium 133, 100, 75, MS-DOS 6.22 with 486 50MHz, 25MHz, 16MHz. For starters - I REMEMBERED CORRECTLY THAT GAMEPLAY IS MUCH SMOOTHER THAN ON DOSBOX.. (sorry for shouting I am just excited about that. ) Even on 16 MHz it felt that inputs are simply crispier. There is Win95 with internet support so if I could set up that connection maybe I will switch from DOSBox to PCem.

Real deal - my replay crashed on all systems - even on 16MHz. So, what to say.. I spent 4 hours to install Win95, MS-DOS, test my replay and find out that MS-DOS at 16MHz is still quicker/doing something differently than DOSBox with 4000 cycles.

I still would prefer that we would agree on either 1. only fast-forwarding validation 2. both "Play button" validation and fast-forwarding validation is ok as far as there are independent confirmations on questionable/unstable replays. Option 3. only "Play button" validation is still somehow distant for my understanding of how game works (that means - I am still assuming that difference between Play and FF happens because of fast computation of something - most probably something during drawing routine).

Btw - I love that story about myself riding alongside the beach (hopefully not straight into the sea) in a fast car.

Btw2 - For some reason - on MS-DOS - Stunts produced invalid replay - marking 20th byte from the end of the replay as 01 and not 00. Now the bigger surprise for me - Stunts showed correct time in the evaluation screen. Maybe it is not surprising.. It shows that Stunts is checking car position not replay length.

@ all testing done - thank you guys. Really, thank you. I think it only shows that - if we are using Play button validation - questionable replays should be verified on different systems/choose one independent system on which we will verify and which is available for verification before the end of a race to all of us.
If we are using fast-forwarding everything replays the same on all systems and there is no difference between systems (at least PCem with various hardware simulation and DOSBox).

[Duplode]

@dreadnaut On your closing question, you can probably guess my answer: the rule should go, because it adds busywork, inconvenience and frustration for essentially no benefit. But I've already said my piece, and there's no need to rehash it.

What I'd like to focus on now is how @Frieshansen 's experiments provide a new and very strong argument in support of changing the rule. Given that replay instability depends on the system/DOSBox configuration, we are bound to get situation in which a replay plays normally on the pipsqueak's machine but becomes unstable on the manager's. That being so, the obvious fair approach to this possibility is accepting unstable replays, and using a verification method which rules them as valid (or at least a large majority of them, if perfect accuracy turns out to be impracticable). I still believe the procedure described in my initial proposal is, all things considered, the best option we have at the moment, though that might change as our understanding of instability advances.

@Argammon , Frieshansen and @Friker : good work on those experiments, they are all very interesting and highly relevant. Also, thanks for the replays, Frieshansen -- I've began assembling a collection, and they will be in it  By the way, both of them play normally here (DOSBox Staging 0.80.1, Arch Linux), at both 20000 and (following Friker's lead) 2241 cycles.

[Erik Barros]

I don't know if it helps, but I can run tests on old hardware, a Pentium 266, with Windows 98.

[Duplode]

@Erik Barros It sure can help  From what we have seen so far, my guess is that instability will still exist on a system like that (actual DOS on a machine that's not too slow, in relative terms), though it will not necessarily show up in the same replays -- I do wonder about those Frieshansen laps!

[Erik Barros]

tomorrow I should be able to download the replays from @Frieshansen and check it, as well as the replay from @alanrotoi from the last race, with the dosbox on my machine the car broke down

[Overdrijf]

Real deal - my replay crashed on all systems - even on 16MHz. So, what to say.. I spent 4 hours to install Win95, MS-DOS, test my replay and find out that MS-DOS at 16MHz is still quicker/doing something differently than DOSBox with 4000 cycles.

Wait, 4000 cycles, or 40,000 cycles?

In that first case I'm not surprised the other option is smoother, but I am surprised about how well you drive on Dosbox.

[Friker]

Wait, 4000 cycles, or 40,000 cycles?

4000 cycles. For me, 4000 cycles is enough to Stunts go slower and skipping some frames (hopefully only from drawing). That is the difference between not crashing (below 4000) and crashing (above 4000) of unstable replays for me.

In that first case I'm not surprised the other option is smoother, but I am surprised about how well you drive on Dosbox.

What do you mean?

Btw - maybe it is not smoother but something definitely feels different. As I have no reflexes so I cannot tell if it is input lag or anti-aliasing/graphical interface or nothing and I am just imaging things.

[Daniel3D]

Wait, 4000 cycles, or 40,000 cycles?

4000 cycles. For me, 4000 cycles is enough to Stunts go slower and skipping some frames (hopefully only from drawing). That is the difference between not crashing (below 4000) and crashing (above 4000) of unstable replays for me.

Then I am curious. If you save a replay at 4000 cycles. Will it set 10 or 20 FPS.
And if you continue your unstable replay after the anomaly and save it again at 4000 cycles.
Will it change the replay from 20 to 10 FPS?
And fix it for normal cycles??

[Argammon]

Ok, I did another experiment, and it gets even worse. I managed to create a replay in which the car finishes with two different finishing times! The evaluation screen occurs in both cases. 

Here is how you find them:

1) Pressing "play" from the beginning causes a crash on the bridge.
2) Using the Duplode method (Load from Options, go to the end of the replay) shows the car finishes in 1.57.95 plus 9 seconds penalty time.
3) Go to frame 1:46:00, press the 0.05 forward button until frame 1:48:00, and press play. The car finishes in 2:20:00 without penalty time.

Comments:
- The methods described in 1-3 are sufficient but not necessary to produce the different endings. There are other ways to produce them, in particular Ending 3).
- Ending 3), which cannot be confirmed with Duplode's, method is slower than ending 2, which can be confirmed with his method. However, I can produce a replay in which Ending 3) is faster than Ending 2).
- Duplode's method may still be better than the status quo. In particular because, according to @Frieshansen's research, it can happen that a replay crashes on the manager's but not the pipsqueak's computer when using the play button. I find that unacceptable. I am hoping we are able to find a better method though.   

You cannot view this attachment.

[Overdrijf]

What do you mean?

I thought you said you wanted to switch to PCem because it runs the game better than Dosbox, but you were merely saying that a certain low setting on PCem runs better than a low setting on Dosbox.

[Friker]

Go to frame 1:46:00, press the 0.05 forward button until frame 1:48:00, and press play. The car finishes in 2:20:00 without penalty time.

This is a crash for me at 1:47:95. 

[Daniel3D]

The method that is most stable in it's outcome regardless of the setup used is in my opinion most fair. PG is unstable. That makes it a challenge in competition. We will have to accept that the replays can also be unstable.. (maybe they all are unstable but just passed this test)