Restunts repository - Git mirror

[dreadnaut]

Thanks for pointing that out, link fixed, and here again: https://github.com/4d-stunts/restunts/

[dstien]

ASMORIG_OBJFILES = $(ASM_OBJDIR)\segments.obj $(ASMORIG_OBJDIR)\seg000.obj
should be ASMORIG of course.

Not that it matters, but this was intentional. segments.asm was handwritten and there was only supposed to be one copy shared by asm and asmorig.

dstien isn't the initial creator/svn maintainer of restunts - clvn is - so he never controlled the source

Yeah, I know. But dstien is traceable to an extent outside this community. I know nothing about clvn. So impossible to tell for me if he maintains a copy..

Sadly his batman.no domain has lapsed, but since anders-e.com is still up I'm assuming he's well, just busy with middle-aged life. clvn was a bit of a celebrity in the Norwegian demo scene in the mid-90's. I was surprised to hear him namedropped in a Norwegian podcast about Amiga music a few years back.

So dereadnaut, just replace

A clone of dstien's restunts SVN repository.

with

A clone of clvn's restunts SVN repository.

dstien got no public SVN repo

Indeed, clvn started and ran this project. He figured out all the bit-trickery that allowed us to rebuild working executables injected with our own code. He did the majority of the analysis and porting to C. I mostly ported low-hanging fruit and cheered on the sideline.

I appreciate the efforts to make the code public, and I have thought about doing it myself from time to time. But I also wanted to do things a bit different, modernise the toolchain, cleaning up cruft, ensuring ported code is both portable and correct, and getting rid of the blobs. Particularly the database from the program whose vendor has a history of refusing to sell to people outside the infosec clique. Seeing the spectacular work of @HerrNove on SuperSight inspired me to have another crack at it. I decided to start with a clean slate to avoid said blobs and avoid offending anyone when deleting their code. Whether the new repo should be placed under the 4d-stunts org or be merged with the mainline repo is not for me to decide, it's still an experiment.

The first order of business was to convert the ID* Pr* database to Ghidra. After discovering that Ghidra provide an XML exporter script for ID*, I got my hopes up. Checking the output of the script confirmed that everything we needed were included, we could just use Ghidra's XML import and this project would be done in an evening or two.

Turned out it would take over a month before I had a working restunts.exe built from Ghidra. First the export script ignored symbols it deemed automatically generated based on their prefix. When we used names like "arg_cheeseburger" it would be dropped. It's just a Python script, so it was a quick fix. Next it turned out that our "arg_cheeseburger" wouldn't be imported at all anyway, as Ghidra's importer ignores function stack frames entirely. I think the reasoning is that Ghidra prefers to trust its own analysis to build the stack frame layout, which it does with its powerful decompiler that can infer arguments and their sizes. Our problem with that, besides losing argument and local variable names, is that Ghidra's support for working with segmented memory appears to be an afterthought. While ID* was first released in 1991 when 16-bit was still dominant, and its segmented memory handling remains a first-class citizen. There are many related issues filed for Ghidra about this, and many remains unactioned for years. There seem to be a, quite natural, distance between the NSA's and the retro community's priorities. I found quite a few pull requests for issues I was having that had been rejected. The maintainers seem understanding, but they don't accept duct tape patches, instead wanting to redesign their cores to properly support the features they initially hadn't accounted for. And it appears there's just never time granted for the NSA to accommodate to nostalgic gamers.

At this point I had long since given up on documenting all the Ghidra issues I found, and just did a LOT of manual fixups. The upside to this was that I got a solid refresher on Stunt's code. The next step was to port the IDC script to Ghidra. I chose the Jython approach to interface with Ghidra's Java API. It's archaic Python 2.7 and very, very slow, but it works right out of the box, no compilation needed, and it's thus very hackable. Initially I thought reproducing TASM output like the original script would be the easiest solution. But nothing is easy. While ID*'s native disassembly syntax is more or less MASM/TASM compatible, Ghidra goes its own way. Things that are implicit in Ghidra may have to be explicit in TASM, and vice versa. When I found myself writing my own x86 disassembler in Ghidra Script in order to format memory operands with proper TASM syntax I realised we might just target a contemporary assembler. I first pivoted towards NASM, it's widely (pun intended) available and has the best documentation of any assembler by far. But its syntax is even further out there. I eventually settled on the Watcom Assembler, whose name unfortunately now means something very different. There are several modern WASM forks, but I wanted to use the whole Watcom toolchain anyway, so I settled on the original to avoid further dependency complexity.

I wanted to make patching of the generated assembly code dynamic. This is done by using the tags <REPLACE>, <INSERT>, <NOP> and <DELETE> in comments. See the README for more details. This also proved to be a convenient crutch when my Ghidra knowledge fell short:


Another change is that only one set of asm files is created, instead using a build-time toggle to determine whether to wire up ported or keep original functions.

Something we wasted a lot of time on back in the day was trying to use unofficial ID* database synchronisation tools so that we could work on it simultaneously. Ghidra comes with its own server which I have set up on re.stunts.no. Anonymous readonly-connections are accepted. For those who want to contribute, send me a PM with a username and I'll add you.

I've gone back to the trusted old wlink. This sacrifices support for Turbo Debugger, but Watcom Debugger also seem very capable:

wlink can also produce DWARF and CodeView debug data, so it may open up for even more debuggers?

For the C code I wanted to carefully pull in one function at a time from the original restunts code. Making sure everything is portable and correct. So far I've only added the good old sin_fast() and cos_fast(), and made unit tests for these. The GitHub repo has a CI-task for building and testing, and test output can be inspected there. I haven't yet made up my mind how we'd go about testing functions with side effects. I don't know how realistic it is, but I want to see how far we can take it by only using Stunts' clib in seg010 and not link in any Watcom libraries as long as we rely on linking with the original code.

I've only tested building on Linux. Adding Windows support again will probably take some work, and I don't know if it's worth it as it probably works out of the box in WSL. There's also no proper dependency tracking for incremental building yet. Compilation is so fast that I've just been doing rm -drf build && make so far. Which I hear is in vogue with the handmade crowd these days anyway.

it would have been very easy if it was originaly developed for 32bit systems - like many other DOS games - then we could only replace the hardware access stuff and it would then run on windows/linux and we could have used todays development tools - but its just way too old

There are two 32-bit builds of Stunts; FM Towns and FM Towns Marty are built for i386, using the Phar Lap 386|DOS-Extender. While FM Towns isn't DOS compatible, and these versions aren't entirely bug-compatible with the DOS versions, there's a lot of shared code that is far more pleasant to explore without segmented memory. This is where the 32-bit anecdote was supposed to end, but I wanted to add these executables to the Ghidra server repo, and since Ghidra don't have a loader for Phar Lap payloads, I had a look at the files to see how much effort it would take:

That's some peculiar structured information after the DATA section... I can't believe it took 32 years for us to find out that WE HAD DEBUG SYMBOLS FOR STUNTS ALL ALONG! It's only public symbols, no types or stack variables, but still. It's over 2000 symbols. Since the porters appears to have kept many of the original DOS funtions as stubs, we even have these names.

Here's a function in the FM Towns Marty port and DOS side-by-side:

This is a striking example showing how spot-on some of the original analysis is, how unknown data now has clear names, and how the FM Towns code deviates from the DOS code when dealing with IO. When the FM Towns code has generated labels (_DAT_addr) it's usually because it is a struct member offset, but we only know the name of the root value.

I added both FM Towns executables in the Ghidra repo with all debug symbols loaded, no further analysis. I discovered this just now, so I haven't explored the symbols in depth yet. I'm not sure if we should adopt all the original names, as some are quitehardtoread compared to our snake_case_notation.

More details on how to connect to the Ghidra server, use the Ghidra script and build is in the repo readme: https://github.com/dstien/restunts2
The problems I encountered myself already are covered in the troubleshooting section.

[Daniel3D]

  I'm going to need some time to wrap my head around this... Christmas is early this year it seems 

[Matei]

As I wrote before:

https://forum.stunts.hu/index.php?topic=4257.msg92963#msg92963

if, by any chance, the Chinese government will want it done, I can assure you that it will be done.

Besides, everything that can be obtained was already. Maybe some more motorbikes would work...

[Duplode]

This is fantastic work @dstien , thank you so much! Can't wait to try it out! (Yes, I've been repeating this since the beginning of the year about e.g. @HerrNove 's improvements, but hopefully I'll stop getting sidetracked anytime soon! 😅) No longer needing the IDA-centric workflows and having the 32-bit symbols as a reference should be a huge boon to analysis.

[Cas]

Amazing, @dstien! 
Is it possible to isolate the physical engine for analysis?  I feel that many parts of the code, such as the menus, file loading functions, etc., would be easier to rewrite than to port, and the graphics engine, while more significant, is the one that could use an upgrade, but the physics engine is crucial so that we can actually reproduce the behaviour, remain replay-compatible, etc. If we could work shell by shell around the physics engine, great things could be achieved!

[llm]

Particularly the database from the program whose vendor has a history of refusing to sell to people outside the infosec clique.

isn't that like the myth that IDA dropped DOS support (which is only true for IDA freeware versions)

i know serveral private people that bought an IDA license - like me - without any infosec background

[Daniel3D]

Particularly the database from the program whose vendor has a history of refusing to sell to people outside the infosec clique.

isn't that like the myth that IDA dropped DOS support (which is only true for IDA freeware versions)

i know serveral private people that bought an IDA license - like me - without any infosec background

Still, the fact remains that IDA is a pain in the but, especially for those that don't use it enough to warrant a paid licence. So whatever way you look at it, the Ghidra conversion makes a huge difference.

[Daniel3D]

Amazing, @dstien! 
Is it possible to isolate the physical engine for analysis?  I feel that many parts of the code, such as the menus, file loading functions, etc., would be easier to rewrite than to port, and the graphics engine, while more significant, is the one that could use an upgrade, but the physics engine is crucial so that we can actually reproduce the behaviour, remain replay-compatible, etc. If we could work shell by shell around the physics engine, great things could be achieved!

If we can isolate the part that replay dump uses I think we are halfway there. Maybe there is a "live" version of that calculation but I'm not sure.

That is the hart of the game, if we understand that part fully we can hook it to CAS his graphics engine, replace the track editor with a simplified version of bliss, replace the car and opponent selection and extend the track elements library with new and exciting possibilities. (For example, add snow, sand and dirt to the grass and water terrain options).

If we don't get to understand it fully, everything except the track elements library is still possible..

[llm]

@dstien

https://github.com/dstien/restunts2/commit/daa05e093c3e56bfa6a7a9610173dc5db2db8cfe

will you also try to compare to the original implementation - something like int_atan2 disassembly + C port over the complete integer range or is something like that too much?

for example: some x/y combinations don't give the same result

range from x/y -1000...1000 - the list contains only different result comparing the C port and the original disassembled function

https://pastebin.com/f9nucieg

my test-code looks somway like this

Code Select Expand

   
    const int16_t min = -1000;
    const int16_t max = +1000;

    for (int16_t x = min; x < max; ++x)
    {
        for (int16_t y = min; y < max; ++y)
        {
            int16_t a = int_atan2_asm(x, y);
       
            int16_t c = int_atan2(x, y);
           
            if(c != a)
            {
              printf("x=%d, y=%d -> C=%d, ASM=%d %s\n",
                     x, y, c, a, (c == a) ? "OK" : "!!FEHLER!!");
            }
        }
    }

compile with current Watcom V2 on Windows for DOS 16bit Model small exe using wcl+wasm
the test run was done on a real DOS 6.22 on VMWare Player (because dosbox failed to run the complete test without hanging)

but im not sure what the defined range for these functions are

[llm]

Still, the fact remains that IDA is a pain in the but, especially for those that don't use it enough to warrant a paid licence. So whatever way you look at it, the Ghidra conversion makes a huge difference.

100% ACK

[Matei]

if we understand that part fully we can hook it to CAS his graphics engine,

CAS's graphics engine is in FreeBASIC, it's slow and it's not finished. You already have a usable graphics engine which includes CAS's part that reads tracks from Stunts, converted from FreeBASIC to C.

https://matei.one/idxscr.html

https://sourceforge.net/projects/simple3d/files/SDL_grf/

Documentation of the graphical functions included. You can combine the part that renders the graphics from SimcarStunts with the part that reads tracks from Stunts, in the same program. I already fixed the 3D models to be rendered properly. Next:

replace the track editor with a simplified version of bliss,

Why not just keep the track editor out as a separate program? Much easier.

and extend the track elements library with new and exciting possibilities.

Which I actually did from the start before using the tracks from Stunts - link above and I also made some programs to edit 3D models, also there.

[Daniel3D]

CAS's graphics engine is in FreeBASIC, it's slow and it's not finished. You already have a usable graphics engine which includes CAS's part that reads tracks from Stunts, converted from FreeBASIC to C.

You are right. Still I personally prefer CAS his approach to the graphics. It isn't finished, maybe it is to slow and would need to be rewritten in C. But it it uses a painters algoritme like stunts. And that gives the same ambiance as the original.

CAS and I share the same opinion about what is unique about the game an should be preserved. That makes it easy to work with each other. I really like what you created and i think you did an amazing job on your games. It's is not what I'm personally looking for.

So that's why my first thought went to CAS. It was not intended to exclude you from this and I am genuinely interested in what it would do with your graphic engine. But that would mean you have to accept stunts pseudo physics.

Why not just keep the track editor out as a separate program? Much easier.

That is true, but stunts has an track editor, so we want to keep it for future players to have one game that includes all. Including a build in car manager.

[dstien]

Is it possible to isolate the physical engine for analysis?

Yes, this was the motivation for the repldump utility. We took the code path that Stunts uses when loading a replay from the options menu and processes it to the end, then ripped away as much as we could without breaking it. Simulation still requires a fair bit of loading and initialisation. The old repldump was a bit hairy since we hadn't analysed enough to properly separate the graphics, audio and input initialisation. That's why it changes video mode on startup and throws a fatal error after processing in order to stop all the interrupts. clvn did a lot of analysis and porting since, so with the current knowledge I have streamlined repldump further.

The essential part is:

Code Select Expand

// ...initialisations...

while (gameconfig.game_recordedframes > state.game_frame) {
    input = td16_rpl_buffer[state.game_frame];
    state.game_frame++;
    player_op(input);
}
repldump is slightly more complicated because we want it to create the exact same game state as the original code, so it also handles the movement of an opponent, the cameras, explosions and FPS smoothing.

player_op() (debug symbol moveplayer) is translated to C in state.c in the original restunts repo. The inner function call to update_car_speed (doengine) is in statecar.c, and update_player_state (movecar) is in stateply.c. These are mostly 1:1 unanalysed ASM to C translations. I can't remember whether they are 100% complete. The other inner function calls upd_statef20_from_steer_input (dosteering) and update_grip (dohandling) are not ported, but these are fairly simple functions. My current focus in restunts2 is to start with the basic leaf functions of the simulation and work my way down the call stack so that all ported code is tested for correctness when it is included.

Particularly the database from the program whose vendor has a history of refusing to sell to people outside the infosec clique.

isn't that like the myth that IDA dropped DOS support (which is only true for IDA freeware versions)

i know serveral private people that bought an IDA license - like me - without any infosec background

Hehe, true. I got a bit carried away with my bombastic parroting of blog hearsay from 18 years ago. At least now I know that 50% of my IDA licence holding friends don't work in the sec biz.

will you also try to compare to the original implementation - something like int_atan2 disassembly + C port over the complete integer range or is something like that too much?

That's a good idea to catch corner cases. I added an "-x" option to the DOS build of the unit tests. The full range should be INT16_MIN + 1 .. INT16_MAX - 1 (peak values crashes both implementations), but it takes an eternity to run, so I just used -0x400 .. 0x400

compile with current Watcom V2 on Windows for DOS 16bit Model small exe using wcl+wasm
the test run was done on a real DOS 6.22 on VMWare Player (because dosbox failed to run the complete test without hanging)

Does this include the fix I pushed the other day? Strangely I can't reproduce it on my end with dosemu2 or DOSBox. Could you attach the exe file? I'm keen to have a look.