Stunts code deep dig.

[Daniel3D]

Disclaimer: this post is subject to change.

I have a dream.. and as with so many of them I lack the time, knowledge and skill to make it happen by myself.

But I asked around and the idea is solid, it can be done, but it is a lot of work.
There are two phases in this project.

• Dissecting a specific part of the game code containing the main menu and its i/o
• Replacing that part of code with newly assembled code.

The first part connects to the restunts project and we can continue on that base.
The second part will be mostly assembly programing on the back of CAS (sorry dude, more work)

Because phase one has a lot of unknowns, I can not be sure if we can reach phase 2.

Me, Duplode and CAS think it will be great and will get started.

Do you want to know what it's all about?
(I'm hesitant to tell the exact plans here because i'm not sure what is possible and how much can be done)
But if you opt in you'll get full disclosure, and you'll know why I'm hesitant.

This journey should not be started if you only care for the end goal and not the adventures along the way

But anyone is welcome to travel with us.

To get a feel of what you'll be facing, check http://forum.stunts.hu/index.php?topic=3714.0

[dreadnaut]

Well, that's a vague manifesto   Here are some vague considerations.

I've long been of the opinion that if there is a future where Stunts evolves into a different software, that's a future in which we have understood the game engine, rebuilt its source code, and brought it forward into a modern package that supports higher resolution, long rendering distance, shadows and fancy graphics in general. But the game feeling stays precisely the same. Think about what source port have done for Doom or Quake for example. Any other game or engine that tries to be Stunts but is not Stunts at its core, will not succeed in holding the attention of the community. Very few people want a remake, and that's why remakes fail.

Alas, we start without sources, and that's a giant obstacle. It could be worth looking for them again, contacting people, investigating and asking around. The internet has changed, and the treasure hunt could attract people. In the meanwhile, reverse engineering takes time and patience, and as we age we might gain the latter, but it's difficult to make Stunts a priority.

If you chaps are renewing the effort in that direction, that's great news! My suggestions would be to document everything as the project evolves. Hopes, small steps, obstacles and breakthroughs—in public. The more information is out there, the easier for interested people to discover it, get involved, and bring help. It is also a project for the ages; you are not the first to start on its path, you might or might not be the ones who complete it.

I'm sure there'll be plenty of support, even if it's just infrastructure or cheering

[Daniel3D]

Well, that's a vague manifesto

You are absolutly right, I made it less vague.
It is a relative small part that could make a lot of difference.
Could make gtaman's april fools joke partly reality..

[Cas]

Well, I've been pretty busy these days and will continue to be. Even CarWorks is taking me time. I was able to complete the difficult part of it, so now other features will be easier to implement, but it's still lacking many things, so I have to dedicate the little free time I have to it.

On the other hand, what Daniel wants to do would be of significant importance among the projects in the community. From what we have been talking, I could say it's the opposite approach to that of my engine, or kind of. I know what Restunts is about, but I don't know the code in enough detail to be sure about whether the parts that are required for this have already been tackled. Essentially, if I understand Daniel correctly, the right approach would be to "detach" the code that's at the main menu screen in Stunts from the rest of the executable and then place new code there, reconnect the cables and turn the machine back on. The new code would, however, include sockets where plug-ins could be directly, well... plugged. This new code could itself be a menu or could be just a "lobby" and the new menu would be a plug-in that may or may not resemble the original one. Plug-ins would have to be aware of the memory structure in Stunts or else, the "lobby" would have to do that and provide some kind of an API. Of course, all plug-in code would be real-mode x86 code, but, while the "lobby" would have to be assembly, the plug-ins don't have to. They could be any compiled executable program, as long as they are real mode.

I tried to explain to Daniel how immense this work would be, but he's got a lot of enthusiasm and optimism and I think anybody with a such a view should be supported. So yes, if the project moves forward, I'll see where I can get some time and you guys can count on me.

[llm]

• Dissecting a specific part of the game code containing the main menu and its i/o
• Replacing that part of code with newly assembled code.

that could become a hell lot of work - and i would not add new assembly code - link c code objs into the game (thats how the restunts porting is done)
you need to know that stuff very much in detail - 16bit assembler, dos memory segmentation, ... etc. if you use assembly

would be better to further/finish port restunts to C - but that will also costs month of hard work

its also possible to involve (testing etc.) yourself in the https://github.com/xor2003/masm2c project - its an converter for the assembly to "functional" equaivalent C code - the author uses restunts as testbed, maybe he needs some help in testing etc.

[Daniel3D]

that could become a hell lot of work - and i would not add new assembly code - link c code objs into the game (thats how the restunts porting is done)
you need to know that stuff very much in detail - 16bit assembler, dos memory segmentation, ... etc. if you use assembly

We were thinking about disassemble the code into flat assembly with Fasm and replacing the relevant part (as little as possible) with new assembly code (By CAS)
that seems sensible, it keeps most of the game code unaltered and increases the chance of success.
The restunts project has done a lot of research already and I did opt to continue this project on that basis (not knowing how its done) so it may well be easier to continue in C.

Don't know at this moment. But thanks for bringing this up.

would be better to further/finish port restunts to C - but that will also costs month of hard work

I fear that will overshoot the goal of the project. It is supposed to be a small surgical operation to increase chance of success. (but the idea is now on the table)

its also possible to involve (testing etc.) yourself in the https://github.com/xor2003/masm2c project - its an converter for the assembly to "functional" equaivalent C code - the author uses restunts as testbed, maybe he needs some help in testing etc.

Thank you. I'll look into this.

On a side note, llm will you join the project? Your knowledge of the restunts project and the game in general would be very valuable to us.

[Cas]

This is indeed a ton of work, like something to take years, really.

I mentioned Flat Assembler because it's the best assembler I know and it allows you to concentrate on instructions and not have to deal with too many directives unnecessarily. But if the code has already been extracted in another assembly format, it would be a whole lot of additional and unnecessary work to translate it if it can already be compiled.

Inserting a piece of code can be done by creating a C function, but it would be a very low level function because it has to be aware of all the pointers in the code if it's going to be the place where other mods are plugged, so it's almost the same as assembly. If this part of the program is already translated to C (I mean, the lobby/main menu), then it could be done in pure C. If it hasn't yet been translated, we could try to translate that part to C or we could just make an assembly routine and plug it there. It wouldn't be very difficult for that particular part because it's the simplest bit of the whole code. The difficulty lies in how it connects to everything else.

[Daniel3D]

It is supposed to be a small surgical operation to increase chance of success.

On that matter, a little piece of success. But it is low-hanging fruit.
The information was there already, it could have been done in 2009 if anyone had been interested in it and looked for it.

http://forum.stunts.hu/index.php?topic=3762.0

[Cas]

Really, there's a lot that can be done with the level of knowledge we currently have about Stunts. We can move further in learning, but I think we haven't been doing as much as we could already!

[llm]

We were thinking about disassemble the code into flat assembly with Fasm and replacing the relevant part (as little as possible) with new assembly code (By CAS)
that seems sensible, it keeps most of the game code unaltered and increases the chance of success.

1. you can't disassemble the code with "Fasm" - its just a assembler not a disassembler - and linear-sweep disassemblers like ndisasm does not work very good with the stunts code - nor is the resulting assembler notation to Fasm, IDA Pro (and maybe Ghidra) is the only choice for disassembling - and sorry the free version of IDA will not work with 16 bit DOS exes (only the old freeware version 5)

2. the code was already and working disassembled more then 10 years ago - the asmorig folder contains the assembler-code that creates a (or did last i checked) a binary identical game.exe - in complete around 230.000 lines of assembler
its in TASM/MASM style - second most used assembler out there

3. the c-code is much easier to read (but not as pure original as the asm-code) but the asm code is and will be become easily 10 times more code compared to the C version

4. you can't port that code easily to windows or something more 32 bit so you need to stay with the old gibberish dos tools of that time - very hard itself
->i've started to port the asm over to a more recent assembler called UASM that is pure 32/64 bit hence no need for DOS for development tools - still a DOS exe result
but my spare time is just too limited currently

On a side note, llm will you join the project? Your knowledge of the restunts project and the game in general would be very valuable to us.

sorry my spare time is too limited - but i can answer any question related the restunts project

tip: first you should build the pure asmorg stunts exe - everything is described in the repo - you should lower your goal if that becomes in any way a obstacle for you and your team

my big hope is the masm2c project - it will finally converts all the asm in asm-like C code buildable under win32/64,linux and then everyone can use their prefered IDE and debugger to help porting - currently you need to use DOS tools and that itself is such a annoying burden

[Daniel3D]

1. you can't disassemble the code with "Fasm" - its just a assembler not a disassembler

I realized that to. I can't program myself, so there is a lot I don't know. But among all the crazy things I come op with, sometimes they are really good.
So I accept that I occasionally miss the bat.

2. the code was already and working disassembled more then 10 years ago

I've been going through the asm-code this weekend and i'm getting the scope of what is already done. And it is impressive.

3. the c-code is much easier to read (but not as pure original as the asm-code)

Although I can not write code, I can read it reasonably well. I have no problems with asm so far.

4. you can't port that code easily to windows or something more 32 bit so you need to stay with the old gibberish dos tools of that time - very hard itself
->i've started to port the asm over to a more recent assembler called UASM that is pure 32/64 bit hence no need for DOS for development tools - still a DOS exe result
but my spare time is just too limited currently

I am looking for people outside the community that like to reengineer games. (no luck so far though)

On a side note, llm will you join the project? Your knowledge of the restunts project and the game in general would be very valuable to us.

sorry my spare time is too limited - but i can answer any question related the restunts project

Whatever you can share is more than welcome.

tip: first you should build the pure asmorg stunts exe - everything is described in the repo - you should lower your goal if that becomes in any way a obstacle for you and your team

I put CAS on the technical stuff. As for the goal. We are setting up base. Looking at how things are connected. We have several goals depending on what we find.
The Ferrari edition was on the table from the start. That was easier than expected. But I knew that would be possible.
Adding a new menu option, a extra button will be mush harder i think. Well see.

my big hope is the masm2c project - it will finally converts all the asm in asm-like C code buildable under win32/64,linux and then everyone can use their prefered IDE and debugger to help porting - currently you need to use DOS tools and that itself is such a annoying burden

That would be assome. That would make changes possible we can now only dream of..

[Cas]

I'm comfortable enough to work on pure assembly, especially DOS assembly code. Only problem is it's a huge lot of code and I'm in one of the busiest periods of my life. I have to learn to understand the directives, but that shouldn't take long

I think it's not necessary to perform a comprehensive analysis of the whole code to be able to work on it. I would concentrate in identifying how the original compiler stored local variables in functions and where it put the global ones. The MZ model is important too, because it will tell us how the position of the different memory segments can vary in case there's any code that assumes a location, but I don't think this is the case because Kevin Pickell once said everything but the 3D rendering (and perhaps collision) engine was made in C. Local variables of routines that we don't want to change can be left alone. With all that identified, we can make a "list" of the points the the program where global variables are accessed and also, local variables of a focal routine, such as the main menu, for instance, could be taken into account. With this, the code could be pretty cleanly separated from this function that would then be replaced by a mod dispatcher. It is complex, but it's not as huge as it seemed at first.

Another thing I'm seeing now is this same procedure could allow for converting the functions one by one to 32 bit code and replace the memory accesses accordingly, since each function almost never will access other segments except for falling other functions. Video memory access can be redirected to a buffer that can then be processed by a native function for each system. In other words, the code could be ported without having to really translate it or even go through a C-phase. I'll try to find a moment to dig more into it.

[llm]

I have to learn to understand the directives, but that shouldn't take long

there are nearly no to none highlevel directives in the code - should be very easy comming from other assembler syntax

I think it's not necessary to perform a comprehensive analysis of the whole code to be able to work on it. I would concentrate in identifying how the original compiler stored local variables in functions and where it put the global ones. The MZ model is important too, because it will tell us how the position of the different memory segments can vary in case there's any code that assumes a location, but I don't think this is the case because Kevin Pickell once said everything but the 3D rendering (and perhaps collision) engine was made in C.

nearly everything is ready analysed in the asmorig code - even with names for the variables
this is the complete source that builds out of the box (using the Readme to understand how) which will produce a nearly binay exact version of the game
https://bitbucket.org/dreadnaut/restunts/src/master/src/restunts/asmorig/

it sounds sometimes a little like you all want to skip that code - but maybe its just my interpretation
month of hard word were invested to reach that point - and there is even a IDA pro file that can be further added with new findings

the current state was done in a very professional manner:

1. using the original game.exe from the combine process and IDA Pro to analyse the game
2. changed the IDA assembler output to get the same binary results (100% identical to game.exe after reassembly)
3. a script that splits (automaticaly) the asm code into segments for easier porting/understanding
4. another part of the script generates source that combines with the c code
-> that means every change in the IDA Pro analyse database will be automaticaly reflected in the asm output
the more analysis, findings, renames etc is done the better the asm output looks
(around 3 Month of hard work of 2-3 developers)

so in short

game.exe -> IDA Pro -> IDA Script which creates
  1. splitted into segments based code that reassembles to the very original game.exe (compared on byte-level with orginal segments)  (src/restunts/asmorig)
  2. the version that mixes the original Asm with C and the MSC 1.5 stdlib () --> src/restunts/asm

and the build is handled with make/batch-scripts

all i want to say is: please all, look first at what others have done already very professional
or related to your "menu" stuff - just search for "menu" in the asmorig code and you will see

Another thing I'm seeing now is this same procedure could allow for converting the functions one by one to 32 bit code and replace the memory accesses accordingly, since each function almost never will access other segments except for falling other functions. Video memory access can be redirected to a buffer that can then be processed by a native function for each system. In other words, the code could be ported without having to really translate it or even go through a C-phase. I'll try to find a moment to dig more into it.

could become hard - there are >700 functions, all segment registers are in use everywhere, so not only ds-register is responsible for accessing the data, offset calculation that is relative to its adressing segment etc. - not that easy

the C-phase solution (or what masm2c does) is more a way to get the complete game into a todays IDE/debugger - because there the changes are easy and much better to test (regression tests etc.) the DOS environment is just too limited for 230.000 lines of assembler to port- without introducing too much new errors

[Daniel3D]

https://bitbucket.org/dreadnaut/restunts/src/master/src/restunts/asmorig/

it sounds sometimes a little like you all want to skip that code - but maybe its just my interpretation

This part I had not found yet correction, it was what I've been digging in. We are very happy with the quality of what you have done. But for me it is very difficult to see/understand how much is done.
I believe I have read all the relevant information about it. But it's tough reading for a noob.

all i want to say is: please all, look first at what others have done already very professional
or related to your "menu" stuff - just search for "menu" in the asmorig code and you will see

That is exactly what I did. But in a few segment, not in the code yet.

The hard coded Countach I found with "set_default_car". Was nicely marked with notes as well. (thanks to Duplode for the hint in this direction)

Edited: extensivly because of failing memory.

[llm]

We are very happy with the quality of what you have done. But for me it is very difficult to see/understand how much is done.
I believe I have read all the relevant information about it. But it's tough reading for a noob.

its a huge project even for very professional developers that do that stuff or system programming on daily basis

to reverse some algorithm is one thing, reverse a complete project or change parts of it is a complete different thing

the biggest problems are non-symbolic references (like non named offsets etc. in asm code) that forces you to keep the binary layout very near to the original
or else you will get hard to understand random bugs - but without a development background its nearly impossible to have an understanding of what that mean for debugging/bug fixing  - i don't how much relative offset based code is still in there - that means for example adding code add the end is mandatory (or right before the stack)

problem ares: you are a noob and CAS did't not like industrie norm development