DOS and DOSBox memory systems and Stunts

[Cas]

Introduction
As I was saying in the shoutbox, I'll post some things I know about how things work in a real DOS PC and how similar or not they could be in DOSBox, particularly as it could affect Stunts. Most of this is just for curiousity, but maybe there's a use for it. Feel free to comment or correct me if I'm wrong.

Memory system
The first versions of DOS were created to run on 16bit PCs, that is, computers that were based on the Intel 8088 processor, which is a cheaper version of the 8086. Both were 16bit (had 16 bit registers and addressing instructions), but the 8088 internally had only an 8bit data bus, which made it somewhat slower. No difference for the programmer.

Real Mode Memory
These microprocessors are special in that they address memory by combining a segment and an offset, not just one address. That is, say you have a 16bit address (pointer). That would be a number from 0 to 65535 (FFFFh in hexadecimal). You have one byte at each address, so you can store a value at 0, another at 1 and so on, up to 65535 byte values. Now, that's only 64K, not much. The classical way to multiply this was to just add memory "banks", so you'd have another register that would hold the bank number. If you had 10 banks, that would total 640K. The problem with this was that it wasted a lot of memory (64K was a lot back then), so Intel, for its 8086, instead of banks, used segments. A segment does not start just where the previous one ends. Instead, it overlaps with most of the previous segment, only offset by 16 bytes. So, for an address consisting of a segment and an offset, the byte at 0000:0000 is the first byte; next is 0000:0001, and so on, but the byte at 0001:0000 is the same as the one at 0000:0010 and the byte at 3000:9A16 is the same as the one at 39A1:0006. Clear?  Well, the purpose of this is to be able to load a program at almost any point in memory so that it'd still run. Stunts is not always loaded at the same address, but the offset at which it loads should always be the same.

On XTs, the maximum addressable memory was 1 megabyte, but because addresses starting at A000:0000 were dedicated to hardware, RAM could only be addressed up to 9000:FFFF, so max RAM was 640K. When the AT was introduced, one thing that was important was to allow for more RAM. This couldn't be done with this old system, so the old addressing system was called "real mode" and a new one was introduced called "protected mode". In protected mode, a segment can start at any point of memory and the segment number has nothing to do with its starting location. Also, the size of a segment does not have to be 64K. Because this is so different from real mode, in general, real mode (DOS) programs can't run in protected mode and vice-versa. Stunts was born in a era in which compatibility with XT was still important, so it only uses this "conventional" RAM up to 640K. But how do other DOS games access "extended" memory?

Extended Memory and HIMEM.SYS
Intel would've liked operating systems to immediately be updated to use protected mode instead of real mode, but things never change that fast. Several tricks came up to access extended memory, but the one provided with DOS usually was a driver called HIMEM.SYS. This driver, when loaded, installed some functions in memory that did things like "copy this amount of memory from this region in extended memory to this other region in conventional memory". The DOS program would call HIMEM in real mode, then HIMEM would temporarily enter protected mode, access memory, copy the data, then jump back to real mode and return control to the program. On the 286, it was not possible to switch back to real mode, so the whole micorprocessor had to be reset, yet preserve the memory, for this to be accomplished. In other words, 286s are extremely slow at accessing extended memory in DOS. 386s and later are able to switch in both directions. Still, switching does take time, so a DOS program that uses XMS (extended memory) is slower at memory access than one that runs in protected mode or one that runs in real mode using only conventional memory.

Expanded Memory and EMM386.EXE
Another popular driver for extra memory is EMM386.EXE. It's story is completely different. Before the AT, if you wanted more than 640K, what you could do is buy an expensive expansion card that you would place in an ISA slot and would add more memory to your system. Because the microprocessor could not address more than 1M, you had to install a driver. Calling some interrupt functions, you were able to change which 16K memory bank from the expansion card were to be mapped to a memory region in high memory (such as C000:0000, for example). These cards were forgotten when the AT came up, but because some programs made use of them, it made sense to create a driver that would emulate these cards with extended RAM. That's what EMM386.EXE did. It was so much easier to use this driver than HIMEM that many DOS games actually require it, even though they came up much later than these expansion cards were no longer used. The 386 or later, instead of switching modes and copying memory, would remain in protected mode all the time and "emulate real mode" using something called "virtual 86 mode", a much faster approach than that of HIMEM (with some drawbacks). But V86 does not exist in 286s. There was a EMM286 driver which worked doing something more like what HIMEM does.

DPMI and VCPI
The latest DOS games used neither of these methods. Instead, a special driver called DPMI or VCPI is loaded just before the game is run. The driver enters true protected mode and installs many memory management functions in memory, then executes the actual game program, which will run in flat memory, meaning it has full access to all of the extended memory without needing to set banks or to copy, just directly. This is a lot faster. Only loading and unloading the program is slower. A disadvantage is that accessing DOS functions now becomes much slower, as you have to jump to real mode, execute the function and return to protected mode, but while utilities do a lot of this, games do a lot less. These programs can run in 32bit native code. In theory, it would be possible to create a 64bit "DPMI" driver, but if it exists, I don't know of it. Latest commercial DOS games date from around 1997, much earlier than 64bit PCs.

DOSBox and memory models
DOSBox is a very powerful emulator. It is also build intelligently. It does not emulate what is not necessary to emulate. For example, the mouse in DOS is accessed by calling its driver functions, so DOSBox emulates the driver, but does not emulate the serial port it was usually attached to, because games normally didn't access that port directly. Same way, DOSBox makes sure memory is address where you expect, but doesn't emulate what would've make it available there in a real DOS computer. I think HIMEM.SYS and EMM386.EXE functions are ready to respond from the moment you run DOSBox. I even believe that some DPMI emulation is already available even if you don't load your own DPMI driver (which was what normally happened with DOS games), but memory access is probably just as fast as if it were direct, because no actual banking or copying is going on and there's no switching between memory modes being made. The microprocessor also may be emulated at a speed that is not typical of a computer that handles memory in the way the emulation is allowing, so some games may get confused about which computer they're running on and execute too fast or too slow. Stunts runs really well on DOSBox, but the default configuration is usually a little bit too slow and you may need to increase its speed to be able to race comfortably. This is because DOSBox is trying to keep emulation slow by default so that speed is consistent with the hardware the game will find emulated.

With DOSBox, some programs will not load at the same memory region as they would in a true DOS machine. If a program loads several modules at different memory locations, they may not end where expected. Most games don't have a problem with this, but sometimes, it is important. For example, the Neverlock crack that comes up when you run Stunts_K expects some relative location for some Stunts modules. If you load something before you load Stunts, such as a different mouse driver, for instance, and it takes up more memory, Stunts may end up loaded where Neverlock doesn't expect it. Both programs will run, but Neverlock will fail to disable the DRM system. If this happens, check to see what is being loaded before Stunts.

[Duplode]

Both this and Cas' investigation about corner signs have been moved to this board, which in turn has been renamed to "Stunts Reverse Engineering", as it had been suggested.