Car Resource files, Stressed and the Wiki

[Cas]

As you may have read in another thread, I've realised that we've been doing it wrong about handling Car Resource files. I think it's important to point this out and explain it and, well, do something about it. I'll expand here:

Origins of the problem
Stunts stores most of its resources in a file format we normally call the resource format. This is nothing but a chunk-based binary format. The header specified a number of chunks and for each chunk, you have an ID and a position within the file. The notable thing about this format when compared to so many other existing chunk-based formats is that it does not specify the chunk lengths. Some format may also not include this, but they seem to have a rule that chunks are described in the same order as they are situated in the file and one always starts just after the previous one. With this, we can easily calculate the chunk length by substracting a chunk's position from the next one's. The thing is, Stunts does not do this, or not always. Some resource files (particularly, Car*.Res files) originally coming with Stunts include their chunks in another order. This complicates loading these resources for editing when creating tools.

What we have done about this in the past
Because most original resource files do have chunks in the expected order, most files have been analysed as what they are. But for Car Resource files, as all original cars had the fields in the same order and the most important one is of fixed length, we have been describing this file with fixed offsets from the beginning of the file. To the point that this is the way in which the Wiki lists the contents of the files. Some tools have been taking it for granted that files will be ordered like this. I can think of a few examples: CarWorks, Dash Manager, Car Swapper, Car Blaster, etc. One tool that does not do this, for instance, is Stressed. And all this would be fine if Stressed saved the files with the chunks in the order we're used to, but this isn't the case. Car Resource files created with Stressed use the growing order instead of Stunts ordering and thus, are incompatible with tools like Dash Manager and Car Swapper.

What I think we should do
Really, we can't blame Stressed. It really tries to be as clean as possible, cleaner than Stunts itself. And it does it well, because Stunts does not complain about the new ordering. That means that it's us that should make the change. To quickly patch this issue, I can try to develop a little program to convert between these two orderings, but this is a patch, not a solution. I will need to change Dash Manager... oh, and CarWorks... to be able to handle the files for input the same way Stressed does. But I recommend that, for saving, when we develop tools, we continue to save in the original order that came with Stunts. Stunts will be able to work with files in any order, but using this order guarantees that tools that didn't take the precaution that Stressed took will be compatible with our files. Apart from this, I think we should be updating the Wiki to make this clear and also to describe car parameters in terms of the chunks they're contained in. I can do that.

A way to do the loading easily
I've been thinking what would be an easy way to load these files, since they don't specify chunk lengths. I suggest the following. We read into an array all IDs and their corresponding offsets. Then, we sort this array by offset and optionally, add one extra item and set its offset to the total file length minus the header's length. Finally, we load each chunk's contents calculating the chunk length from the next chunk's offset minus the current chunk's offset. When we get to the last one, this calculation will work too if we've included the extra item. It's not super nice, but it's easier than any other loading method I can think of that is compatible with any ordering. This loading procedure can be use with any resource file. Saving is a lot easier.

[Duplode]

But for Car Resource files, as all original cars had the fields in the same order and the most important one is of fixed length, we have been describing this file with fixed offsets from the beginning of the file. To the point that this is the way in which the Wiki lists the contents of the files. Some tools have been taking it for granted that files will be ordered like this. I can think of a few examples: CarWorks, Dash Manager, Car Swapper, Car Blaster, etc. One tool that does not do this, for instance, is Stressed. And all this would be fine if Stressed saved the files with the chunks in the order we're used to, but this isn't the case. Car Resource files created with Stressed use the growing order instead of Stunts ordering and thus, are incompatible with tools like Dash Manager and Car Swapper.

I'd just add there is a workaround for that issue in Stressed: resources can be reordered by right clicking on the resource list. That way, CAR*.RES resources can be put in the usual simd-edes-gsna-gnam order. I have attached a version of Ryoma's CARBRAV.RES adjusted in that way.

But I recommend that, for saving, when we develop tools, we continue to save in the original order that came with Stunts. Stunts will be able to work with files in any order, but using this order guarantees that tools that didn't take the precaution that Stressed took will be compatible with our files.

Sounds like a sensible precaution for CAR*.RES-specific tools, for the sake of Car Blaster at least. (By the way, if I ever get to rewrite Car Swapper I'll use proper resource loading rather than just assuming fixed offsets.)

I've been thinking what would be an easy way to load these files, since they don't specify chunk lengths. I suggest the following. We read into an array all IDs and their corresponding offsets. Then, we sort this array by offset and optionally, add one extra item and set its offset to the total file length minus the header's length. Finally, we load each chunk's contents calculating the chunk length from the next chunk's offset minus the current chunk's offset. When we get to the last one, this calculation will work too if we've included the extra item. It's not super nice, but it's easier than any other loading method I can think of that is compatible with any ordering. This loading procedure can be use with any resource file. Saving is a lot easier.

Yup, that would work. FWIW, the latest, post-0.21 Stressed builds do that upon loading resources, and I wouldn't be surprised if it turned out dstien added that in early 2013 to deal with the problem you are describing here.

[Cas]

if I ever get to rewrite Car Swapper I'll use proper resource loading

I wouldn't have known about Car Swapper doing it this way if it weren't that Daniel3D asked me why the Bravo didn't show the car name properly, but gibberish. He thought it was a problem with the car. I had tried to open the RES file with Dash Manager and it behaved very strangely, so I immediately guessed it was about the order or the chunks. I took a look at the binary contents and realised all this.

I hadn't noticed that it was possible to reorder the resources with Stressed. That's a good thing. As I said, I think Stressed is doing it well. Dstien had to choose between following the typical rule and using a comprehensive approach to resource files. He chose the latter, which in my opinion, is the right way of doing the things. This is the same reason why some time ago I defended my overlay format for tracks: a set of files having something in common does not mean there's a standard telling you have to do it that way. We have to understand what is possible, what works and what does not and build on that.

[Daniel3D]

I'd just add there is a workaround for that issue in Stressed: resources can be reordered by right clicking on the resource list. That way, CAR*.RES resources can be put in the usual simd-edes-gsna-gnam order.

I knew of that from an other topic. I believe it had to do with carblaster compatibility.
But it doesn't help if you don't know what the right 'original' order is.

simd-edes-gsna-gnam. - noted.. 

[Ryoma]

Noted too

[Daniel3D]

With the model file it's not really a issue.
That's both automatically sorted right and very well named.
Car0 is the largest then car1 and car2
Then of course exp1-4..

When editing stressed scrambles it up regularly, but I always fix the order before I save.
I didn't notice the wrong order of the Res file because I always use stressed..

[Cas]

Original Stunts resource files, if I know well, all have the chunks described in the same order as they are stored except Car*.Res files. The reason I can think of for this is that CAR*.RES files have one chunk ("simd") which has a fixed size. Putting it first allows for the offsets of local car configurations to be fixed as well even after changing the car name, which has a variable length. This suggests that DSI may have been using some tool they had made quickly that edited car characteristics directly without properly handling the RES file, akin to Car Blaster while files probably were originally "compiled" with another tool and each component likely edited separately before being packed together. This is all just guessing, of course.