I remember when I offered my track 4AM.TRK to be raced on ZakStunts, there was a problem downloading it. It turned out it was because the file had an overlay I had created back in 2006 when I designed it. It had to be truncated so pipsqueaks could download it. So I assume ZakStunts still does not support track files with overlays.
Now I'm planning to add to my editor Cas-Stunts an option to add meta-data to tracks. My idea is to place this meta-data as an overlay just after the track data in track files, but I'm concerned this will interfere with ZakStunts. Of course, you will still be able to edit tracks without the meta-data, but if ZakStunts could detect this and truncate the file automatically when processing it, or better, actually read the meta-data, it would be much better.
I know for a fact that Stunts does not complain about TRK files having overlays. My idea is to include information such as a long track title, author's name, creation date, editing time, suggested car and other championship oriented information. If we could agree on a format, ZakStunts could even display this info. For example, you would not need to manually enter the track title. The file could be named ZCTxxx.TRK and still contain a title that would be automatically displayed on the site.
What do you guys think?.... Zak? Dreadnaut? ::)
The main issue with TRK metadata is the way Stunts behaves when saving files:
- Saving the track within Stunts would delete all metadata
- Saving a replay on a track with metadata would not carry the information along
This would make the record too easy to lose. Beyond that, it would also be very easy to modify, making it impossible to tell tracks apart using a hash of the file.
I'm now fairly convinced that metadata should follow along in an external file (readable without any tools), or be collected in a replicable database —if I had time, track.zakstunts.hu would be a thing already :P
<uninvited-comment>
Quote from: dreadnaut on August 12, 2016, 07:20:29 PM
The main issue with TRK metadata is the way Stunts behaves when saving files [...]
You have thought this out more thoroughly than I had done. IMO the first point is not so bad (after all, in-game saving already destroys tracks with special effects), the second is a significant annoyance and the third one is potentially a deal-breaker.
</uninvited-comment>
Yes, Stunts internal editor will truncate the track file to the first 1802 bytes, but unless the editor is used, the data remains there. About the replay, yes, the file would be truncated too. I don't think the meta-data is needed to identify the file; that can be done from the first 1802 bytes. The meta-data is only there for our archives. I could also make a companion text file, but it'd get easily lost. Or I could do both things, so when one is lost, it can be recovered from the other. All the site would have to do then is to throw away anything beyond the first 1802 bytes.
A problem with text files for this purpose is that, by just opening the file to look at them with a text editor and then accidentally saving it instead of just closing it (which is a very common thing), you could make the file not readable by the editor or by the site, because of things like line terminating characters not being the same across OSs, some editors appending a final end of line and others not, some editors appending an EOF, some using tabs and others, spaces, some including the BOM at the beginning of the file, etc. In other words, I can make the editor produce these files, but it shouldn't read them.
I'm not sure I understand the third issue. As I say, the first two are not really a problem. If you mean a way to identify the track, then as I say, you can just ignore anything after the first 1802 bytes, that is, same track with different meta-data is still the same track.
Quote from: Duplode on August 12, 2016, 08:36:01 PM
(after all, in-game saving already destroys tracks with special effects)
To be precise, saving does not destroy the special effects, it's the act of
editing the track the deletes the invalid tiles, and Stunts usually shows a message pointing out the error. The TRK file is still untouched, unless the player ignores the error message and overwrites the track.
The problem with postfix metadata is its
invisibility. You don't know when it's there, you don't know whether it has changed, and you don't get a warning when it's gone. It exists only when a specific external tool is used. A text file is instead universal: you can change indentation and encoding, but it's still readable by many tools and requires no instruction or specific precautions.
Regarding the identity problem, I don't think two identical tracks with different internal metadata should be considered the same. "ZCT181 by Zak 2015" and "ZCT181 by dreadnaut 1927" may look the same, but one is the original track while the other is a cheap forgery. For
archival, one would need to tell the difference, so the hash needs to include the whole file.
Uhm... well, I see two different things happening here, none of which is really a problem from my perspective. This will be a little long. Sorry. The most important part is at the end.
One thing is about the transferability and persistence of the extra data, that is, how to make sure the extra data are not lost. About this, we know the following:
- Stunts modifies track files in memory, but unless saved from the internal editor, they won't be modified
- We've already been dealing with information being lost if the built-in editor is used, as Duplode explained. I am not concerned about this happening
- Extra data would also not be transferred to replays, but there isn't any way in which this could be done by Stunts and truly, I don't see any reason why we would want it to be that way (yet, there is one way we could approximate this, as I will explain later)
- If extra data are passed in a separate file, Stunts will not overwrite it, but the chance of losing information will actually be increased, since it's very easy not to pass the companion file when copying it. If the only way to lose the information is to use the built-in editor, it is very easy to avoid it
The other thing is about properly identifying a track file and about what constitutes the track. We know:
- That meta-data could be easily modifiable, like Dreadnaut stated. And I would like to add that it would be as easily modifiable as the track itself. Actually, the track is easier to modify, because you can use many tools for that purpose, whereas the meta-data can only be modified by Cas-Stunts
- If we consider the meta-data to be not part of the track, we should work a hash function only on the first 1802 to identify a track, which would ensure that a track is identified regardless of the meta-data being tampered with, however unlikely this is. No problem.
- If we consider the meta-data to be part of the track, we should apply the hash function to the whole file, which is also just as feasible as doing it on the first 1802 bytes. The meta-data is really very unlikely to be lost and if it were lost, it is good knowing it was. I think this would be the best approach, personally. Notice that if the meta-data is passed on a separate file, the hash function will not be applied to it. So no problem either.
Most important part:
What I think is a problem is a simpler fact:
- We have seen that submitting a track file longer than 1802 bytes fails in ZakStunts, even though only the first 1802 bytes are necessary by the site to display the track and to verify it contains the same elements to the extent that they may affect the driving enough as to make a difference in record-making
- What I wanted to know was if it was possible to prevent that in any way and, possibly, but not necessarily, actually use the meta-data I would include. If this is not possible, which is OK, of course, then I will have to ensure that Cas-Stunts keeps allowing to save tracks that are only 1802 bytes, because that's the only other solution.
End of most important part :P
Parallel things that can also be done, though not exactly solutions:
- If a text file is appreciated and wanted, I can make Cas-Stunts produce on demand a text file to accompany tracks, that will look nice and include all information so the reader can get some info about who made the track, when, etc. I will consider it data export, since I don't believe it is safe to rely on the file format for reading information, but it will still serve a nice purpose.
- I can add an option so that designers can choose to save the track in a format to be read by ZakStunts, which can be just plain as now or any other format you would suggest. I can still support another format I choose
- I have noticed that some track code points (between 185 and 220 approximately) can be entered on a track file and will be interpreted by Stunts as a "vertical" road. These can be used to add information inside the 1802 bytes without affecting racing, although data space will be limited and the track will not look beautiful when seen in the perspective display screen in Stunts when selecting the track. I have no idea of how ZakStunts would react to these values.
For me the most important part and the rest are tightly connected: I wrote a library to read well specified file format, and you are asking me to relax some of the checks to accept a non-standard way of storing track information which I consider unwise for the following reasons:
- obscurity: metadata is invisible except to Cas-Stunts, and possibly phpstunts/ZakStunts
- unsafe: other standard track editing software might lose the metadata, or be unable to open the file
- frail: metadata can be lost following normal Stunts use, if you are not aware of it
- impractical: if aware of metadata, one needs to behave differently when handling the track, constantly mindful of the risk of losing metadata
- fragmenting: it creates different kinds of tracks, and splits tools between standard and cas-aware
- confusing: non-programmer players unaware of the metadata are likely to encounter trouble, or at least be confused by the different file size
And at the moment, the only use case for my changes is "Zak would not have to type the title and author of the track every now and then."
I hope I'm not sounding too harsh, but breaking standards is a big deal. I am also looking at this with an historical perspective: Stunts is 25 years old, and in 25 more years what would I like to see? Will Cas-Stunts and phpstunts still work? Will tools have been updated to handle non-standard tracks? I can't confidently answer yes to these questions.
A text file might have other issues, but I know that it will still be readable years in the future. In the meanwhile, I'd rather build a different system if I need to archive authorship information, rather than store them in a non-standard, unsafe way.
I didn't mean to sound harsh either. All I wanted to point out is that I wasn't asking or demanding anything, but instead, I was asking (as in questions) whether something could be done on your end, before just doing it the way I felt... and whether there was something you would like to use this feature for, besides the things I want to use it for. I do not expect you to change the way you do the things and while your way of handling files is strictly conservative, I don't consider that to be a bad thing, just like improving it isn't either. I will find my own best way of doing this so that it won't interfere with ZakStunts.
Now, to defend, I necessarily have to explain that your position is good, but subjective. I wouldn't like pipsqueaks to think that I'm doing something obscure, usafe, etc., so I will proceed to explain for them, in case they'd like to read:
Quoteobscurity: metadata is invisible except to Cas-Stunts, and possibly phpstunts/ZakStunts
Cas-Stunts metadata is
not obscure. I created this thread precisely in look for possibilities that Cas-Stunts be not the only one to see these data. Even if ZakStunts will not implement functions to access it, the format and contents will be public and anybody creating new tools will be able to access this information, so it is not invisible.
Quoteunsafe: other standard track editing software might lose the metadata, or be unable to open the file
Cas-Stunts metadata is
not unsafe. While saving a track with the internal editor will cause it to skip the meta-data, the information required to run the track will still be present, so there is no risk. Besides, the built-in editor is the only one to produce this behaviour and if that were unsafe, then illusion tracks would be unsafe as well and things like using paved roads on water would be as well, which is a common practice in ZakStunts. Nobody ever called TrackBlaster "unsafe" because of it allowing to produce these combinations and they are lost as well when a track is edited with the built-in editor.
Quotefrail: metadata can be lost following normal Stunts use, if you are not aware of it
Cas-Stunts metadata is
not frail, for the same reasons explained in the last point. Firstly, people are already used to the fact that they should not save a track with the built-in editor to avoid losing TrackBlaster changes. Secondly, there are many ways in which I can make tracks recover the meta-data is damaged that way and I plan to implement that.
Quoteimpractical: if aware of metadata, one needs to behave differently when handling the track, constantly mindful of the risk of losing metadata
Cas-Stunts metadata is
not impractical. Once again, you're repeating the same thing, which is incorrect. People can use Stunts comfortably with no fear of losing the information. Also, not doing something can't be the most "practical" way of doing it. I'm seeking for the best way to implement meta-data, not the only of not-implementing it.
Quotefragmenting: it creates different kinds of tracks, and splits tools between standard and cas-aware
Cas-Stunts metadata is
not fragmenting. I do not want to and I simply can't create a format that be incompatible with Stunts. If I do so, the tracks won't work in Stunts and there will be no point in doing it. Stunts will never be Cas-aware, so I realise whatever feature I add will have to keep compatibility with it. Track Blaster was also created with this in mind and, while it allows for creating things that can't be done with Stunts and that won't be preserved by the internal editor, IS compatible with Stunts. Cas-Stunts will be totally compatible in the exact same sense.
Quoteconfusing: non-programmer players unaware of the metadata are likely to encounter trouble, or at least be confused by the different file size
Cas-Stunts metadata is
not confusing. If you're a not programmer, odds are you don't even know what the size of a track file is. And if you're a programmer, it's obvious there are more data in the file. If you want to know what the information is, Cas-Stunts will provide the format spec with it. If you don't care, you can just ignore the additional information and it will work just as well. If a designer were to create a track and give it a name, author name, etc., and this information could not be stored or the file had be damaged for some very unlikely reason, Cas-Stunts will fix that for them. Besides, no software that we know of, by normal use, damages track files to the extent that they won't work in Stunts, which is the only official target. If Cas-Stunts or Track Blaster are not TRK format targets, neither is ZakStunts.
QuoteAnd at the moment, the only use case for my changes is "Zak would not have to type the title and author of the track every now and then."
Wrong. I really don't know why my proposal was taken so negatively. If you read the post carefully, without rejection in mind, you will see that I gave that as an example and I stated it could be used to store any information that would be useful to ZakStunts, championship oriented information, etc., such as recommended car, which season the track was raced on, etc. I'm sure you could think of uncountable pieces of data like this. But again, I didn't mean to force you to do anything. I just wanted to offer you a feature that we could cooperate on.
Quotebreaking standards is a big deal. I am also looking at this with an historical perspective: Stunts is 25 years old, and in 25 more years what would I like to see? Will Cas-Stunts and phpstunts still work? Will tools have been updated to handle non-standard tracks? I can't confidently answer yes to these questions.
Stunts track format is not a
standard. It is a file format that's interpreted by
one program. All other tools try to exploit this to their maximum. No matter what we do in our tools, Stunts won't change how it processes these files, yet, we've found ways to maximise the possibilities. Track Blaster is the best example of it and the improvements in that project were embrased. I am doing nothing different. The future of Stunts? If any of us had been asked about that in 1990 or say, 1992, we would have said Stunts had no future, but while we're a small community, it's 2016 and we're still here. I really can say nothing confidently about Stunts, its tools and users in the future.
Finally,So Dreadnaut, you're telling me I'm not Stunts and I admit it. But you're not Stunts either. We, somehow, are Stunts all together. I say, let's add, because if we're just looking at the "intersection" of our ideas, it will always be null. Our contributions to the game and the community are of entirely different types. We're not a group people that think the same way. We just have it in common that we like this game. If I want to paint it pink and you want to paint it black, let's do both.
Okay, I guess a
third opinion might be of some use here (there is also a modest proposal at the end):
Quote from: Cas on August 14, 2016, 09:08:19 PM
Quoteobscurity: metadata is invisible except to Cas-Stunts, and possibly phpstunts/ZakStunts
Cas-Stunts metadata is not obscure.
I am quite sure dreadnaut didn't mean "obscurity" in the underhanded, closed-source-and-undocumented sense. The stance involved here -- one that I must confess I sympathise with -- holds that
any binary metadata is obscure because it is not human-readable, and that, all things equal, text metadata is preferable. In our case, however, not all things are equal: using binary metadata in the way Cas proposes makes it possible to keep the metadata and the track as one piece, which, at least in principle, is a big gain in a world of messy Stunts folders. So, while binary obscurity might be a disadvantage, it is not a deal-breaker.
Quote from: Cas on August 14, 2016, 09:08:19 PM
Quoteunsafe: other standard track editing software might lose the metadata, or be unable to open the file
Cas-Stunts metadata is not unsafe.
Again, the unsafety involved here is not general unsafety but unsafety specifically regarding the preservation of the metadata. On the one hand, illusion tracks already suffer from a comparable kind of unsafety. On the other hand, given that the goal of adding metadata is long-term archival it might be argued that "unstable" archival metadata is a more significant concern than "unstable" track layouts. On the
third hand, however, I believe it is not unreasonable to expect that a track file set for long-term archival won't be willingly edited by users, and that, since none of the relevant track editors offers auto-saving, the risk of accidental overwriting is pretty low. If you buy that assumption, it becomes possible to think of Cas-Stunts as an external signing tool of sorts -- by adding metadata, the author would add something akin to a warranty seal to the track, which one might expect to be broken by any undue tampering (the analogy only goes so far, of course -- there probably won't be any actual security features in the metadata -- but that doesn't really matter here). That being so, I don't think this particular sort of unsafety is a concern.
(As I mentioned earlier, there is a related sort of metadata instability that concerns me: the fact that replay files won't carry the metadata, which is a not insignificant blow for long-term archival. Perhaps it is possible to add metadata to replay files, but I don't think that is a good idea -- I'd say that lies beyond the point of diminishing returns.)
Quote from: Cas on August 14, 2016, 09:08:19 PMSecondly, there are many ways in which I can make tracks recover the meta-data is damaged that way and I plan to implement that.
Perhaps I am missing something, but the only way of doing so without extra pieces of data would be somehow encoding the metadata into the 1802 bytes? If so, I don't think it is a good idea at all, as besides any potential gameplay effects it would break compatibility with all track viewers around.
Quote from: dreadnaut on August 14, 2016, 02:45:34 PM
- impractical: if aware of metadata, one needs to behave differently when handling the track, constantly mindful of the risk of losing metadata
My two cents as the author of Stunts Cartography: the function which reads track files in Stunts Cartography checks whether the file length is exactly 1802 bytes, as a handy sanity check that the file is (probably) a track. Ideally I would prefer to keep that as it is, for the sake of simplicity, but changing it to handle the Cas-Stunts format wouldn't be too big of big deal as long as the format is well-documented and includes an upper bound in file length.
There is definitely a cost to extending a format, even though I am personally willing to pay it. (In particular, authors of old tools may be unable to pay it. In our case, TrackBlaster, being the established DOS external editor, would be the main concern. I have just tested its behaviour with files with trailing bytes: beyond discarding the extra data, it refuses to load them through the F3 main menu command. You can, however, bypass the check by using the next/previous track shortcuts in editing mode -- the same trick we have to do to open custom cars in CarBlaster -- and for that reason I would say the results are tolerable.)
Quote from: Cas on August 14, 2016, 09:08:19 PM
Quotebreaking standards is a big deal. I am also looking at this with an historical perspective: Stunts is 25 years old, and in 25 more years what would I like to see? Will Cas-Stunts and phpstunts still work? Will tools have been updated to handle non-standard tracks? I can't confidently answer yes to these questions.
Stunts track format is not a standard. It is a file format that's interpreted by one program. All other tools try to exploit this to their maximum.
If you pressed me for a position on this issue, I would say the track format as seen in the game is a
de facto standard, with Stunts being its reference parser. That being so, however, one might legitimately ask whether "sharp corners cannot be placed on water terrain" is part of the standard. And if the answer is "no", one might wonder whether having a track file larger than 1802 bytes is any more significant as a violation... So I'd say this discussion is mainly about semantics. That doesn't, however, invalidate the concerns about keeping things simple and future-proof, which I share with dreadnaut to some extent. Nor does it mean that we, as a community, cannot agree on sensible ways to extend the formats.
All that said, here is my suggestion. It you two agreed on the specifications, Cas-Stunts might add support for an equivalent text metadata format alongside the in-TRK binary one, as well as a command for interconversion. That would make it possible to have a compatibility mode in Cas-Stunts, which would generate only plain TRK files and text metadata. (If Cas is willing to make a further concession, another possibility is giving TRK files with binary metadata a different extension, signalling they are primarily meant for archival and redistribution. I'm not sure how satisfactory this second compromise would be to all of you, though -- either way, it is not an essential part of my suggestion.)
Duplode, I appreciate your constructive criticism and will to provide solutions and not just argue.
I have been thinking about each of these things, as I'm not the type that just ignores everything that's not their own idea. Again, I want to emphasise that I consider Dreadnaut one of the greatest contributors to the community and my criticism on his words is also meant to be constructive. I won't ignore his point of view.
What I meant by "ways to prevent data loss" was, for example, to have a local (hard drive) registry of tracks meta-data. When one is found that's supposed to include it, but does not, the data will be recovered from the registry. It is indeed possible to use the 1802 bytes, but the risk of incompatibility with other tools is too high, even though that risk does not exist with Stunts itself.
In the "overlay vs. companion file" discussion, I believe the companion file, while not what I like the most, is a viable option that I'm willing to use in order to achieve maximum compatibility. Besides, I'm still studying the effects of overlays and wouldn't implement them until I've verified risk is zero. More on that research, we may talk about another time.
In the "binary vs. text" discussion, there is a good reason why I defend binary. This data block is to be parsed by tools, not to be edited by users, which could break it. Having them as binary ensures users will only edit this while editing the track, not separately by accident or trying to cheat, for example. Parsing binary files is easier than parsing text. The only benefit of text is that of easy editing from any application. Something I wouldn't want. I want the application to know Stunts. If it's a standard text-editor, I prefer that it be unable to edit the meta-data. Does that make sense?
I agree to develop a format in which the meta-data will accompany the track in another file as long as this companion file not be discarded. Say, if a pipsqueak submits a track file in a ZIP with the companion file, that file should be distributed together with the track. I can make the format "visually" text, so that it can be viewed, but I prefer not to make it editable by non Stunts-oriented applications. And of course, as always, I would never make Cas-Stunts save files only in a format that would break other tools. I'll publish a couple of possible formats soon.
Quote from: Cas on August 15, 2016, 04:43:37 AM
What I meant by "ways to prevent data loss" was, for example, to have a local (hard drive) registry of tracks meta-data. When one is found that's supposed to include it, but does not, the data will be recovered from the registry.
Oh, that's entirely fine of course. I suspect there might be a fair amount of overlap with the track.stunts.hu idea dreadnaut mentioned above...
Quote from: Cas on August 15, 2016, 04:43:37 AM
In the "binary vs. text" discussion, there is a good reason why I defend binary. This data block is to be parsed by tools, not to be edited by users, which could break it. Having them as binary ensures users will only edit this while editing the track, not separately by accident or trying to cheat, for example. Parsing binary files is easier than parsing text. The only benefit of text is that of easy editing from any application. Something I wouldn't want. I want the application to know Stunts. If it's a standard text-editor, I prefer that it be unable to edit the meta-data. Does that make sense?
It make sense, even though I'm not sure I agree, at least from an end-user perspective (the safety benefits, in particular, are quite marginal IMO). In any case, as I implied above I feel binary-vs-text is a secondary discussion, one that is not as important as combined-vs-separate one.
Quote from: Cas on August 15, 2016, 04:43:37 AM
I agree to develop a format in which the meta-data will accompany the track in another file as long as this companion file not be discarded. [...] I'll publish a couple of possible formats soon.
Seems promising :)
And I'm back, sorry for the delay!
Also apologies for sounding very negative and censoring toward Cas' idea. I do think it's not a great solution, but maybe I'm just not thinking about the right problem, stuck instead on my own idea of an requirement to which postfix metadata are not a solution.
So I'll go a step down: what is the problem we are trying to solve?
Thank you for the positive attitude, Dreadnaut. I didn't mean to be pushy with my idea either and I'm sorry if I didn't analyse the option of text format any further.
What was my concern when I first posted was that I wanted to somehow add the ability to add a track title, author, etc., and an overlay seemed to me like the best solution, but then I remembered that had caused a conflict with ZakStunts in the past, so I wanted to agree on something. Maybe you would want to change something or actually use the new "feature" or maybe you would say you wouldn't change anything, but I could have an idea on the best way to proceed so that it would not interfere with ZakStunts. This was the issue.
At this time, I have already implemented this feature and it's working, although I have not published it. Currently, Cas-Stunts is able to save track meta-data either as an overlay or as a separate file. I created a very simple format for it, which is binary, but we could agree on a text format, that is, if ZakStunts will use this info. If ZakStunts will not use the meta-data, my humble opinion is that it's better to just leave it binary, because it's simpler to parse by tools, but I am willing to read constructive criticism :P
I realise ZakStunts has been standing for a very long time and of all that, this is probably the least active moment and that you guys are tired of keeping it up for so many years and it's not comfortable to have to change anything at this stage. I don't want to be extra weight, but a working hand here.
I finished the changes I was making and Cas-Stunts 2.3.3 beta is published. Now I post here the spec for the format the editor currently uses for meta-data. It's designed to be very simple and the same format is used when the track is saved in a single file or as two files.
Details:
- First 1802 bytes of tracks are what we are used to always, in all formats, except maybe for the last byte
- Last byte of the 1802 has been used by Cas-Stunts as "format byte" since the beginning. Stunts and TrackBlaster would leave it as a zero
- Cas-Stunts 2.0 through 2.3.2 (the ones I've published here) all would set the format byte to 150, but keep its value if a track is loaded and has a different one
- Cas-Stunts 2.3.3 introduces metadata. The format byte is kept, like before, unless the user forces to change the format. A format byte 151 indicates an overlay contains meta-data, whereas a 152 means there's a file called <trackname>.smd containing the meta-data
- Meta-data, whether as an overlay or as a separate file, is expected to start with a "magic number" containing the string "smdf". Other meta-data formats could be implemented and use a different magic number. Currently, Cas-Stunts will skip meta-data in other formats
- If the meta-data is "smdf", a 32bit hash follows that is calculated from the first 1802 bytes in the track. Cas-Stunts will not load the meta-data if the track contents do not match this value. Yet, the data can be read or modified externally without reading or changing this value
- Following is a series of fields. Each has a four-character name and then a word indicating the length of the field
- Titl contains the track title
- Autr contains the author's name
- Comm contains a comment
- Chmp contains championship/tournament information
- Date contains the creation date. A word for the year, a byte for the month and another byte for the day
- Tool contains the name of the tool with which it was created followed by its version. Version is an dword: 20303 stands for 2.3.3, 11300 is 1.13
- Etim contains the editing time. It's a dword with the number of seconds
- If any field is omitted, its value is assumed unknown or empty. Cas-Stunts does not create the fields if they are null
- Currently, Cas-Stunts can only enter ASCII, but will keep non-ASCII (UTF/extended) values
This is the current approach. It can be improved or another format can be implemented specially for ZakStunts if necessary. If text is used, producing the file is simple, but parsing it becomes very complicated with non-text fields, such as the date and the tool version number, because bounds have to be checked and delimiters have to be located. It could make it work as an export format very quickly, though.
If necessary, I will publish the algorithm used to produce the hash code. It's very simple, but proved effective in my tests.
A few technical questions about the spec (for the moment I won't speculate much about interoperability or alternative export formats -- dreadnaut can comment on that with more propriety than I do):
- Would it be possible to leave the format byte as zero, at least when there is no metadata in the TRK? From a third-party point of view, modifying it in a 1802-byte TRK introduces a difference that carries little meaning but would show up e.g. in any hash of the track file.
- Does the 32-bit hash use the format byte? At first I found it a little surprising that the same track would give different hashes depending on how its metadata was saved, but then I realised you might have intended that as a further sanity check.
- Are there size limits to the variable-length text fields? In principle, I suppose there should be a maximum metadata size for safety reasons, from which maximum field sizes would follow to stop users from (trying to) enter too much text.
Quote from: Cas on August 27, 2016, 08:24:59 AM
If necessary, I will publish the algorithm used to produce the hash code. It's very simple, but proved effective in my tests.
That would be useful indeed (for instance, it will be necessary in the future if anyone else wants to generate Cas-Stunts-compatible metadata through another tool).
My two cents on the previous discussion:
Quote from: dreadnaut on August 24, 2016, 12:20:56 AM
what is the problem we are trying to solve?
Preserving the key historical facts about a track so that they do not get lost easily as the track is moved around or redistributed. Two approaches to that are appending the information to the TRK (in a documented binary format) and maintaining a centralised online track database (indexed by track hashes of some sort, and probably accompanied by a text exchange format for the metadata). On the one hand, the first approach allows authors can use it independently of ZakStunts or wherever the database is kept. On the other hand, the second approach makes it unnecessary to modify the TRK files (and to use specialised client-side tools to do so). Nothing stops us from using both approaches at once, though ideally that would be done in a coordinated way so that all implementations are as compatible with each other as feasible.
P.S.:
Quote from: Cas on August 26, 2016, 03:35:16 AM
this is probably the least active moment
I'd say Summer 2011 was definitely worse, even if forum activity has regressed to what it was during that season.
I'll go in order:
- Yes. If the track was created originally with the format byte zero (i.e: with another editor), Cas-Stunts will keep this format-byte as default, unless the user forces to change it at run-time by changing the saving format. Of course, Cas-Stunts won't look for meta-data if this byte is zero, but I have an idea for the future about that** On the other hand, if a track is originally created with Cas-Stunts, the format-byte will not be zero, but it doesn't matter because whatever value it is, it will be kept. In particular, if the track is submitted to ZakStunts, I assume it will be in split format, so this value will always be 152 (constant) and therefore, not create problems with the hash.
- Yes. It reads the whole first 1802 bytes as a string. I thought about skipping the format-byte, but then thought it makes more sense to include it. I can change that if you think it's better, though. This is still a beta version.
- Cas-Stunts will currently not allow to enter more than 64 characters for each of those strings. Yet, if some other program generates the meta-data and creates a longer string, Cas-Stunts will still succeed to read the strings to a maximum size of 32767 bytes (because of the signed short value, which is signed in case I want to include some form of error detection or anything like that). Anyway, the whole meta-data should never exceed 12 kilobytes, because when saved as an overlay, that would make Stunts overwrite past the replay area in memory. When loading the strings, Cas-Stunts will truncate them to the first 64 characters currently.
- I'll post the hash function below
- I'd had a similar idea to the centralised ZakStunts registry. I believe I could make Cas-Stunts access the ZakStunts database in the future if that is implemented, but I also think I should not rely solely on that. I mean, what if the site one day goes online? All data will be lost. What if the IP changes for some reason? Cas-Stunts will have to be updated. What if I'm not here? Somebody else will have to delve in my code to fix it. (**)I have the idea of including an offline registry with Cas-Stunts that would contain meta-data for the built-in tracks and could also include that for ZakStunts tracks. If ZakStunts were not available or there were no internet, then Cas-Stunts could fall back to the offline registry.
The Hash function I'm using is this:
'Generate a 32bit hash value out of the given string
Function Hash32(content As String) As ULong
Dim i As Short, hash As ULong
For i = 1 To Len(content)
hash Or= 1
hash *= (i + ASC(Mid(content, i, 1)))
Next i
Return hash
End FunctionIn other words, we start with a hash=0. For each of the 1802 iterations, the hash is made odd (to avoid accumulation of trailing zeroes), then multiplied by the (one-based) position plus the ASCII value of the byte at that position. This is the simplest function I could think of that has these properties:
- It will quickly fill and overflow the 32 bits, making the hash more random
- Adding a constant to one byte and substracting it from another or similar actions will not generate the same hash
- Swapping two bytes will not generate the same hash
- No particular value (zero, for instance) will cause the procedure to converge to a constant (usually zero)
- Despite the repeated mutiplications, factors will not accumulate (especially 2, since we're using binary), for the hash is forced odd at each iteration
If you find a problem with this function, I really would like to know it :P
Thank you for the clarifications. Right now only one of those issues concerns me to any extent: the format byte. To present a better case, I tried to think of concrete scenarios where it might lead to complications. I think might have found a not too contrived one, and I wonder how do you see it.
Let's say I want to archive metadata for the tracks of my old Southern Cross competition. Suppose that there are two popular ways of long-term metadata archival: Cas-Stunts and an online ZakStunts database, and that I want to use both, one serving as a fallback for the other. The ZakStunts database would be indexed by a 1802-byte hash, and it wouldn't read the Cas-Stunts binary metadata (at least not directly -- there might be a shared export format, etc.). Now, a natural thing to do would be using Cas-Stunts to enter the metadata and save it in the split format. If I do so, however, the format byte will be changed. Why does that matter? Because if I then add the track to the online database it will go under a different hash than the original TRK, which was available from the competition site up to that point. A pipsqueak who happens to have the old file and tries to check it against the database will get a false negative. Furthermore, I can't even circumvent the issue by writing a tool that creates standalone SMD files, as Cas-Stunts will not load them without the format byte change. If the format byte is not changed, however, there is no possible mismatch. While this issue is more obviously relevant to the split format, I believe it also applies, to a lesser extent, to the combined format as well. Suppose I pick an old, add combined metadata to it, drive a lap on the modified track and extract a TRK from the replay. If the format byte is left unchanged, the TRK at the end of the process would be identical to the original one, which is a small but IMO significant advantage.
In a nutshell, then, I believe the format byte should never be changed by Cas-Stunts. It seems to me a small price to pay for eliminating any potential incompatibilities with other tools as far as the 1802 bytes are concerned -- even with tools unaware of Cas-Stunts, and even when dealing with tracks older than Cas-Stunts itself.
As I said in the beginning, I don't think the other points they bring any more issues. The 32-bit hash question will become a moot point if you accept my main suggestion, but in any case that hash is meant for Cas-Stunts consumption, and other tools won't have to deal with it except when creating or validating metadata in the Cas-Stunts format, so it shouldn't really concern anyone. As for the field length, I first thought of 256 as a generous value (more than three 80-char lines of text!) that would be enough for even quite extended comments. A maximum length of 256 for all current variable length fields would amount to (if I counted it right) a maximum block size of 1086 bytes, almost 10% of the hard limit. It may sound like nothing, but who knows how many extra fields might be added in the future... 64 might be a fine value too, or perhaps there might be different limits for each field.
I had written a very long message. I'm editing it now, to make it more concise. I think your example is very good and we should continue to use these thought experiments to explain our positions. I already have an idea of a solution. It'd be good to chat and be able to discuss with more clarity, but I'll state my thinking here first:
- I think the format byte can easily be left as zero if the track is saved in "combined" mode, since the file length shows it's clear there's meta-data. For split files, I think leaving it at zero will force the editor to always look up in the data-bases and if the SMD file is lost, which is something very likely to happen (same as when you forget to copy a HIG file), you will not know there was meta-data. Not a good thing. This is my thought experiment.
- In the case you describe, you're talking about an old track that has been raced before, probably never going to be modified again, with a name that won't change. For tracks like this and for the tracks at ZakStunts, it makes sense that meta-data be stored in a read-only registry that could be online and/or distributed with a tool like Cas-Stunts, for offline access. For new tracks, I think it's a lot more sensible to keep the info in the most comfortable format for the editor and for passing it without data loss. The best format that serves this purpose is the combined one. It is the safest. This is an important point I've been thinking of. We must admit that the format Dreadnaut proposes is indeed the safest for ZakStunts and archival, but also the format I encourage (combined) is clearly the safest for editing and passing track files.
- I believe we must seek a solution not by trying to change each other's original concepts, but by trying to make these two concepts compatible. Tracks to be edited, kept and passed in combined format until they are ready to be used, then submitted to ZakStunts registry (and possibly tournament) in the format Dreadnaut considers best. Once a file is not to be modified anymore, meta-data will be read from the registry. Before that, it will be read from the overlay. This is an idea. What do you think? I would like to discuss this carefully in a chat.
- Also, if ZakStunts will implement a registry system where everybody can submit tracks, it would be great if I could integrate Cas-Stunts to it, for example, making it possible to submit tracks from within Cas-Stunts to the website or to download them and their meta-data from there. Cas-Stunts would send the data in ZakStunts format and on retrieval would store it back in Cas-Stunts format. For fixed files (no longer to be changed), the files would still be 1802 bytes in the hard drive.
- About field length. I agree. 256 bytes would be optimal. I used 64 because it was easy to do at the moment. Can change it easily.
Quote from: Cas on August 29, 2016, 05:28:29 AM
For split files, I think leaving it at zero will force the editor to always look up in the data-bases and if the SMD file is lost, which is something very likely to happen (same as when you forget to copy a HIG file), you will not know there was meta-data.
That is one objection I hadn't thought of. It is certainly worth considering, and after pondering about a bit I still haven't a settled opinion about it yet...
Quote from: Cas on August 29, 2016, 05:28:29 AM
It'd be good to chat and be able to discuss with more clarity
It would indeed -- this discussion is growing faster than linearly already :) I believe our forum chat room -- http://forum.stunts.hu/chat/ -- would be a convenient meeting place. I will be online there from around 11 PM onward.
Very good chat :) There I am
I'm there now (a little late, sorry).
Ah, the headache. As we knew, Stunts does not store Bliss metadata in the replay, but it turns out it does store the value of the reserved byte. I made the mistake to be "open" with the code, and simply ignore that byte in the tracks, and only reject tracks above the standard size.
Now, I've spend the last three hours trying to clean up the mess: I have tracks ending in 150, 151, 152, and... 2 and 3, which should be allowed values. But they all come from Bliss: I made ZCT208 Invaders with the editor, told it to save without metadata, and it's the one ending with 3.
That would be easy to clear: zero the last byte, update the file hashes... but! For the same track, I have replays containing the same track data, but with a different reserved bytes. So I also have to scrub all the stored replays, zeroing the last byte of the track, etc etc. That's going to be at least another hour.
All this brings me back to non-standard standards, the in-track metadata, and the discussion in this thread :(
It's a Bad Idea™, can we get rid of it?
I can shine a little light on the issue....
First, for history, on how these extra bytes ended that way. A long time ago, in the first version of the editor which nobody ever had here, I wanted a way that a track made with it could be recognised. I saw that any trailing bytes were lost when the track was edited with the internal editor, but not the 1802nd byte, so I decided I'd set that byte to 149. You won't see any track file ending in this number because this editor was never used by the community. When I made the second version of the editor, that became public, I changed that to 150, to make a difference. So far, so good. This one shouldn't be an issue, but later on, I made a mistake.
At some point, we realised there was a conflict between trailing data in the track file and ZakStunts, besides other programs that would fail to run properly unless the file was exactly 1802 bytes long. Because of this, I created a second format, "split", so that one could save in either format depending on convenience. And here I made a terrible mistake. I thought it was important for each format to have a different final byte. Not only that, but also, I thought 150 should be left behind as an old format. So format byte 151 would come to overlayed files, whereas format byte 152 would go for split files. Some time later, I realised this was a mistake, because now one same track could be saved with different format bytes, so the 1802 bytes were not guaranteed to be the same. What I did was fix this to a single byte value, but, this was to be yet a new one, 153. Right now, Bliss only produces files ending in 153.
How to solve this
For tracks that have extra bytes 0, 150 or 153, there is actually no problem, even though we'd all like all those numbers to be the same. There is no problem because the extra byte is passed to replays and there exist only one of these values for the same track. The internal editor and Track Blaster both keep the extra byte value. But, bytes 151 and 152 are a problem. My opinion is that we can't just change them all to zero and it'll be fine, because if we later encounter a replay made with the other version, we'd have a new problem. From my point of view, the very simplest way of solving this is the following:
1 - Upon encountering track files ending in 0, 150 or 153, leave them as they are. Changing the byte would create a new version of the same file, with another trailing byte. I admit it'd be better if they were all the same, but if we change them now, we'd make it worse.
2 - When a track file comes up with bytes 151 or 152, no matter what we do, we'll still have a problem. Fortunately, they are not many. So if you like this number to be 0, change it. You're not making things any worse. Yet, this is not enough to solve the problem. Number 3 is the fundamental 1:
3 - When comparing tracks or a track and a replay, compare only the first 1801 bytes, not the whole 1802 typical track length. This single action solves the whole issue and it makes the greatest sense. In a way, the whole problem is never a problem as long as we understand tracks being made of 1801 bytes. Byte 1802 never has had any impact on what the track is. Stunts ignores this byte. So why go crazy about it? I don't mean to argue. Just think about it for a moment. No solution is simpler than that. Of course, hashes should also be made about the 1801 first bytes.
4 - As for me, Bliss currently creates tracks only with final byte 153. This should not cause any problem, but if you wish, I can make the next version use byte 0 instead. Tracks opened with Bliss and then saved again currently keep the same byte. I don't remember how obsolete values 151 and 152 are saved after edition on the new editor, but I can make sure all track files in the new version are saved with last byte 0 if they have a last byte 151 or 152, and new ones always. This should help for the future. I offer an apology about this mistake, but I request to not be blamed on more than I actually did.
You already know my philosophical view of the matter: there is no "standard" here; there never was. Only changes that cause effects and changes that don't. Track Blaster introduced a great deal of changes that weren't previously allowed and nobody came screaming about standards violations. Quite the contrary, people were happy to embrace "illusion tracks" and road on water, even though they did conflict with the internal editor when one would accidentally open the file with it. What I've done is just changing a single byte that does nothing. While I did make the mistake I described above, I've created no mess. If you just blame me, we'll just keep on exchanging messages like this and it will be useless.
So... do we proceed like that? Any other ideas? What can I do to help?
As grumpy as my previous message could be, you should notice that I did not blame you, I described the problems I encountered. Note that the last byte of ZCT208 is actually 3, not 153, so I'm not sure what happened there.
I am not going to redefine the hash of a track as the hash on the first 1801 bytes, because that's not what the hash of a file is, and doing so would confuse anyone trying to compare tracks and unaware of the special definition.
What I'm going to do remove the lenient check and just zero the last byte of every track uploaded on ZakStunts. I'll probably have to do the same for replays as well. Which means that instead of calling sha1_file on the file and be done, I have to load the replay, check the Stunts version, zero the correct byte, calculate the hash on the data.
I would ask you to update Bliss to set the last byte to zero, all the time. If you want to append metadata, you can still do it: have the word at the 1803rd position be a magic number that you can use to recognise the record, followed by the same data as before.
Scratch that, I'll do the opposite and take the concept of reserved byte as strictly as possible. Software does not need to know what it is for, it just needs to persist the value, as Stunts does.
Make sure that Bliss writes one specific value there when you create a new track, if you must, and we're good. However, double check the code: ZCT208 ends with 3, while ZCT200 and ZCT201 end with 2. All the other tracks in ZakStunts end with 150, 151, or 152, which means people need to update to the latest version.
P.S. —In the end I would still feel more comfortable if that value were zero. Setting a non-zero value creates a precedent, and someone might write a program that changes that number to something else. For example, an automatic track downloader that stamps its tracks with 99, because hey, there's a spare byte, and there's no standard. And then we would have different versions of the same track, for no good reason.
P.P.S —apologies, there seems to be some extra bitterness above, but it's been a long day; thanks for not raising the stakes and keeping the conversation on thinking about solutions!
Sorry, mate. I read this thread and I was about to leave home at that time, so I replied with all the heat in my head. When I cooled down, I wasn't home to change anything. I'm a little susceptible to bitterness.
I had noticed before about some files being saved with last byte 2 or 3. First time, I thought somebody had changed this value for some reason. Even Bliss wouldn't recognise it. Then I found a bug in Bliss and fixed it. I think most pipsqueaks currently have a version new enough so that that doesn't happen anymore. It's bug. I never meant for those values to be output.
Alright, next version, I will switch this byte to zero, but... I will make Bliss not change the last byte on preexisting track files. Otherwise, I'd be making matters worse. In th meantime, with the current version, as I said, all new tracks should only have the value 153, so that shouldn't be a problem.
A rational reply to your slightly bitter "P.S.". Yes, you're right, but that supposed person creating that supposed tool wouldn't be doing what I did. They would be "modifying" a previous value. Bliss doesn't do that. It just uses a value for new tracks that's not the same as that of tracks created with the internal editor. In short, I do understand what you hate about this byte being changed. I am just not doing it.
And about the hash. I guess you're using a function to which you pass a whole file and it returns hash. In that case, it could be complicated to go around the last byte. I was imagining something more like I did in Bliss (I wrote my own hash function and tested it to have balanced sensitivity). Well, I guess number 1 priority is to make sure the value is fixed. That, we already have with the current version of Bliss. Even though I, personally, see no problem in this value having a value other than zero as long as it is fixed, I will change it to zero as I described, on future versions.
Quote from: Cas on February 24, 2019, 08:33:31 PMA rational reply to your slightly bitter "P.S.". Yes, you're right, but that supposed person creating that supposed tool wouldn't be doing what I did. They would be "modifying" a previous value. Bliss doesn't do that. It just uses a value for new tracks that's not the same as that of tracks created with the internal editor. In short, I do understand what you hate about this byte being changed. I am just not doing it.
Yes, when I realized that with Bliss you are creating new tracks, that makes more sense. Which is way you could leave 153 as a marker. As you said, though, when a tool reads an existing track it should
not change the last byte if they save it again.
I'll double check to make sure that ZakStunts follows that rule, and leave all the tracks and replays as they are, because they match the tracks that all the pipsqueaks have on their computers. And I'll go and download the most recent Bliss :)
And sorry again for the grumpy text above :|
Don't worry, mate. It's something we have in common.