Car repository indices (CRIs)

[Cas]

As I mentioned in the thread about Pretty Garage, I'm ready to develop the part that interacts with repositories. This is not something only for the Car Archive or sites like Mark's Stunts Custom Cars or Southern Cross, but it can also be very useful for tournaments. For example, Daniel3D has expressed his interest in implementing this for CCC and ZakStunts, which every season has a set of allowed cars, could provide a list of allowed cars that way. In R4K, it depends on the race, but there could be a list of cars that can be downloaded there too, although it wouldn't be as important.

I call these lists CRIs because that's what they are, car repository indices. The format could be any we agree on or there could be several, as Dreadnaut said they could in the other thread. What's important is what information at least has to be present in the lists. For my garage program and probably, for any garage program, what seems important to have is a list of the IDs of the cars with their names and the list of the file names each car includes. This latter could be the car files themselves or a zip file containing them, but the program interfacing with it should know which file or files can be downloaded. There could be more info, but it's not necessary.

I can make my program able to read more than one format, but I'd like to agree on at least one to know it'll help. Dreadnaut gave an example of a YAML file for an individual car configuration, with much more information. I wonder if it'd be good to have a file that just contains the whole config for all cars in blocks, or to separate these two things, have the individual configurations as one file for each car and have the car list only have the basic info. I think that'd be better, but I'm open to ideas. If the list is YAML too, it could be as simple as something like this:

Code Select Expand

---
car:
 id: PMIN
 name: Porsche March Indy
 files:
  - CARPMIN.RES
  - STPMIN.P3S
  - STDAPMIN.PVS
  - STDBPMIN.PVS
car:
 id: CDOR
 name: Melange XGT-88
 files:
  - CARCDOR.RES
  - STCDOR.P3S
  - STDACDOR.PVS
  - STDBCDOR.PVS
.
.
.
I would be writing a very simple parser that only reads this, of course, so if somebody would add some extra YAML thing, such as a multi-line or quotes, or a car has "file: PMIN.ZIP" instead of "files:" and a list, then it'd break. I don't want to write more code than necessary, so that's what I would like to agree on these details. What do you guys think?

[dreadnaut]

My suggestion would be to separate the list of cars from the cars files: competitions publish a basic list (e.g., CDOR, PMIN, PCAR, ZF40), and a tool can retrieve the needed cars from its favorite archive front-end.

This way there is only one place that needs to know about which files are needed for a car. And we avoid the risk of two sources defining different files for the same car.

ZakStunts already publishes the list, at least in one format:

- https://zak.stunts.hu/api/seasons/2024
- https://zak.stunts.hu/api/seasons/2024/cars

In the future, I would probably replace the car details with links to the archive 🤔

[Duplode]

My suggestion would be to separate the list of cars from the cars files: competitions publish a basic list (e.g., CDOR, PMIN, PCAR, ZF40), and a tool can retrieve the needed cars from its favorite archive front-end.

Delegating the retrieval of the files to an instance of the Car Archive (I'll use capitals when talking about the new project) does sound like a good design, and this kind of triangulation in which e.g. Pretty Garage grabs an ID list from CCC and then fetches the car files through the Car Archive API would be nifty.

If I'm reading it right, though, Cas' concept looks somewhat different: a list which could be made available from any site (even a static one like Southern Cross) so that a tool like Pretty Garage picks it up and figures out what it has to download. That would require knowing from the list file, or assuming (by convention?), facts about how the car files are laid out -- e.g. that in Southern Cross for the moment they are all under https://scr.stunts.hu/files/mods/ and packaged as one zip file for each car (except for the DTM cars, which currently are in a single zip) with no directory structure within.

This CRI design, with a car list file pointing directly to the files, could be a way to get integration with something like Southern Cross, which wasn't originally meant as a machine-readable archive. It feels like a less natural fit for the Car Archive. Doing as much as possible through the Car Archive API instead sounds like a good deal both for the Car Archive itself (avoiding duplicated information and abstraction breakage) and for Pretty Garage (being able to use all info the API will offer, rather than just names and file paths).

[Cas]

Uhm... I think these are two different things. That is, I agree with the idea and it's good, only I was trying to refer to a different aspect of it. Let me explain...

The Car Archive is meant to be a centralised project, which makes sense. The idea is that there's an official place where you can find a car so that you know that that one is the "official" one and that all cars you look for will be at that spot, but as I see it, this centralisation has more to do with the information the car represents than with the files that make up the cars, even though, of course, those will also be available there. In other words, it's the identity of the cars that's protected by this centralisation. Do I see it correctly?

Now, as regards availability, it's best to have as many sources as possible. Just as identification works better centralised, mirroring strengthens availability. It is because of this that we've been gradually developing several car sources. So, given that I see that there exists Mark's site and also Southern Cross and CCC has cars and ZakStunts already has cars too, etc., it's those sources that I've been seeing as "repositories". The Car Archive is a much more ambitious project with a lot more to do than what a simple repository does. It can, of course, also work as a repository, but if I've understood it well, that's not the main idea.

CRIs are descriptors for repositories. One for each. The idea of having them contain this info is that one can just curl them down from a web address, no need to send any special information, and then, once the list is down, there's no need to keep on interacting with the website unless one actually wants to download a car. This minimises information exchange. Besides, the CRIs can be cachéd and updated every now and then (at the client side). So anybody can go up and set up their own repository freely.

As regards which files, it could be all zipped or separate. If zipped, the garage program would realise and unzip the file after downloading it. It may be better to not have several cars zipped in the same file, but if that were the case, we can still refer to the same zip file for all cars involved. The important thing is that the program knows which file has to be downloaded and from where. Downloading a file this way would also be a simple curl. No need to exchange packages of specialised data.

The Car Archive API is very useful if you want to retrieve information from cars and I suppose, give it a filter and have it return a result, complex things like that. I could implement that too, for example, if there's an option in Pretty Garage to find all details about a car you have or to obtain the latest official version, etc. But that's a different thing. It's two different features.

[dreadnaut]

The Car Archive is meant to work like a wiki, where anyone can go and add cars. If you know that car CDOR exists, you can look for the latest version at https://stunts.hu/cars/CDOR.zip.

That is meant to avoid multiple people having to collect files, make sure they have the latest version, zip them up in a consistent format, keep their page up to date. Anyone can still build a website with their favourite list of cars, but they can defer the boring work to the archive.

In this model, all that a website, competition, etc... needs to publish is a list of card identifiers.

[Cas]

Good, then yes, we're approximately on the same page
I've been thinking about this and looking at the two links from the ZakStunts API. The second one, I realised, would be a CRI except it doesn't include links to car files, but then I thought... sometimes it's useful to be able to download a list even if the list doesn't provide such links. It'd be like creating a group in the garage, but instead of having to do it manually, you query online and so you can automatically generate a group with the cars used in ZakStunts this season or in CCC for this race, or the cars made by Alan Rotoi, or something like that. Since you say you'll probably change this format once the Archive is up, I'll leave ZakStunts for later, but generally, what I'd be interested is in something like https://zak.stunts.hu/api/seasons/2024/cars. It doesn't matter if you can download cars from it or not and if you can, it doesn't matter if they're stored locally or somewhere else.

When the Car Archive is ready, there could additionally be something to allow obtaining cars in general. There, the list would be less important, because it'd be just the "list of all cars".

[dreadnaut]

Good, then yes, we're approximately on the same page

I am uncertain, I wonder if I managed to be too vague   I'll go direct, apologies in advance if I will sound blunt:

I think the "CRI" as described above are a bad idea which creates duplicate work for anyone interested in "listing cars" for whatever reason, and ambiguity by spreading car files around — which was exactly what we wanted to avoid by having a single car archive.

[Cas]

Yeah, I got that feeling from our previous discussion in the previous posts of this thread, but I still think we're on the same page on what the Car Archive intends to do.

Well, to clarify on my end, I don't intend to duplicate anything. I am aware that repositories already exist and have existed for a very long time. I only want my garage program to be able to access them as well as the Archive and I would like that, if anybody happens to have an repository for any reason (for example, it'd make sense for a creator to want to have their own car repository with their own cars even in spite of the existence of a centralised archive), they can easily gain access to their own files through Pretty Garage. This should not interfere in any way with the Archive. That's to explain the idea I originally came with and I still have.

Now, after our conversation here, I have taken what you said into consideration and developed a different idea (in addition, not to replace the other one), that is that, sometimes somebody will want to be able to retrieve a list of cars, but not a repository. For example, currently, ZakStunts does not provide cars, but it does provide a list of allowed cars for each season. This may change with the Archive, but it's still something that may happen in other occasions. So the garage should be able to retrieve the list of cars from a "CRI" like the one there, but it'd be a group, not a repository. The user would have to have the cars locally. It'd nevertheless be useful, because it'd help them quickly configure their Stunts to have ZakStunts-allowed cars at hand.

I apologise for making it all too long. It's always very hard for me to summarise my ideas, especially when I'm trying to prevent clashes and conflicts. As you already know from me, I like to create and I usually have no problem to retouch, but I don't like abandoning productive ideas. I try to count to ten before replying because I tend to be more clashy otherwise. I think we both know this feeling.

[Daniel3D]

So .. if I understand correctly...
The question is, can we create a communication standard so a frontend program like pretty garage can communicate with a backend car database.

Preferably only one backend, car archive, but mark will probably keep his up as he should.

Mark can flag his cars as bug fixed, since he only fixes graphics related issues.

A frontend could inform the user that a alternative download is available (if not yet updated by the main archive) to name a sensible use case..

[Duplode]

Preferably only one backend, car archive, but mark will probably keep his up as he should.

Mark can flag his cars as bug fixed, since he only fixes graphics related issues.

There's a distinction here that I think we shouldn't lose track of. Mark's cars page is not just a collection of cars, but also a primary source for his bugfix releases. Other examples of primary sources include Alan's Google Drive, Ryoma's Mega, and even the threads in the Custom Cars board here. Primary sources will keep existing independently, because authors will share their new creations however they find most convenient, and we can't really demand them to do that through the Car Archive, or some other broader repository scheme.

In what seems to me as the most likely scenario, authors will release their cars in the usual ways (that won't necessarily be integrated into any database), and then the new car (or version) will be catalogued and added to the Car Archive (either by the author themselves or by someone else) in a separate step. On the one hand, independent primary sources will continue to exist, and we'll keep an eye on them for new stuff; on the other hand, when it comes to telling people where to find cars we'll point to the Car Archive, a consolidated, tidied up and convenient secondary source.

(A different approach to this distinction that, in theory, could be supported by the project, would be for authors to run the Car Archive software on their sites with a different repository of cars that provides the latest versions of their cars. Clients could then retrieve cars from different source repositories, kinda like how it's done for Linux distribution package repos. I don't actually see that as the main way new releases will be done in the future, though -- in particular, uptake is highly unlikely to be universal.)

[Daniel3D]

On the one hand, independent primary sources will continue to exist, and we'll keep an eye on them for new stuff; on the other hand, when it comes to telling people where to find cars we'll point to the Car Archive, a consolidated, tidied up and convenient secondary source.

That is what I meant. And i think cas as well.
There is the main source for cars in the archive. But other sources should not be prohibited. It's up to the user if he wants to use them. Like you can make Android accept apps from other sources than the playstore.
A standard for presenting the cars (API?)
Would allow for new cars to quickly reach the user.

If it needs to match the level of detail the repository has, I don't know.
Maybe pretty garage can automatically generate the ini based on the source (for instance ryoma) or combine it with archive information (in case of and updated version of Mark)

[Duplode]

I am aware that repositories already exist and have existed for a very long time. I only want my garage program to be able to access them as well as the Archive and I would like that, if anybody happens to have an repository for any reason (for example, it'd make sense for a creator to want to have their own car repository with their own cars even in spite of the existence of a centralised archive), they can easily gain access to their own files through Pretty Garage.

If it needs to match the level of detail the repository has, I don't know.
Maybe pretty garage can automatically generate the ini based on the source (for instance ryoma) or combine it with archive information (in case of and updated version of Mark)

Here, I think, lies the core issue. In abstract, we can think of something like Cas' initial design for CRIs as a way to make the downloads in any Archive-unaware site (for instance, Southern Cross) accessible to tools like Pretty Garage. @dreadnaut's point, however -- and I tend to agree with him -- is that introducing a second repository format at this stage would put at risk some of the advantages in having an unified archive (a crucial one, for me, being clarity about car versions). It would also disperse efforts when we'd really want to concentrate them instead. (Ergo my insistence in that the Southern Cross collection will be superseded by the Car Archive once it becomes fully functional, and any improvements to the former should be regarded as addressing a temporary situation).

The Android app store/Linux distro repository analogies don't really work once we allow two different formats. For that model to work well, you'd want to have all source sites running the full-blown Car Archive software (or at least offering the same interface) and providing cars through the same format, so that tools can learn and meaningfully decide about available versions, rather than being limited to a minimal set of fields shared between two formats. While this is a good model in theory, I don't think it is realistic for us as a general solution, as I can't see it being accessible enough for creators universally wanting to provide their cars in such a way from their own sites.

The alternative design Cas mentions is a more natural fit for working in tandem with the Car Archive. It could play out like this: a site (competitions would make for a typical use case) makes a list of required cars available from a file (that would be essentially a Pretty Garage group definition, perhaps optionally augmented by exact version specifications whenever that becomes necessary). Pretty Garage would fetch this list and use it to download its cars from the Car Archive (or any mirror thereof).

[Cas]

Hey, I think this got too complex. I've realised the solution for all of us is very simple. When we see something we don't like, we tend to feel that our work is being sabotaged and we have the urge to respond quickly. I understand Dreadnaut perfectly on this because I feel the exact same way. But we both know this is just a feeling and we really just want to do our work and follow our plans and develop. So... I cooled down and thought "what is it that is causing the problem and how can we get rid of only that?"

Well, the idea I proposed initially was that CRIs should have: an ID, car name and list of files. Any more information can be there, but is not necessary. Now I think we just need to restructure it this way: an ID, car name, list of files and where to find them. With that, the conflict is over. Example (doesn't have to be this format):

Code Select Expand

car:
 name: Porsche March Indy
 id: PMIN
 files:
 - pmin.zip
 location: https://whatever.com/car
car:
 name: Some Custom Car
 id: SCCR
 files:
 - carsccr.res
 - stsccr.3sh
 - stdasccr.vsh
 - stdbsccr.vsh
 location: http://this.otherplace.net/carfiles
This means that, if I want to grab pmin.zip, I'll curl it down from https://whatever.com/car/pmin.zip and if I want to get stdasccr.vsh, I'll go and download it from http://this.otherplace.net/carfiles/stdasccr.vsh.

Done. So, if you have a tournament and want to host your cars, that's OK. If you want to have people get them from the Archive instead, it's OK too. I don't get into convincing people to do either way. But, regardless or where the cars are located, you, in your site, provide the list, the CRI, which is the thing that doesn't make sense to centralise. There really isn't much philosophy here.

Like always, I'm open to ideas on what to add or change. I just don't want to be told not to develop something. I'm trying to cooperate here. I hope that is always clear.

[Duplode]

After a nice chat with @Cas about this topic, I now have a better understanding of how CRIs are meant to work. In particular, I no longer feel they clash with the vision we're going for with the Car Archive. Below is a summary of why I have changed my view.

The key point is that a CRI does not belong to a repository, nor it is meant as a manifest of the full contents of a repository. Rather, a CRI is at heart a Pretty Garage group definition, with added information on how to obtain the car files from chosen sources. The goal of writing a CRI should not be describing or exposing the contents of a car collection, but merely offering Pretty Garage the means to fetch a selected group of cars.

Here goes a concrete example of an use case. In the future, when the Car Archive is fully functional and the Southern Cross collection has been made obsolete by it, I may nonetheless want to provide an easy way for visitors to the Arrabona project to download the set of custom cars that were used in USC. One way I might accomplish that is by making a CRI available. This CRI would list the USC cars and specify the Car Archive as the source from which to fetch them.

The example illustrates two aspects of how CRIs can be put into use. Firstly, there need not be a conflict between sites offering their own CRI files and centralising downloads and metadata at our consolidated Car Archive: to promote centralisation, all we have to do is to encourage site admins to use the Car Archive as the source in their CRIs. Secondly, while in theory I might try writing an omnibus CRI file indistinctly covering the whole legacy Southern Cross collection, that's not the intended usage for the format, or even a particularly useful thing to do. Pretty Garage is primarily not a remote collection browser, but a local collection manager, and an indistinct omnibus group makes little sense in that context.

CRIs, then, are not an alternative way of defining repositories, but a way for Pretty Garage, or a client like it, to access repository contents. Now, there might be finer aspects on the interaction between client and repository which might affect the CRI format -- for instance, perhaps using directly using URLs for the files will prove inconvenient, and it will be better for the file to just point to the Car Archive root and let further talk to be done through the API. Those, however, are technical decisions we can work out, rather than fundamental philosophical disagreements. 
   

[dreadnaut]

Is it worth at least converging to one format? Apart from the difficulty that @Cas would have building a YAML parser, which I realised going through the spec, it would save consumers effort.

A search or multiple-car request to the Archive would respond with something like this:

Code Select Expand

[PMIN]
name = Porsche March Indy
nickname = ...
urls[wiki] = https://wiki.stunts.hu/...

[CDOR]
name = Melange XGT-88
nickname = ...

Which is basically the same content as the example in the first post. I could extend it to make it explicit where to download the zipped version

Code Select Expand

[PMIN]
name = Porsche March Indy
version = dsi-1.1
zip = https://stunts.hu/cars/PMIN/dsi-1.1.zip

while the list of files in the repository index could be specified as

Code Select Expand

[PMIN]
name = Porsche March Indy
version = dsi-1.1
files[] = readme.txt
files[] = CARPMIN.RES
files[] = STPMIN.P3S
files[] = STDAPMIN.PVS
files[] = STDBPMIN.PVS

[...]

You might note a version number there. The discussion above made me realise that we need to keep that explicit, because an old competition might want to point to a specific version of the car, and not to the most recent one. Versions can "branch", so it's not simple ordering. I think I'm roughly proposing WXYZ/<version-id> as a versioned car identifier.