The Protocol - Ideas on unification and integration

[Cas]

I sent a PM to Dreadnaut about a set of related half-ideas I've had. He replied suggesting that I create a topic so that everybody can contribute and I think that's a good idea. Only problem is I can't find the "sent" PMs, so I'll just have to try to remember and rewrite the concept. Here it goes:

It all started when I was trying to enhance the messaging system in Race For Kicks website. I though it'd be cool if I created a PM system there. What stopped me was the thought that we already have way too many ways of communicating. We have the very basic chat system at R4K, but we also have the Shoutbox at ZakStunts. And we have the forum, which can handle PMs too. And we tend to only check one of them most of the time and the others seldom and it's not always the same we check the most. This makes some of us get messages late. Besides, each system has its advantages and disadvantages. So.... the following things came to my mind:

1 - I was wondering if there could be a way in which the R4K site could use ZakStunts username/password as a "passport" system or if we could create a whole new system that both sites could share, so that pipsqueaks don't have to log in to each. This would simplify matters. I'm sure this is possible because I have access to R4K code and Zak and Dreadnaut have access to ZakStunts code, but I don't know how easy or hard it would be.
2 - If [1] is possible, then we might be able to make the Shoutbox and R4K chat system one thing, so we could all check the chat at any of the sites and/or we could create a PM system for both ZakStunts and R4K and share that (even if the general chat/shoutbox is not shared).
3 - Now, if we make PM system for R4K or ZakStunts or both, then the forum's PM system would be redundant. I wonder (and this time I don't know how possible this is) if we could somehow also make the forum work in conjunction with the sites, that is, read from the same PM pool and possibly use a "passport" system compatible with the sites. About the PM system, if this is too hard, it's easy for us to create a very good PM system for ZakStunts and R4K that will be much more powerful than the one in the forum, so we would just use the forum as a forum and PM on the sites.
4 - If this level of unification can be achieved, it can be used for other purposes too. For example, Dreadnaut mentioned an idea of a non-web-based program that would automatically submit replays as they are created for faster interaction in live races. This same client could be much more than that, and allow a whole non-web system to access the community sites and tournaments. A program like this could allow plugins and modules so it'd be much easier to add things to it than it is to add them to the sites (I think). Of course, this could be a risk for the security of the sites if not done properly. It's just an ambitious idea so far.
5 - Bliss has some minimal integration with ZakStunts currently. I could enhance that. Currently, I don't have Wine on my computer, so I would have to reinstall it, which I'm having a few problems with, but I can solve that, in order to compile Bliss or a client program for Windows. I am ready to compile for GNU/Linux, however.

If you guys have any good idea even if vaguely related to any of these points, please add to the brainstorming!    And if you know the answers to my questions and doubts, of course, I would like to know. Also, whatever I can do, count on me. I have much more experience with non-web programming, but online, I can reasonably handle PHP, although I've never used databases. As some of you already know, I've been a lone wolf all these years, so my programming style is very particular and I have hermit views   I don't get along well with OOP and I prefer to write my own code to using libraries whenever possible. But I'm open minded and can settle to middle-ground, ha, ha.

[dreadnaut]

Hahaha, your pm was far shorter than this!

Anyway, above are a bunch of interesting idea, and connecting competitions would be infinitely cool. I have some reservations on building a PM system, even on ZakStunts, as there are now a thousand ways to communicate and I think I'd better direct my effort in a different direction

However... receiving messages and replays should not be too difficult. There's a bit of work to prevent abuse (possibly signing messages with short lived keys, etc.) but it's definitely doable.

The "passport" bit is a bit more complicated. That's what the industry calls Single Sign-On, and you step into OAuth territory. It's stuff I've used at work, there's library for it, but it requires some effort to set up.

One thing I'd like work on for multiple competitions would be a standard format to archive a race. Put the track, the final replays, details about submissions (just CSV would work), all inside a zip file. We could make a library for that format and use it in ZakStunts or R4K, and we could store races in an easy format. One day, when we build the final archive of all track and all replays (!!!), it could read the archives and import then.

[Cas]

It's true there are too many forms of communication. That's what stopped me from creating a PM system in R4K. The simple chat, I though necessary, though, so somebody could say if they had a problem with a RPL or something and would be easily seen, for example. But for general chat or PM, it'd be best for all to be just one thing. Yet, from what you say, it looks complicated to share a pool of messages.... unless.... we don't actually share it. I mean, what if posting at one place would post to all?  Anyway!

About race format, yes, that would be good. For the time being, R4K is saving each race as a directory containing the replays and a scoreboard file that's like an INI file. The track file is also copied there. I could zip all that. I haven't done it yet because that would make people unable to download replays individually, but for archiving purposes, I guess the ZIP file could be sent somewhere else or created when necessary.

For the track registry, maybe we could similarily plan some formats. Of course, if the information is the same, the format doesn't matter as long as it can be shared. But it'd be nice to be able to look up a track and see in which tournaments and which races it was played. Maybe tracks and replays could hold "dynamic tags", meaning that a pipsqueak can come and say "Oh, this track is fast" and then add the tag "fast" to it or maybe "very hard" or "loops" if it has many loops. Similarly, a replay could bear tags such as "inverted crash" or "powergear". These tags would aid in finding a replay we're looking for. I say "dynamic" because we don't have to set all tags when the track or replay is first submitted, but at any point in the future.

Also, so far, we have a vague definition of "track name". The "file name" is indeed well-defined, but for example, if a track is used twice in ZakStunts, then it'll have two different file names. Also, in another tournament, it may have yet another file name and still, it's the same track. This has put me to think that the "fantasy name" is perhaps more unique and more deserving of recognition, so I prefer to call it "track title" most of the time. But the track name could be some other thing too. Whatever, as long as it's agreed on and is unique.

Well, I think I'm just moving to another topic   I'm just thinking aloud.

[dreadnaut]

I experimented with extending ZakStunts to be an OAuth provider, and it should not be too difficult. A competition on a different website could add a "Login through ZakStunts" button and a process equivalent to this, and they would receive the pipsqueak's name, three-letter code, and a token. That would give all ZakStunts pipsqueaks immediate access to R4K, no excuses!

It would be a bit insecure, because stunts.hu does not have a HTTPS, but the thread model is quite relaxed around here

The code above is connected to this tutorial, which is nice and readable I think.

Also, so far, we have a vague definition of "track name". The "file name" is indeed well-defined, but for example, if a track is used twice in ZakStunts, then it'll have two different file names. Also, in another tournament, it may have yet another file name and still, it's the same track. This has put me to think that the "fantasy name" is perhaps more unique and more deserving of recognition, so I prefer to call it "track title" most of the time. But the track name could be some other thing too. Whatever, as long as it's agreed on and is unique.

Good point, I'm having some trouble in ZakStunts because a track and a race are the same thing, and there are multiple races on the same track, which is still saved twice (e.g., Oligoway). Maybe the 8-char name should just be the "download name". It's a bit disappointing that Stunts does not include a title in the format

[Cas]

We could start up by creating a very, very rudimentary and basic version of the track registry and assign each track a unique ID number, for example. The track title is a good thing, but we have to accept that many tracks never actually had one at all.

I liked that you've implemented UTF-8 in ZakStunts. I think by default any text properties for a track, or race, or replay, should be UTF-8. Metadata in Bliss is currently ASCII. It's not possible to enter extended characters. So it's compatible with UTF-8. I'll eventually update it to it.

I'm reading about OAuth. It looks complex. Uhm... yeah, maybe the risk isn't much in our case, but I don't know, the sites are working well now. Still, it's something to keep in mind when we want to unify something else. Like, the PM idea I had mentioned took me to consider logging in with ZakStunts, but you're right that the PM system is probably not really necessary. Then maybe the logging system should be implemented when there is another thing to make it worth all the work. I don't know. I'll keep it roaming in my head, though.

[dreadnaut]

I'll throw this here

If you go to the "All results" page for a track, the download links for the replays now contain the replay sha1 hash:

Code Select Expand


http://zak.stunts.hu/replays/d3079e8bb43f91c4eb630da6828cf718664204de

Take the hash, and put it in this new endpoint:

Code Select Expand


http://zak.stunts.hu/api/replays/d3079e8bb43f91c4eb630da6828cf718664204de

and you switch to the wonderful world of data!

There is very little at the moment (replays and tracks), and what is there is likely to change when I split races and tracks, results and replays. But hey, it's a start.

Now, what can we build on it?

[Cas]

Ah!  I just so you had posted. Sometimes I don't realise there's a new message.
That's very good!  Finally, the beginning of what will be The Registry!  Uhm... if I have to think fast and say what's next, I'd say this: there should be a file one knows to look for that contains the list of the replay hashes and/or file names. Otherwise, I have to know in advance the ID for a specific replay.

And about the contents, it's good. You are able to obtain the track details from the race number, which is contained in this file. Yeah, you could add some track info there, such as the track name, but it's still already accessible indirectly as I said.

In reality, to have the replays and tracks (at least the ones raced in ZakStunts) all in the registry, the only thing you're left to add is lists like what I said. After that, the rest of the matter is an interface, perhaps, so one can navigate the data online comfortably. Now, to be able to incorporate tracks and replays that are not part of ZakStunts, or to add information about tracks that have been raced somewhere else too, yes, something else could be done.

When we talked about the registry in the past, I imagined things like, you'd get to a page that'd be the track's page (not the race page) and see the map and metadata, like track's title, author, creation date, etc. Then below, you'd have the list of events that this track formed part of (whether the information is taken from the track's registry file or from the tournament's or whatever) and you can click on, say, the author, and it shows you what else he has done or on the tournament and you get to the tournament's registry and so on.

I had also considered what it'd be like if one could "submit" a track or replay to the registry independently, that is, not by participating in a race, which would make that automatic. If that were possible, some things have to be taken into account, such as... is this a new track or a version of an existing track?  Does this user have the privileges to submit or change the track?  What happens if I submit a track, then realise there's something I did wrong, so I fix it and want to replace it?  Etc.

Anyway, brainstorming as usual

[Duplode]

Now, what can we build on it?

*Puts on pestering end-user hat*

I pine for the day when I no longer have to clone and then massage the database in order to do stuff like this