I could not rate the track because was not good/enjoyable enough to be raceable.
This is an interesting point, but a difficult one to solve. There is currently a workaround, where one could send a list-filler, and therefore be able to rate the track. It's a double effort (submit a replay, and come back to rate) so it's a poor workaround

One alternative would be to offer a "Forfeit race" option, where you give up on the race, cannot submit further replays for it, and are allowed to rate it. This is also a poor fix, because it lowers the threshold of abuse. Maybe once can only forfeit if they raced on the previous track?
At the same time, the track of the year is meant to celebrate good and fun tracks. If a track is not good, it will have a lower average anyway. Allowing rare "really bad" votes is not going to affect the result that much. My feeling at the moment is to leave those out, and accept the minimal statistical error.