VeryBlindDate.com and BestThing.info host contests where people vote to select the best choice from a large pool of user-submitted options. It uses a fairly simple system based loosely on the idea of the Condorcet criterion for voting systems.
The system gives each one a number based on, out of the things that it has faced, what percentage of them it usually beats. So each one gets the percentage score NumberOfThingsIBeat / (NumberOfThingsIBeat + NumberOfThingsThatBeatMe). Also, things are barred from the top list until they've been compared to a reasonable percentage of the set.|
If there's a Condorcet winner in the set, this method will find it. Since that's unlikely, given the statistical nature, it finds the one that's the "closest" to being a Condorcet winner. One nice thing is that this doesn't really suffer from the bash.org problem that the longer something is on the list, the higher its ranking can go.
The ideal algorithm would be one that had a measurement it could apply to a given ordering of the list -- specifically, it would probably be the total number of votes that were satisfied by the ordering. It would use simulated annealing to search the permutation space of this list to find the ordering that satisfied things the most. The reason we went with the simpler count is twofold.
One, it's simpler. It's easier to program, but more importantly it's easier to understand. It's an implementation with no parameters and no advanced mathematics that gives a pretty good result. It's also less computationally intensive and the results are easier to bound -- as powerful as the server running this whole game is, we want to easily handle millions of votes, and we're only so confident in our programming skills.
Two, it leads to fewer violent moment-by-moment upheavals of the Top List. As one relative peak rose above another in the problem space, the list could jump wildly -- a single vote could flip the entire top 10. While it may be algorithmically more accurate, when a system is ongoing, that sort of behavior is unsettling to voters who are watching the results like a horse race. This system still finds a pretty good answer, and it doesn't have much bias toward older winners, but the list can't be completely overturned by a single vote.
I think this is a pretty good method. I'm sure I'm not the first to try it, though I haven't seen any examples of its use before. If anyone has any research on the subject, please email me and let me know! And if you find an example of the application of simulated annealing to a list using a mass-vote-satisfaction metric like that, let me know as well, and I'll stop calling it the XKCD Algorithm.