Algorithmic Arrangements at OkCupid
Tom Quisel (via Hacker News):
There could be two different interpretations for the question, and you just answered one of them. Then the population is evenly split on a question not because people feel strongly about the answer, but because they have different interpretations of the question. Knowing this can happen, we use the algorithms to help us understand the statistics behind each question, and we’ll try to identify questions that are the most likely to be mistaken in this way so that we can remove them.
We also examined messaging patterns as a backup, and correlated answers to other questions. So if the question is an outlier compared to many other questions, we’ll tend to count it less; or if messaging patterns don’t line up with answers to the question, we would sometimes use that as a reason to remove the question as well.
[…]
One change we debated quite a bit was our rating system. Originally we had a rating system that allowed people to score other people from one to five stars. And we thought, well, it would be a simpler user interface to just use a yes or no answer. That would be more straightforward, but then again we would lose a lot of information resolution, and was that really worth it? We were pretty torn on it, and couldn’t come to a decision through discussion alone, so we resorted to an experiment to understand which would lead to better messaging patterns.
[…]
The ethics around experimentation really depends on what you’re trying to accomplish with the question. The goal should be improving the product for people, and you should focus on not degrading the experience very much for any one person—don’t hurt someone too much for the experiment.
