Move Fast and Break Nothing
What happens is this: a request will come in as usual and run the existing (old) code. At the same time (or just right after it executes), we’ll also run the new code that we think will be better/faster/harder/stronger (pick one). Once all that’s done, return whatever the existing (old) code returns. So, from the user’s perspective, nothing has changed. They don’t see the effects of the new code at all.
[…]
Science (and its sister library, github/dat-analysis) can generate a graph of the number of times the code was run (the top blue bar to the left) and compare it to the number of mismatches between the control and the candidate (in red, on the bottom). In this case you see a downward trend: the developer saw that their initial deploy might have missed a couple use cases, and over subsequent deploys and fixes the mismatches decreased to near-zero, meaning that the new code is matching production’s behavior in almost all cases.
[…]
All of this gives you evidence to prove the safety of your code before you deploy it to your entire userbase. Sometimes we’ll run these experiments for weeks or months as we widdle down all the — sometimes tricky — edge cases. All the while, we can deploy quickly and iteratively with a pace we’ve grown accustomed to, even on dicey code. It’s a really nice balance of speed and safety.
This is the sort of thing that’s easier to do with hosted software. But it can be applied to apps as well: for example, a debug version of SpamSieve that runs both the old and new e-mail parsers and logs any differences in output.
