Dropbox’s Exodus From the Amazon Cloud Empire
Over the last two-and-a-half years, Dropbox built its own vast computer network and shifted its service onto a new breed of machines designed by its own engineers, all orchestrated by a software system built by its own programmers with a brand new programming language. Drawing on the experience of Silicon Valley veterans who erected similar technology inside Internet giants like Google and Facebook and Twitter, it has successfully moved about 90 percent of those files onto this new online empire.
[…]
In essence, they built their own Amazon S3—except they tailored their software to their own particular technical problems. “We haven’t built a like-for-like replacement,” Agarwal says. “We’ve built something that is customized for us.”
[…]
But according to Agarwal, the main reason for moving off the Amazon cloud is raw economics—not politics.
[…]
Crowling, Turner, and others originally built Magic Pocket using a new programming language from Google called Go. […] But Go’s “memory footprint”—the amount of computer memory it demands while running Magic Pocket—was too high for the massive storage systems the company was trying to build. Dropbox needed a language that would take up less space in memory, because so much memory would be filled with all those files streaming onto the machine. So, in the middle of this two-and-half-year project, they switched to Rust on the Diskotech machines.
As Tim Cook says, “We believe that we need to own and control the primary technologies behind the products that we make.”
Dropbox is trying to be more of a platform than a feature.
Update (2016-03-15): A Dropbox engineer is answering questions on Reddit. And on Hacker News he has commented more about Dropbox’s use of different languages.
Update (2016-03-16): Akhil Gupta:
Dropbox was founded by engineers, and the ethos of technical innovation is fundamental to our culture. For our users, this means that we’ve created a product that just works. But there’s a lot that happens behind the scenes to create that simple user experience.
[…]
This is the first of a series of blog posts about the Magic Pocket. Over the next month we’ll share a lot of the technical details around what we learned from building our own high-performance cloud infrastructure.
Update (2016-04-14): Cade Metz:
Dropbox stunned the tech world last month when it revealed it had spent the last two-and-a-half years moving about 90 percent of its operation off of the Amazon cloud. Meanwhile, its biggest rival, Box, is moving the other way.
