{"id":17034,"date":"2017-01-27T14:41:19","date_gmt":"2017-01-27T19:41:19","guid":{"rendered":"http:\/\/mjtsai.com\/blog\/?p=17034"},"modified":"2017-01-30T14:31:23","modified_gmt":"2017-01-30T19:31:23","slug":"flickrs-year-without-buying-storage","status":"publish","type":"post","link":"https:\/\/mjtsai.com\/blog\/2017\/01\/27\/flickrs-year-without-buying-storage\/","title":{"rendered":"Flickr&rsquo;s Year Without Buying Storage"},"content":{"rendered":"<p><a href=\"https:\/\/code.flickr.net\/2017\/01\/05\/a-year-without-a-byte\/\">Archie Russell<\/a> (via <a href=\"https:\/\/news.ycombinator.com\/item?id=13385873\">Hacker News<\/a>):<\/p>\n<blockquote cite=\"https:\/\/code.flickr.net\/2017\/01\/05\/a-year-without-a-byte\/\"><p>One of the largest cost drivers in running a service like Flickr is storage. We&rsquo;ve described multiple techniques to get this cost down over the years: use of <a href=\"https:\/\/yahooeng.tumblr.com\/post\/116391291701\/yahoo-cloud-object-store-object-storage-at\">COS<\/a>, <a href=\"https:\/\/code.flickr.net\/2015\/06\/25\/real-time-resizing-of-flickr-images-using-gpus\">creating sizes dynamically<\/a> on GPUs and <a href=\"https:\/\/code.flickr.net\/2015\/09\/25\/perceptual-image-compression-at-flickr\/\">perceptual compression<\/a>. These projects have been very successful, but our storage cost is still significant.<\/p>\n<p>At the beginning of 2016, we challenged ourselves to go further &mdash; to go a full year without needing new storage hardware. Using multiple techniques, we got there.<\/p>\n<p>[&#8230;]<\/p>\n<p>Because we were concerned that further rollout of dynamic thumbnail generation would place a heavy load on our resizing infrastructure, we targeted only thumbnails from less-popular images for deletes. Using this approach, we were able to handle our complete resize load with just four GPUs. The process put a heavy load on our storage systems; to minimize the impact we randomized our operations across volumes. The entire process took about four months, resulting in even more significant gains than our storage threshold adjustments.<\/p>\n<p>[&#8230;]<\/p>\n<p>Running our users&rsquo; original photos through lossless compression was probably our highest-risk approach. We can recreate thumbnails easily, but a corrupted source image cannot be recovered. Key to our approach was a re-compress-decompress-verify strategy: every recompressed image was decompressed and compared to its source before removing the uncompressed source image.<\/p><\/blockquote>\n<p>Previously: <a href=\"http:\/\/mjtsai.com\/blog\/2015\/10\/22\/real-time-resizing-of-flickr-images-using-gpus\/\">Real-time Resizing of Flickr Images Using GPUs<\/a>.<\/p>\n<p>Update (2017-01-30): I don&rsquo;t know whether this is related, but Flickr <a href=\"https:\/\/twitter.com\/mjtsai\/status\/825881924828856321\">is now<\/a> showing thumbnails from other people&rsquo;s photos in our private album.<\/p>","protected":false},"excerpt":{"rendered":"<p>Archie Russell (via Hacker News): One of the largest cost drivers in running a service like Flickr is storage. We&rsquo;ve described multiple techniques to get this cost down over the years: use of COS, creating sizes dynamically on GPUs and perceptual compression. These projects have been very successful, but our storage cost is still significant. [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"apple_news_api_created_at":"","apple_news_api_id":"","apple_news_api_modified_at":"","apple_news_api_revision":"","apple_news_api_share_url":"","apple_news_coverimage":0,"apple_news_coverimage_caption":"","apple_news_is_hidden":false,"apple_news_is_paid":false,"apple_news_is_preview":false,"apple_news_is_sponsored":false,"apple_news_maturity_rating":"","apple_news_metadata":"\"\"","apple_news_pullquote":"","apple_news_pullquote_position":"","apple_news_slug":"","apple_news_sections":"\"\"","apple_news_suppress_video_url":false,"apple_news_use_image_component":false,"footnotes":""},"categories":[2],"tags":[357,116,177,138,96],"class_list":["post-17034","post","type-post","status-publish","format-standard","hentry","category-technology","tag-compression","tag-flickr","tag-jpeg","tag-optimization","tag-web"],"apple_news_notices":[],"_links":{"self":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/17034","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/comments?post=17034"}],"version-history":[{"count":2,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/17034\/revisions"}],"predecessor-version":[{"id":17059,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/17034\/revisions\/17059"}],"wp:attachment":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/media?parent=17034"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/categories?post=17034"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/tags?post=17034"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}