{"id":6532,"date":"2012-11-09T17:07:47","date_gmt":"2012-11-09T22:07:47","guid":{"rendered":"http:\/\/mjtsai.com\/blog\/?p=6532"},"modified":"2013-03-05T20:42:23","modified_gmt":"2013-03-06T01:42:23","slug":"fogbugz-on-demand-and-hurricane-sandy","status":"publish","type":"post","link":"https:\/\/mjtsai.com\/blog\/2012\/11\/09\/fogbugz-on-demand-and-hurricane-sandy\/","title":{"rendered":"FogBugz on Demand and Hurricane Sandy"},"content":{"rendered":"<p><a href=\"http:\/\/blog.fogcreek.com\/hurricane-sandy-wrap-up\/\">Mendy Berkowitz<\/a>:<\/p>\n<blockquote cite=\"http:\/\/blog.fogcreek.com\/hurricane-sandy-wrap-up\/\"><p>That evening, the storm surge from Sandy, assisted by a rising tide and full moon, flooded the basement of our data center, cutting off the fuel supply to the backup generator. The next morning, Peer1 informed us of an impending emergency shutdown of the generator. We executed a protective shutdown to prevent loss or corruption of customer data. Later, when we&rsquo;d secured confirmation from Peer1 that there was no imminent danger of power loss, we restarted our systems. The total duration of this unplanned, voluntary downtime was 3 hours.<\/p><\/blockquote>\n<p><a href=\"http:\/\/www.fogcreek.com\/fogbugz\/\">FogBugz<\/a> is a great service, and the team <a href=\"http:\/\/blog.stackoverflow.com\/2012\/11\/se-podcast-36-we-got-hit-by-a-hurricane\/\">worked hard<\/a> to minimize the downtime. The data center is still not on city power, but the service is running smoothly. Once things are back to normal, I think we&rsquo;re owned an explanation for why there was only one data center. When I signed up for the hosted FogBugz service, it was based on <a href=\"http:\/\/www.joelonsoftware.com\/items\/2007\/07\/09.html\">Joel Spolsky&rsquo;s 2007 description of their infrastructure<\/a>:<\/p>\n<blockquote cite=\"http:\/\/www.joelonsoftware.com\/items\/2007\/07\/09.html\"><p>Rather than setting up Los Angeles as a mere <em>backup<\/em>, we decided it would be completely live. Half our customers will be hosted from Los Angeles, and half from New York. That way we know at any time that both data centers are working and set up correctly, and we don&rsquo;t have to wait until a massive failure to discover the problems with the backup data center.<\/p>\n<p>Copies of the database backups are maintained in both cities, and each city serves as a warm backup for the other. If the New York data center goes completely south, we&rsquo;ll wait a while to make sure it&rsquo;s not coming back up, and then we&rsquo;ll start changing the DNS records and start bringing up our customers on the warm backup in Los Angeles.<\/p>\n<\/blockquote><p>However, the current <a href=\"http:\/\/www.fogcreek.com\/fogbugz\/on-demand-infrastructure.html\">infrastructure page<\/a> describes only a single data center.<\/p>\n<p>Update (2013-03-05): <a href=\"http:\/\/blog.fogcreek.com\/cage-set-match-fog-creek-infrastructure-changes\/\">Mendy Berkowitz<\/a>:<\/p>\n<blockquote cite=\"http:\/\/blog.fogcreek.com\/cage-set-match-fog-creek-infrastructure-changes\/\"><p>All excellent questions, but as we debated the various answers we realized there was nothing wrong with our current datacenter that a second datacenter wouldn&rsquo;t fix. In a disaster (natural or man made)  situation, a second geographically diverse datacenter with a tested and practiced failover procedure is our best option for providing our customers with continued service.<\/p><\/blockquote>","protected":false},"excerpt":{"rendered":"<p>Mendy Berkowitz: That evening, the storm surge from Sandy, assisted by a rising tide and full moon, flooded the basement of our data center, cutting off the fuel supply to the backup generator. The next morning, Peer1 informed us of an impending emergency shutdown of the generator. We executed a protective shutdown to prevent loss [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"apple_news_api_created_at":"","apple_news_api_id":"","apple_news_api_modified_at":"","apple_news_api_revision":"","apple_news_api_share_url":"","apple_news_coverimage":0,"apple_news_coverimage_caption":"","apple_news_is_hidden":false,"apple_news_is_paid":false,"apple_news_is_preview":false,"apple_news_is_sponsored":false,"apple_news_maturity_rating":"","apple_news_metadata":"\"\"","apple_news_pullquote":"","apple_news_pullquote_position":"","apple_news_slug":"","apple_news_sections":"\"\"","apple_news_suppress_video_url":false,"apple_news_use_image_component":false,"footnotes":""},"categories":[2],"tags":[43,96],"class_list":["post-6532","post","type-post","status-publish","format-standard","hentry","category-technology","tag-fogbugz","tag-web"],"apple_news_notices":[],"_links":{"self":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/6532","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/comments?post=6532"}],"version-history":[{"count":2,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/6532\/revisions"}],"predecessor-version":[{"id":7069,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/6532\/revisions\/7069"}],"wp:attachment":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/media?parent=6532"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/categories?post=6532"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/tags?post=6532"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}