{"id":35060,"date":"2022-02-17T16:35:13","date_gmt":"2022-02-17T21:35:13","guid":{"rendered":"https:\/\/mjtsai.com\/blog\/?p=35060"},"modified":"2022-03-09T16:00:30","modified_gmt":"2022-03-09T21:00:30","slug":"apple-ssd-benchmarks-and-f_fullsync","status":"publish","type":"post","link":"https:\/\/mjtsai.com\/blog\/2022\/02\/17\/apple-ssd-benchmarks-and-f_fullsync\/","title":{"rendered":"Apple SSD Benchmarks and F_FULLSYNC"},"content":{"rendered":"

Hector Martin<\/a> (Hacker News<\/a>):<\/p>\n

It turns out Apple’s custom NVMe drives are amazingly fast - if you don’t care about data integrity.<\/p>

[…]<\/p>

On Linux, fsync() will both flush writes to the drive, and ask it to flush its write cache to stable storage.<\/p>

But on macOS, fsync()<\/code> only flushes writes to the drive. Instead, they provide an F_FULLSYNC<\/code> operation to do what fsync()<\/code> does on Linux.<\/p>

[…]<\/p>

So effectively macOS cheats on benchmarks; fio on macOS does not give numbers comparable to Linux, and databases and other applications requiring data integrity on macOS need to special case it and use F_FULLSYNC<\/code>.<\/p>

[…]<\/p>

So, effectively, Apple’s drive is faster than all the others without cache flushes, but it is more than 3 times slower than a lowly SATA SSD at flushing its cache.<\/p><\/blockquote>\n\n

As far as I can tell, the summary is:<\/p>\n

    \n
  1. fsync()<\/a><\/code> does different things on Mac and Linux for historical reasons.<\/li>\n
  2. Many non-Apple SSDs don’t actually flush their cache when doing F_FULLSYNC<\/code>; they seem faster because they lie.<\/li>\n
  3. Compared with other SSDs that actually do flush, Apple’s are (for unknown reasons) much slower, though they are faster when not flushing. Or, perhaps, these non-Apple SSDs are lying, too.<\/li>\n
  4. Often, what you really want is F_BARRIERFSYNC<\/code>, not F_FULLSYNC<\/code>.<\/li>\n<\/ol>\n\n

    Dominic Evans<\/a>:<\/p>\n

    \n

    Surely that’s a mischaracterisation to claim they’re “cheating” — this is just legacy diversions. On earlier versions of the Linux kernel and in posix fsync()<\/code> didn’t used to flush the cache either. Darwin independently added the special fnctl<\/code> to do a “FULLSYNC<\/code>” long ago<\/p>\n

    Yes newer kernels (2.6 onward or something?) changed the semantics of fsync()<\/code> to request the full cache flush too. Darwin didn’t change their fsync<\/code> because they already had their fnctl<\/code> to provide the option where needed.<\/p>\n<\/blockquote>\n\n

    Dominic Giampaolo<\/a>, in 2005:<\/p>\n

    \n

    On MacOS X, fsync()<\/code> always has and always will flush all file data\nfrom host memory to the drive on which the file resides. The behavior\nof fsync()<\/code> on MacOS X is the same as it is on every other version of\nUnix since the dawn of time (well, since the introduction of fsync<\/code>\nanyway :-).<\/p>\n

    I believe that what the above comment refers to is the fact that\nfsync()<\/code> is not sufficient to guarantee that your data is on stable\nstorage and on MacOS X we provide a fcntl()<\/code>, called F_FULLFSYNC<\/code>,\nto ask the drive to flush all buffered data to stable storage.<\/p>\n<\/blockquote>\n\n

    Rosyna Keller<\/a>:<\/p>\n

    \n

    Force Unit Access, what this “flush to permanent storage, not disk cache” command is called, is ignored by the majority of drive types (either through lying firmware or a bridge).<\/p>\n

    It’s not enabled by default in most kernels (Linux, Windows) due to synchronous writes being slow.<\/p>\n

    […]<\/p>\n

    However, every disk Apple ships actually supports Force Unit Access (F_FULLSYNC<\/code>), and is under a different flag because most cross-platform developers don’t expect fsync()<\/code> to actually be synchronous, leading to massive performance losses compared to drives that don’t support it.<\/p>\n

    If you write software they uses full flushing on firmware that isn’t a lying liar and actually goes through a flush to permanent storage, remember that every time you’re doing the full sync, you significantly impact the performance of the entire system, not just your software.<\/p>\n<\/blockquote>\n\n

    Andrew Wooster<\/a>:<\/p>\n

    \n

    I was the backupd performance lead and I’d love to move on but it keeps coming up. 🤷‍♂️🤷‍♂️<\/p>\n

    […]<\/p>\n

    I am thankful for the various people at Apple who made sure Apple hardware functioned correctly. Otherwise it would’ve been impossible to have both performance and correctness. The former is easy if you ignore the latter.<\/p>\n<\/blockquote>\n\n

    Unfortunately, there were problems in another layer that made Time Capsules corrupt their data all the time.<\/p>\n\n

    Hector Martin<\/a>:<\/p>\n

    So you’re saying my WD NVMe drive lies about flushes, and yet they’re 10x slower than not flushing? Must be really bad at lying then…<\/p>

    The problem is Apple SSDs are 1000x slower when flushing. That’s called a firmware bug.<\/p><\/blockquote>\n\n

    Maynard Handley<\/a>:<\/p>\n

    As I described elsewhere, the traditional solution to ordering writes on unix is fsync<\/code>. This is a highly sub-optimal solution because it does much more than required.<\/p>

    Apple’s solution is to use the equivalent of barriers, rather than flushes, to enforce ordering; and it works every bit as well as the equivalent solution (barriers rather than flushes) in a CPU pipeline.<\/p><\/blockquote>\n\n

    Scott Perry<\/a>:<\/p>\n

    \n

    There’s a third sync operation that lets you have your performance and write ordering too: F_BARRIERFSYNC<\/code>. SQLite already uses it on Darwin, and it’s part of the best practices guide<\/a> for I\/O reduction<\/p>\n<\/blockquote>\n\n

    Update (2022-03-09): See also: Howard Oakley<\/a>, MacRumors<\/a>, Howard Oakley<\/a>, JP Simard<\/a>.<\/p>\n\n

    Russ Bishop<\/a> (Hacker News<\/a>):<\/p>\n

    \n

    I tested a random selection of four NVMe SSDs from four vendors. Half lose FLUSH’d data on power loss. That is the flush went to the drive, confirmed, success reported all the way back to userspace. Then I manually yanked the cable. Boom, data gone.<\/p>\n

    The other half never lost data confirmed after a flush (F_FULLFSYNC on macOS) no matter how much I abused them. All four had perf hit from flushing so they are doing some work.<\/p>\n

    Top two performers on flush? One lost data 40% of the time. The other never lost any.<\/p>\n

    I guess review sites don’t test this stuff. Everyone just assumes data disappearing on crash\/power loss is just how computers work?<\/p>\n

    I feel bad for the other two vendors who must have test suites and spent engineering hours making sure FLUSH works, only to find out no one cares<\/p>\n<\/blockquote>","protected":false},"excerpt":{"rendered":"

    Hector Martin (Hacker News): It turns out Apple’s custom NVMe drives are amazingly fast - if you don’t care about data integrity.[…]On Linux, fsync() will both flush writes to the drive, and ask it to flush its write cache to stable storage.But on macOS, fsync() only flushes writes to the drive. Instead, they provide an […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"apple_news_api_created_at":"2022-02-17T21:35:16Z","apple_news_api_id":"3460061c-e09f-4e0e-966b-3f66247c8a7c","apple_news_api_modified_at":"2022-03-09T21:00:33Z","apple_news_api_revision":"AAAAAAAAAAAAAAAAAAAABQ==","apple_news_api_share_url":"https:\/\/apple.news\/ANGAGHOCfTg6Waz9mJHyKfA","apple_news_coverimage":0,"apple_news_coverimage_caption":"","apple_news_is_hidden":false,"apple_news_is_paid":false,"apple_news_is_preview":false,"apple_news_is_sponsored":false,"apple_news_maturity_rating":"","apple_news_metadata":"\"\"","apple_news_pullquote":"","apple_news_pullquote_position":"","apple_news_slug":"","apple_news_sections":"\"\"","apple_news_suppress_video_url":false,"apple_news_use_image_component":false,"footnotes":""},"categories":[2],"tags":[1321,143,31,2078,448,30,2077,138,71,183,425,503,216],"class_list":["post-35060","post","type-post","status-publish","format-standard","hentry","category-technology","tag-data-integrity","tag-database","tag-ios","tag-ios-15","tag-linux","tag-mac","tag-macos-12","tag-optimization","tag-programming","tag-ssd","tag-sqlite","tag-timecapsule","tag-timemachine"],"apple_news_notices":[],"_links":{"self":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/35060","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/comments?post=35060"}],"version-history":[{"count":5,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/35060\/revisions"}],"predecessor-version":[{"id":35244,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/35060\/revisions\/35244"}],"wp:attachment":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/media?parent=35060"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/categories?post=35060"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/tags?post=35060"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}