{"id":33301,"date":"2021-08-03T17:13:01","date_gmt":"2021-08-03T21:13:01","guid":{"rendered":"https:\/\/mjtsai.com\/blog\/?p=33301"},"modified":"2021-08-03T17:19:34","modified_gmt":"2021-08-03T21:19:34","slug":"automating-test-retries","status":"publish","type":"post","link":"https:\/\/mjtsai.com\/blog\/2021\/08\/03\/automating-test-retries\/","title":{"rendered":"Automating Test Retries"},"content":{"rendered":"<p><a href=\"https:\/\/pspdfkit.com\/blog\/2021\/automating-test-retries\/\">Akshat Patel<\/a>:<\/p>\n<blockquote cite=\"https:\/\/pspdfkit.com\/blog\/2021\/automating-test-retries\/\">\n<p>Once we had a list of the flaky tests, we tried to go through each one and determine why they were failing. We found that some UI elements such as menus and popovers were particularly prone to flakiness &mdash; they would sometimes be dismissed by the system for no discernable reason!<\/p>\n<p>[&#8230;]<\/p>\n<p>Since we already had the JUnit parsing code, we decided to build on top of that and rerun only the failed tests. By using the <code>xcodebuild<\/code> command&rsquo;s <code>-only-testing<\/code> flag, we ran only the failed tests again. Another optimization we made was to build the project only once, even when testing multiple times. We accomplished that by using the <code>xcodebuild build-for-testing<\/code> and <code>xcodebuild test-without-building<\/code> commands.<\/p>\n<p>[&#8230;]<\/p>\n<p>Flaky tests still exist, but they no longer slow down the workflow of our developers. CI automatically retries any failing tests, and almost all flaky tests pass when run again. If a test actually fails three times in a row, only then it is considered an actual failure and the build is marked as failed.<\/p>\n<\/blockquote>\n<p>Xcode 13 has a built-in option to do this. But why are these tests flaky?<\/p>","protected":false},"excerpt":{"rendered":"<p>Akshat Patel: Once we had a list of the flaky tests, we tried to go through each one and determine why they were failing. We found that some UI elements such as menus and popovers were particularly prone to flakiness &mdash; they would sometimes be dismissed by the system for no discernable reason! [&#8230;] Since [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"apple_news_api_created_at":"2021-08-03T21:13:06Z","apple_news_api_id":"5b59ee2e-bdc5-4d60-bebb-dd918c93d3a3","apple_news_api_modified_at":"2021-08-03T21:19:39Z","apple_news_api_revision":"AAAAAAAAAAAAAAAAAAAAAA==","apple_news_api_share_url":"https:\/\/apple.news\/AW1nuLr3FTWC-u92RjJPTow","apple_news_coverimage":0,"apple_news_coverimage_caption":"","apple_news_is_hidden":false,"apple_news_is_paid":false,"apple_news_is_preview":false,"apple_news_is_sponsored":false,"apple_news_maturity_rating":"","apple_news_metadata":"\"\"","apple_news_pullquote":"","apple_news_pullquote_position":"","apple_news_slug":"","apple_news_sections":"\"\"","apple_news_suppress_video_url":false,"apple_news_use_image_component":false,"footnotes":""},"categories":[4],"tags":[69,31,1837,71,268,226],"class_list":["post-33301","post","type-post","status-publish","format-standard","hentry","category-programming-category","tag-cocoa","tag-ios","tag-ios-14","tag-programming","tag-testing","tag-xcode"],"apple_news_notices":[],"_links":{"self":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/33301","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/comments?post=33301"}],"version-history":[{"count":2,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/33301\/revisions"}],"predecessor-version":[{"id":33311,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/33301\/revisions\/33311"}],"wp:attachment":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/media?parent=33301"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/categories?post=33301"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/tags?post=33301"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}