{"id":30044,"date":"2020-09-08T14:37:20","date_gmt":"2020-09-08T18:37:20","guid":{"rendered":"https:\/\/mjtsai.com\/blog\/?p=30044"},"modified":"2024-06-15T14:06:16","modified_gmt":"2024-06-15T18:06:16","slug":"applebot","status":"publish","type":"post","link":"https:\/\/mjtsai.com\/blog\/2020\/09\/08\/applebot\/","title":{"rendered":"Applebot"},"content":{"rendered":"<p><a href=\"https:\/\/support.apple.com\/en-us\/HT204683\">Apple<\/a> (via <a href=\"https:\/\/news.ycombinator.com\/item?id=24338152\">Hacker News<\/a>):<\/p>\n<blockquote cite=\"https:\/\/support.apple.com\/en-us\/HT204683\"><p>Applebot is the web crawler for Apple. Products like Siri and Spotlight Suggestions use Applebot.<\/p>\n<p>Traffic coming from Applebot is identified by its user agent, and reverse DNS shows it in the *.applebot.apple.com domain, originating from the 17.0.0.0 net block.<\/p><\/blockquote>\n\n<p><a href=\"https:\/\/news.ycombinator.com\/item?id=24338758\">jd20<\/a>:<\/p>\n<blockquote cite=\"https:\/\/news.ycombinator.com\/item?id=24338758\"><p>Some fun facts:<\/p><ul><li>Applebot was originally written in Go (and uncovered a user agent bug on redirects, revealing it&rsquo;s Go origins to the world, which Russ Cox fixed the next day).<\/li><li>Up until the release of iOS 9, Applebot ran entirely on four Mac Pro&rsquo;s in an office. Those four Mac Pro&rsquo;s could crawl close to 1B web pages a day.<\/li><li>In it&rsquo;s first week of existence, it nearly took Apple&rsquo;s internal DNS servers offline. It was then modified to do it&rsquo;s own DNS resolution and caching, fond memories&#8230;<\/li><\/ul><\/blockquote>\n\n<p>Previously:<\/p>\n<ul>\n<li><a href=\"https:\/\/mjtsai.com\/blog\/2019\/07\/24\/google-open-sources-robots-txt-parser\/\">Google Open Sources robots.txt Parser<\/a><\/li>\n<\/ul>","protected":false},"excerpt":{"rendered":"<p>Apple (via Hacker News): Applebot is the web crawler for Apple. Products like Siri and Spotlight Suggestions use Applebot. Traffic coming from Applebot is identified by its user agent, and reverse DNS shows it in the *.applebot.apple.com domain, originating from the 17.0.0.0 net block. jd20: Some fun facts:Applebot was originally written in Go (and uncovered [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"apple_news_api_created_at":"2020-09-08T18:37:23Z","apple_news_api_id":"3221afb9-2135-4025-9d45-3298a38d50f9","apple_news_api_modified_at":"2024-06-15T18:06:19Z","apple_news_api_revision":"AAAAAAAAAAAAAAAAAAAAAA==","apple_news_api_share_url":"https:\/\/apple.news\/AMiGvuSE1QCWdRTKYo41Q-Q","apple_news_coverimage":0,"apple_news_coverimage_caption":"","apple_news_is_hidden":false,"apple_news_is_paid":false,"apple_news_is_preview":false,"apple_news_is_sponsored":false,"apple_news_maturity_rating":"","apple_news_metadata":"\"\"","apple_news_pullquote":"","apple_news_pullquote_position":"","apple_news_slug":"","apple_news_sections":"\"\"","apple_news_suppress_video_url":false,"apple_news_use_image_component":false,"footnotes":""},"categories":[2],"tags":[38,728,846,295,652,247,1042,96,2612],"class_list":["post-30044","post","type-post","status-publish","format-standard","hentry","category-technology","tag-apple","tag-domain-name-system-dns","tag-go-language","tag-history","tag-mac-pro","tag-siri","tag-spotlight","tag-web","tag-web-crawlers"],"apple_news_notices":[],"_links":{"self":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/30044","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/comments?post=30044"}],"version-history":[{"count":1,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/30044\/revisions"}],"predecessor-version":[{"id":30045,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/30044\/revisions\/30045"}],"wp:attachment":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/media?parent=30044"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/categories?post=30044"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/tags?post=30044"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}