{"id":43782,"date":"2024-06-19T15:32:10","date_gmt":"2024-06-19T19:32:10","guid":{"rendered":"https:\/\/mjtsai.com\/blog\/?p=43782"},"modified":"2024-07-30T14:10:44","modified_gmt":"2024-07-30T18:10:44","slug":"apple-intelligence-training","status":"publish","type":"post","link":"https:\/\/mjtsai.com\/blog\/2024\/06\/19\/apple-intelligence-training\/","title":{"rendered":"Apple Intelligence Training"},"content":{"rendered":"

Apple<\/a>:<\/p>\n

In the following overview, we will detail how two of these models — a ~3 billion parameter on-device language model, and a larger server-based language model available with Private Cloud Compute and running on Apple silicon servers — have been built and adapted to perform specialized tasks efficiently, accurately, and responsibly.<\/p>

[…]<\/p>

Our foundation models are trained on Apple’s AXLearn framework<\/a>, an open-source project we released in 2023. It builds on top of JAX and XLA, and allows us to train the models with high efficiency and scalability on various training hardware and cloud platforms, including TPUs and both cloud and on-premise GPUs. We used a combination of data parallelism, tensor parallelism, sequence parallelism, and Fully Sharded Data Parallel (FSDP) to scale training along multiple dimensions such as data, model, and sequence length.<\/p>

We train our foundation models on licensed data, including data selected to enhance specific features, as well as publicly available data collected by our web-crawler, AppleBot.<\/p><\/blockquote>\n\n

David Pierce<\/a>:<\/p>\n

Wild how much the Overton window has moved that Giannandrea can just say, “Yeah, we trained on the public web,” and it’s not even a thing. I mean, of course it did. That’s what everyone did! But wild that we don’t even blink at that now.<\/p><\/blockquote>\n\n

John Voorhees<\/a>:<\/p>\n

As a creator and website owner, I guess that these things will never sit right with me. Why should we accept that certain data sets require a licensing fee but anything that is found “on the open web” can be mindlessly scraped, parsed, and regurgitated by an AI? Web publishers (and especially<\/em> indie web publishers these days<\/a>, who cannot afford lawsuits<\/a> or hiring law firms to strike expensive deals<\/a>) deserve better.<\/p>

It’s disappointing to see Apple muddy an otherwise compelling<\/a> set of features (some of which I really want to try) with practices that are no better<\/a> than the rest of the industry<\/a>.<\/p><\/blockquote>\n\n

Colin Cornaby<\/a>:<\/p>\n

The justification of “if you posted it on the public web - it’s ok for us to train AI on” is really bizarre - and not completely legally sound? Posting something on the public web doesn’t mean you surrender the copyright.<\/p>

That’s actually exactly the basis of the NYT’s suit against OpenAI. The NYT proved that OpenAI was able to reproduce articles that it had scraped from the NYT.<\/p><\/blockquote>\n\n

Apple<\/a>:<\/p>\n

\n

With Applebot-Extended, web publishers can choose to opt out of their website content being used to train Apple’s foundation models powering generative AI features across Apple products, including Apple Intelligence, Services, and Developer Tools.<\/p>\n

[…]<\/p>\n

Applebot-Extended does not crawl webpages. Webpages that disallow Applebot-Extended can still be included in search results. Applebot-Extended is only used to determine how to use the data crawled by the Applebot user agent.<\/p>\n<\/blockquote>\n\n

The models were trained before<\/a> they told us<\/a> how to opt-out. If you update your robot.txt<\/tt> to exclude Applebot Extended, it’s not clear when your data will be removed from the models. It can take a long time to re-train a model, and I don’t know whether the on-device models are tied to OS updates.<\/p>\n\n

Joe Rosensteel<\/a>:<\/p>\n

\n

Literally the same presentation talks about protecting your privacy from unscrupulous internet companies. Your data is isolated by a whole auditable cloud solution and will never be used for modeling. BUT if that same Apple customer posted anything on the open web then it’s fair game for Apple to use regardless of copyright, licenses, or expectations. Doing it before anyone could ever object is all the more damning.<\/p>\n<\/blockquote>\n\n

Eric deRuiter<\/a>:<\/p>\n

Disabling Apple AI via robots.txt is not supported on Squarespace as you can’t edit your own robots.txt file.<\/p><\/blockquote>\n

Apple does offer a way to opt out entirely via a <meta><\/code> tag, but I don’t see a way to use that to exclude only the AI stuff.<\/p>\n\n

Dan Moren<\/a>:<\/p>\n

\n

To test this out, I’ve added those directives to my personal site<\/a>. This turned out to be slightly more confusing, given that my site runs on WordPress, which automatically generates a robots.txt<\/code> file. Instead, you have to add the following snippet of code to your functions.php<\/code> file by going to the administration interface and choosing Appearance > Theme File Editor and selecting functions.php from the sidebar.<\/p>\n

[…]<\/p>\n

If you want to go beyond Apple, this same general idea works for other AI crawling tools as well. For example, to block ChatGPT from crawling your site<\/a> you would add a similarly formatted addition to the robots.txt<\/code> file, but swapping in “GPTBot” instead of “Applebot-Extended.”<\/p>\n

Google’s situation is more complex: while the company does have a Googlebot-Extended that powers some of its AI tools, like Gemini (née Bard), blocking that won’t necessarily remove your site’s content from being crawled for use in Google’s AI search features<\/a>. To do that, you’d need to block Googlebot entirely, which would have the unfortunate effect of removing your site from its search indexes as well.<\/p>\n<\/blockquote>\n\n

Robb Knight<\/a> (via Nick Heer<\/a>, Hacker News<\/a>):<\/p>\n

\n

[Perplexity is] using headless browsers to scrape content, ignoring robots.txt, and<\/em> not sending their user agent string. I can't even block their IP ranges because it appears these headless browsers are not on their IP ranges<\/a>.<\/p>\n<\/blockquote>\n\n

John Voorhees<\/a>:<\/p>\n

\n

Over the past several days, we’ve made some changes at MacStories to address the ingestion of our work by web crawlers operated by artificial intelligence companies. We’ve learned a lot, so we thought we’d share what we’ve done in case anyone else would like to do something similar.<\/p>\n<\/blockquote>\n\n

Previously:<\/p>\n