Archive for June 19, 2024

Wednesday, June 19, 2024

Apple Intelligence Training

Apple:

In the following overview, we will detail how two of these models — a ~3 billion parameter on-device language model, and a larger server-based language model available with Private Cloud Compute and running on Apple silicon servers — have been built and adapted to perform specialized tasks efficiently, accurately, and responsibly.

[…]

Our foundation models are trained on Apple’s AXLearn framework, an open-source project we released in 2023. It builds on top of JAX and XLA, and allows us to train the models with high efficiency and scalability on various training hardware and cloud platforms, including TPUs and both cloud and on-premise GPUs. We used a combination of data parallelism, tensor parallelism, sequence parallelism, and Fully Sharded Data Parallel (FSDP) to scale training along multiple dimensions such as data, model, and sequence length.

We train our foundation models on licensed data, including data selected to enhance specific features, as well as publicly available data collected by our web-crawler, AppleBot.

David Pierce:

Wild how much the Overton window has moved that Giannandrea can just say, “Yeah, we trained on the public web,” and it’s not even a thing. I mean, of course it did. That’s what everyone did! But wild that we don’t even blink at that now.

John Voorhees:

As a creator and website owner, I guess that these things will never sit right with me. Why should we accept that certain data sets require a licensing fee but anything that is found “on the open web” can be mindlessly scraped, parsed, and regurgitated by an AI? Web publishers (and especially indie web publishers these days, who cannot afford lawsuits or hiring law firms to strike expensive deals) deserve better.

It’s disappointing to see Apple muddy an otherwise compelling set of features (some of which I really want to try) with practices that are no better than the rest of the industry.

Colin Cornaby:

The justification of “if you posted it on the public web - it’s ok for us to train AI on” is really bizarre - and not completely legally sound? Posting something on the public web doesn’t mean you surrender the copyright.

That’s actually exactly the basis of the NYT’s suit against OpenAI. The NYT proved that OpenAI was able to reproduce articles that it had scraped from the NYT.

Apple:

With Applebot-Extended, web publishers can choose to opt out of their website content being used to train Apple’s foundation models powering generative AI features across Apple products, including Apple Intelligence, Services, and Developer Tools.

[…]

Applebot-Extended does not crawl webpages. Webpages that disallow Applebot-Extended can still be included in search results. Applebot-Extended is only used to determine how to use the data crawled by the Applebot user agent.

The models were trained before they told us how to opt-out. If you update your robot.txt to exclude Applebot Extended, it’s not clear when your data will be removed from the models. It can take a long time to re-train a model, and I don’t know whether the on-device models are tied to OS updates.

Joe Rosensteel:

Literally the same presentation talks about protecting your privacy from unscrupulous internet companies. Your data is isolated by a whole auditable cloud solution and will never be used for modeling. BUT if that same Apple customer posted anything on the open web then it’s fair game for Apple to use regardless of copyright, licenses, or expectations. Doing it before anyone could ever object is all the more damning.

Eric deRuiter:

Disabling Apple AI via robots.txt is not supported on Squarespace as you can’t edit your own robots.txt file.

Apple does offer a way to opt out entirely via a <meta> tag, but I don’t see a way to use that to exclude only the AI stuff.

Dan Moren:

To test this out, I’ve added those directives to my personal site. This turned out to be slightly more confusing, given that my site runs on WordPress, which automatically generates a robots.txt file. Instead, you have to add the following snippet of code to your functions.php file by going to the administration interface and choosing Appearance > Theme File Editor and selecting functions.php from the sidebar.

[…]

If you want to go beyond Apple, this same general idea works for other AI crawling tools as well. For example, to block ChatGPT from crawling your site you would add a similarly formatted addition to the robots.txt file, but swapping in “GPTBot” instead of “Applebot-Extended.”

Google’s situation is more complex: while the company does have a Googlebot-Extended that powers some of its AI tools, like Gemini (née Bard), blocking that won’t necessarily remove your site’s content from being crawled for use in Google’s AI search features. To do that, you’d need to block Googlebot entirely, which would have the unfortunate effect of removing your site from its search indexes as well.

Robb Knight (via Nick Heer, Hacker News):

[Perplexity is] using headless browsers to scrape content, ignoring robots.txt, and not sending their user agent string. I can't even block their IP ranges because it appears these headless browsers are not on their IP ranges.

John Voorhees:

Over the past several days, we’ve made some changes at MacStories to address the ingestion of our work by web crawlers operated by artificial intelligence companies. We’ve learned a lot, so we thought we’d share what we’ve done in case anyone else would like to do something similar.

Previously:

Update (2024-06-20): Nick Heer:

The question seems to be whether what Perplexity is doing ought to be considered crawling. It is, after all, responding to a direct retrieval request from a user. This is subtly different from how a user might search Google for a URL, in which case they are asking whether that site is in the search engine’s existing index. Perplexity is ostensibly following real-time commands: go fetch this webpage and tell me about it.

But it clearly is also crawling in a more traditional sense. The New York Times and Wired both disallow PerplexityBot, yet I was able to ask it to summarize a set of recent stories from bothpublications. At the time of writing, the Wired summary is about seventeen hours outdated, and the Times summary is about two days old. Neither publication has changed its robots.txt directives recently; they were both blocking Perplexity last week, and they are blocking it today. Perplexity is not fetching these sites in real-time as a human or web browser would. It appears to be scraping sites which have explicitly said that is something they do not want.

Perplexity should be following those rules and it is shameful it is not. But what if you ask for a real-time summary of a particular page, as Knight did? Is that something which should be identifiable by a publisher as a request from Perplexity, or from the user?

Update (2024-06-24): John Gruber:

Apple should clarify whether they plan to re-index the public data they used for training before Apple Intelligence ships in beta this summer. Clearly, a website that bans Applebot-Extended shouldn’t have its data in Apple’s training corpus simply because Applebot crawled it before Apple Intelligence was even announced. It’s fair for public data to be excluded on an opt-out basis, rather than included on an opt-in one, but Apple trained its models on the public web before they allowed for opting out.

But other than that chicken/egg opt-out issue, I don’t object to this. The whole point of the public web is that it’s there to learn from — even if the learner isn’t human.

Louie Mantia (via Federico Viticci):

This is a critical thing about ownership and copyright in the world. We own what we make the moment we make it. Publishing text or images on the web does not make it fair game to train AI on. The “public” in “public web” means free to access; it does not mean it’s free to use.

Besides that, I’d also add what I’ve seen no one else mention so far: People post content on web that they don’t own all the time. No one has to prove ownership to post anything.

Someone who publishes my work as their own (theft) or republishes my work (like quoting or linking back) doesn’t have the right to make the choice for me to let my content be used for training AI.

That same argument would also apply to indexing for search.

Update (2024-07-19): Tim Hardwick:

[Apple] emphasized that since OpenELM is not integrated into Apple Intelligence, the “YouTube Subtitles” dataset is not being used to power any of its commercial AI features.

Update (2024-07-30): John Voorhees:

If you still had doubts whether Apple scraped the web to build its foundation model and only gave publishers an option to opt-out after the fact, it’s all laid out here.

Steve Troughton-Smith:

Apple clearly has vacuumed up data from European websites and open-source projects to build its Foundation Models, which makes it incredibly distasteful for them to be trying to hold Apple Intelligence hostage as a bargaining chip against EU regulation.

If for some reason regulators were to angrily demand an immediate purge or audit of the affected data, it could set Apple Intelligence back years and push it well out of the iOS 18 timeframe.

Hartley Charlton:

The paper reveals that Apple utilized 2,048 of Google’s TPUv5p chips to build AI models and 8,192 TPUv4 processors for server AI models. The research paper does not mention Nvidia explicitly, but the absence of any reference to Nvidia’s hardware in the description of Apple’s AI infrastructure is telling and this omission suggests a deliberate choice to favor Google’s technology.

Customizable Control Center and Lock Screen Buttons in iOS 18

Tim Hardwick:

In iOS 18, when you invoke Control Center with a swipe down from the top-right of your iPhone’s screen, you can continuously swipe to get to more control center screens. There’s a new controls gallery that lets you customize the controls you see, and you can change the size of buttons to emphasize priority. Additionally, developers can include controls from their own apps, enabling quick access to controls like remotely starting a car, for example.

Steven Aquino (Mastodon):

In the current Face ID era, which dates back to the 2017 introduction of the iPhone X, turning off an iPhone or iPad entails a high-tech game of Twister. One must hit the volume buttons in quick succession while simultaneously holding the side button. Depending on the level of one’s fine-motor skills, performing these movements can be infuriating if not impossible.

[…]

In a broad scope, it’s worth reiterating the notion that this new Power button in iOS 18’s Control Center is hugely beneficial as a de-facto accessibility feature.

Juli Clover:

iOS 18 includes a new LockedCameraCapture framework for developers, which will bring some useful new functionality to third-party apps with camera features. The update will let a third-party camera app be opened directly from the Lock Screen.

This is great. It does not, however, let you customize which app opens when you swipe.

Simon B. Støvring:

Thanks for making the Control Center more configurable, Apple, so I can move the controls to an area I can reach with my thumb. Now, please make it so the Control Center is opened from a reachable area on the screen.

Previously:

Update (2024-06-20): John Spurlock:

As someone that worked on the Android lock screen to camera transition in a previous life, I don’t see Apple ever letting you do this.

Tight integration is needed to ensure that the transition looks great, and the app launches quickly.

The Camera app is probably the last app you’d pick to launch at a moment’s notice - needs gobs of working memory and fires up multiple hardware sensors.

Tab Bar and Sidebar in iPadOS 18

Tim Hardwick:

In iPadOS 18, Apple has redesigned the tab bar experience for many apps like Podcasts, Apple TV, and Apple Music. Previously in these apps, tabs sat at the bottom of the interface. When in portrait orientation however, tab bars now sit at the top of these apps to be closer to other navigation controls, while a customizable sidebar appears in landscape mode.

[…]

To accompany the new tab bars, sidebars in apps have been updated to help users navigate apps and provide quick access to top-level destinations.

Together, side bars and tab bars support user customization, so it's now possible to drag and drop items from the sidebar into a pinned section on the tab bar for even quicker access to favorite content.

Nico Reese (via Steve Troughton-Smith):

Apple has made significant updates to UITabBarController, which now supports displaying a sidebar as well. By default, the tab bar floats at the top of the screen and uses only text labels, essentially placed in the navigation bar’s title view area. Optionally, developers can add a sidebar button to the tab bar. When tapped, the tab bar morphs into a sidebar on the left side of the screen.

Unfortunately, this change brings many complex issues, some of which I want to discuss as I try to integrate these updates into my app, Gamery. These issues are particularly frustrating for me as a developer attempting to implement this new UI.

When sidebars were first introduced in iPadOS 14, it seemed like Apple was moving the iPad experience closer to that of a desktop. It was a step in the right direction. Now, it feels like they are regressing, making the app experience confusing and unnecessarily complex.

Federico Viticci:

Earlier today on Mastodon, I shared some concerns regarding the Books app in iPadOS 18 and how Apple implemented the new tab bar design in the app. Effectively, by eschewing a sidebar, the app has returned to feeling like a blown-up iPhone version – something I hoped we had left behind when Apple announced they wanted to make iPad apps more desktop-class two years ago.

[…]

[S]ince tab bars now sit alongside toolbar items, the entire UI can get very condensed, with buttons often ending up hidden away in an overflow menu.

[…]

If the goal was to simplify the iPad’s UI, well, now iPad users will end up with three ways to navigate apps instead of two, with the default method (the top bar) now generally displaying fewer items than before, without glyphs to make them stand out.

Kyle Howells:

This new combined side bar and tab bar design just seems to make both the tab bar and the side bar worse and the iPad even more limited….

The 2 big UI changes this year, Photos & the new tab bar and both flops that I’m hoping will be reverted.

Previously:

Update (2024-08-07): Andy Lee:

Is iPadOS really removing the search field from the tabs view? I’ve been assuming it was a temporary omission while they rework other things. If it hasn’t been restored in the next beta I will have to write some strongly worded… uh… words.

Previously:

Update (2024-10-17): Léo Natan:

The news sidebar implementation in “iPad”OS is absolutely terribly broken, by design and by implementation. First of all, it’s completely broken. Throws layout constraint failures, overlaps the navigation title, doesn’t trim properly, crashes from broken collection view updates.

But that is only half the problem. By design, it is half-assed at most. The resulting sidebar is not a controller, so there is no regular <-> compact size handling of any sort. After all, why would Apple support responsive design?

So developers that wish to implement this idiom, still need to keep their old sidebar controller, and then listen to trait collection changes and modify their navigation controller to have the old sidebar as the root, while removing it for large layouts.

And even that is broken, as the tab bar attempts to put the overflowing tabs as “more” controllers from iOS 6 fame days, and, you guessed it, that functionality is completely broken too, and cannot be disabled.

Arvind, RIP

Adam Zewe (via Hacker News):

A prolific researcher who led the Computation Structures Group in the Computer Science and Artificial Intelligence Laboratory (CSAIL), Arvind served on the MIT faculty for nearly five decades.

[…]

As a scientist, Arvind was well known for important contributions to dataflow computing, which seeks to optimize the flow of data to take advantage of parallelism, achieving faster and more efficient computation.

In the last 25 years, his research interests broadened to include developing techniques and tools for formal modeling, high-level synthesis, and formal verification of complex digital devices like microprocessors and hardware accelerators, as well as memory models and cache coherence protocols for parallel computing architectures and programming languages.

Wikipedia:

In 2000, Arvind took two years off from teaching at MIT to build Sandburst, Inc, a fabless manufacturing semiconductor company. He served as its president until his return to MIT in 2002. In 2006, Sandburst was acquired by Broadcom Corporation.

In 2003, he cofounded Bluespec, Inc., headquartered in Waltham, Massachusetts. They produce proven electronic design automation (EDA) synthesis toolsets. With Lennart Augustsson, Arvind codeveloped the programming language Bluespec SystemVerilog (BSV), a high-level functional programming hardware description language, which is a Haskell variant extended to handle chip design and electronic design automation in general.

I really enjoyed his graduate Computer System Architecture class. He was so excited about the material and clear in communicating it, both verbally and notationally using term-rewriting systems.