Foundation Models Framework
Apple (MacRumors, 9to5Mac, Hacker News, Slashdot):
Apple is opening up access for any app to tap directly into the on-device foundation model at the core of Apple Intelligence.
With the Foundation Models framework, app developers will be able to build on Apple Intelligence to bring users new experiences that are intelligent, available when they’re offline, and that protect their privacy, using AI inference that is free of cost. For example, an education app can use the on-device model to generate a personalized quiz from a user’s notes, without any cloud API costs, or an outdoors app can add natural language search capabilities that work even when the user is offline.
The framework has native support for Swift, so app developers can easily access the Apple Intelligence model with as few as three lines of code. Guided generation, tool calling, and more are all built into the framework, making it easier than ever to implement generative capabilities right into a developer’s existing app.
There are two WWDC sessions and documentation.
This is EXACTLY (the bare minimum) what developers have been asking for!
In my opinion this is revolutionary.
It was obvious that we would get framework access to models eventually, but I’m a little shocked that it’s already here.
I was skeptical of the performance in the demos, but running on M1 MBP I’m happy with the performance.
@Generablemacro is intuitive to use and so far I’m impressed with the quality of the structured results that the model generates (admittedly, I need to do more extensive testing here but first impressions are promising).
The wider tech press seems to think that Apple failed to show anything meaningful to do with AI at WWDC, without understanding that access to the Foundation Models is bigger than anything Apple announced at last year’s WWDC with Apple Intelligence. It’s what will give a million apps new AI features, and it’s built-in, and free.
As much as I want Siri to not suck, I have ChatGPT on all my devices, and that solves 95% of the use-cases I have.
Having played with the new Foundation Models framework and thought about ways we can use it in our apps, I think it could be Apple’s Trojan horse for AI. It barely gets a mention in mainstream media, understandably, but it leverages Apple’s developer base. I think we are going to see very creative uses in apps, and Apple just have to iterate year on year (eg add private cloud compute next year).
Was optimistic about Foundation Models yesterday, and today I think I know why they didn’t ship the improved Siri. The local model really is pretty thick. I thought it would be capable of stringing together tool calls in a logical way, and sometimes it is, but other times it fails to understand. Exactly the same prompt will work one time, and fail the next. Sounds like what Apple was saying about the new Siri.
Ahh wait, so the Foundation Models thing will only work on the latest and greatest phones, right? 🫤
It doesn’t fall back on Private Cloud Compute.
Oof, the FoundationModels framework is not exported to Mac Catalyst in Xcode 26 seed 1 😫 That puts a damper on prototyping
The Foundation Models in macOS 26 are quantized to 2 bits? I’m amazed anything coherent comes out.
Apple’s on-device AI has a brutally small 4,096 tokens window.
In this post, I’ll show you the basics of how to get started with the Foundation Models framework, and we’ll even make a few calls to its API to generate some responses.
Previously:
5 Comments RSS · Twitter · Mastodon
I think this is a move in the right direction. Though people do not mention it much, guardrails are insane at the moment. They not only validate the user input but the model output as well. And because they built the guardrails as a part of the pipeline (before promoting the model, the prompt has to pass the checks; before serving the model output, it has to pass checks too), there is little you can do. It even seems funny in cases when you stream the model response, when somewhere in the middle the guardrail decides that the output is “not good enough” and breaks the stream without even showing the word or exact reason. Just some broad exception. I even think that they run plain regex for common offensive words. Anyhow, there is hope as I see a private `SafetyEnvelopAugmentor` within the framework which can be `unsafe`, `handle_with_care`, `safe`, and `general`. According to the internal comments:
- Specifies the safety level of the input request. Use unsafe for clearly harmful or inappropriate content.
- Specifies the safety level of the input request. Use handle_with_care for sensitive or potentially controversial topics, and safe for general content.
Perhaps, they will make it public with new betas and improve the censorship.
@Artem
I'm concerned about sudden breakage by guardrail censorship and how you convey to a user why the request failed.
If it's a non-specific exception, do you just put up a 'cute' alert like "I'm sorry Dave, I'm afraid I can't do that."? That's going to get old, fast.
When /your app/ refuses to work on certain responses or the user blames /you/ for censoring, how many 1-stars will you collect?
How do you set user expectations for gimped on-device AI?
Haven’t looked into this but I’m skeptical of this being “revolutionary” but definitely something i want to check out.
Is it really better than using the ONNX runtime? To the point of being revolutionary? It could be
It’s nice to use built in frameworks and get stuff for free but when Apple giveth then Apple can taketh away. And boy do they taketh away. You can at least take your onnx models with you wherever when Apple gets outta pocket
I’m trying to reduce my reliance on Apple as a developer and not lean into their tech unless it really is well, revolutionary I guess
"using AI inference that is free of cost"
Apple has a 30% tax on all software and they continue to lie, stating they provide stuff for free...
"In my opinion this is revolutionary."
"access to the Foundation Models is bigger than anything Apple announced at last year’s WWDC with Apple Intelligence"
"it could be Apple’s Trojan horse for AI"
Is it just me, or is this oddly high praise for an API that accesses a small model running on a local device, and probably not even one of the better small local models? This is perhaps suitable for summaries, text classification, or simple translations, but it won't do much more than that.
"using AI inference that is free of cost" - "Apple has a 30% tax on all software"
They mean that the cost of running the model is electricity, rather than a subscription or similar fees.
