{"id":40676,"date":"2023-09-18T15:20:34","date_gmt":"2023-09-18T19:20:34","guid":{"rendered":"https:\/\/mjtsai.com\/blog\/?p=40676"},"modified":"2023-09-18T15:21:10","modified_gmt":"2023-09-18T19:21:10","slug":"apples-new-transformer-powered-predictive-text-model","status":"publish","type":"post","link":"https:\/\/mjtsai.com\/blog\/2023\/09\/18\/apples-new-transformer-powered-predictive-text-model\/","title":{"rendered":"Apple’s New Transformer-Powered Predictive Text Model"},"content":{"rendered":"

Jack Cook<\/a> (via Hacker News<\/a>):<\/p>\n

The feature will occasionally suggest more than one word at a time, but this is generally limited to instances where the upcoming words are extremely obvious, similar to the autocomplete in Gmail.<\/p>

[…]<\/p>

I have to say that this vocabulary file strikes me as pretty unique, but it’s definitely not out of the question for a language model deployed in this setting.\nI’ve personally never seen emojis featured so prominently in a language model’s tokenizer, but existing<\/a> research<\/a> has shown that domain-specific models and tokenizers can drastically improve downstream model performance.\nSo it makes sense that a model trained for use in things like text messages, in which emojis and contractions will be used a lot, would prioritize them.<\/p>

[…]<\/p>

GPT-2 has four main parts: token embeddings, positional encodings, a series of 12-48 decoder blocks, and an output layer.\nThe network described by unilm_joint_cpu<\/code> appears to be the same, except with only 6 decoder blocks.\nMost of the layers within each decoder block have names like gpt2_transformer_layer_3d<\/code>, which would also seem to suggest it’s based on a GPT-2 architecture.<\/p>

From my calculations based on sizes of each layer, Apple’s predictive text model appears to have about 34 million parameters, and it has a hidden size of 512 units.\nThis makes it much smaller than even the smallest version of GPT-2.<\/p><\/blockquote>\n

The early reports about auto-correct in iOS 17 and macOS 14 seem to be positive. I’m cautiously optimistic that it will fix the biggest problems for me with the old system, which are that it suggests words that are not spelled correctly and even changes correct words that I entered into mistakes.<\/p>\n\n

Previously:<\/p>\n