Apple’s Information Systems & Technology Division
Alex Kantrowitz (MacRumors, Hacker News):
IS&T is made up largely of contractors hired by rival consulting companies, and its dysfunction has led to a rolling state of war. “It’s a huge contractor org that handles a crazy amount of infrastructure for the company,” one ex-employee who worked closely with IS&T told me. “That whole organization is a Game of Thrones nightmare.”
Interviews with multiple former IS&T employees and its internal clients paint a picture of a division in turmoil, where infighting regularly prevents the creation of useful software, and whose contract workers are treated as disposable parts.
[…]
When IS&T’s projects are finally completed, they can cause even more headaches for Apple employees, who are left with a mess to clean up. Multiple people told me their Apple colleagues were forced to rewrite code after IS&T-built products showed up broken.
From what I’ve heard, this is a longtime problem, and it’s a mystery to me why this group has been immune to the Cook Doctrine. Apple buys forests to manage the paper used in its packaging and designs the desks its employees use and even the pizza boxes for its cafeteria. But when it comes to building the software that runs the company, that’s not considered a core competency.
Previously:
- Where to Get Apple Products Repaired
- Feedback Assistant Replaces Bug Reporter
- Apple Contractors “Regularly Hear Confidential Details” on Siri Recordings
Update (2020-04-08): chubot:
If you’ve worked in a huge org like Apple or the government, you’ve seen incredible inefficiencies. How can all these smart and often well-paid people be doing something so wrong (i.e. wasting time with political battles, etc.)?
But that’s only true if you look at the “medium view” of the organization. If you look at the large view, how the money actually flows, then it might be beneficial for one part like IS&T to be kind of broken, as long as the rest of the company works.
I have seen this dynamic at other companies. Internal tools can sometimes hold too much leverage over the organization. They can almost “blackmail” people into getting their way because they have a literal monopoly over what they do. It might be better to let multiple teams fight battles amongst themselves, which seems inefficient if you’re working as a regular joe, but could be efficient from the CEO’s perspective.
I was part of a team inside Apple that developed internal-facing tools for the retail teams. Our team was formed because of the extremely high cost in dollars and time that IS&T wanted to charge to develop some fairly straightforward tools. My team was taken from the departments that used the tools, and we developed things that were very custom-fit to what those departments needed.
Eventually politics overcame us, and IS&T finally managed to take over our team. We all became contractors in order to support the migration to their infrastructure, and continue development of the tools. A bit later we all became fired as our project was outsourced to contractors in India, managed by IS&T PMs who had no idea what our tool was used for and had never been inside a retail store. The tools all died shortly after that, some killed by IS&T, some petered out due to lack of use now that they no longer worked correctly or were a good fit for the users.
As far as we could tell, IS&T was run as a unique company inside Apple, which did it’s fair share of price gouging in order to make itself money to keep going. Its purpose never appeared to be helping Apple customers or Apple employees, it was simply to get bigger, absorbing more money and power wherever possible, with no apparent reigning in from the parent org.
I had a similar experience w/Apple. A startup I worked for was brought into Cupertino for a meeting w/their internal business teams. They wanted us to build them an app which seemed relatively simple involving their internal Cafe Mac cafeterias, data centers, and possibly retail locations. It was something that a company like Apple could build in their sleep. But the business folks we met with said that’s how it is at Apple, all the engineering talent goes towards the product side and almost nothing is left for internal IT. They told us how they struggled to get anything done and there were almost no resources available, so the business teams had to go and hire their own IT if they needed things done.
Is there any big consumer product company out there that doesn’t deprioritize management and engineering talent for internal tools and systems ?
Former IS&T. It’s been that way since 2003 and pretty sure long before then. Cook even made it worse at one point where he forced all non product dev team to be inside IS&T. Not sure if it’s changed, but at the time they were managed by someone that didn’t want to change.
See also: Hacker News, John Gruber, Slashdot.
Personally I find the discussion about IS&T less interesting than how SWE is broken at Apple — e.g., the engineering quality and resources in Eddy-org vs Craig-org, etc
and yes IS&T tools are trash but some of the SWE tools are just as bad
One of my favourite examples about Apple’s broken SWE tooling is the insanely bad crash reporting website that performs like it runs on a Mac Mini on an engineer’s desk.
Meanwhile Uber, with 1/20th of Apple’s resources, has crash analytics and reporting infra that is years ahead
There’s a lot of talk about Apple’s IS&T group dysfunction without referencing how it’s relevant to any of us: IS&T run Radar. If you’ve wondered why it’s been so hard to improve Radar for external devs, there is a big part of the reason — we’re not IS&T’s customers
See also: John Gruber (tweet).
A “friend” who works at the local Apple Store constantly complains about the bugginess of the POS software on the handheld devices. “Like sending soldiers into battle with rubber swords” is a memorable quote.
20 Comments RSS · Twitter
Are the feedbackassistant.apple.com website and developer forums IS&T projects?
Appstore Connect must be an IS&T project.
@someone @Iggy From what I’ve heard, Feedback/Radar and App Store Connect are not IS&T—and are actually quite a bit better than the software the book is talking about.
Michael, my info is about 12 years out of date, so I don't know about App Store Connect or Feedback Assistant, but back in day IS&T definitely developed Radar and RadarWeb.
@Jeff Perhaps the answer is that Radar is organizationally under IS&T but has its own team of Apple employees, not contractors? Pretty sure App Store Connect is under the Services group.
Well, again, this is from 12 years ago, but IS&T had a skeleton crew of Apple employees who were nominally running the show, while most of the work, including Radar, was done by contractors. At its heart, Radar is "simply" an Oracle database, with various interfaces. The Radar Cocoa app is one interface. RadarWeb was another interface. I don't know what's happening there now, but from the sound of the article, I doubt much has changed.
>all the engineering talent goes towards the product side
>and almost nothing is left for internal IT
That's what this is.
This is actually a pretty difficult problem to solve. If a software company has a team that produces software for external clients, software which directly creates revenue for the company, and a team that produces internal software, which doesn't create revenue, you naturally create a two-class system of developers.
The developers who create revenue will always be more valuable, which eventually will result in better wages, higher bonuses, better job security, better career path, and so on. The revenue teams will hire the best engineers away from the non-revenue teams, and the non-revenue teams will fight for budget and talent. Any time the company needs to cut costs, the non-revenue teams will be fired first, and their software will be abandoned. You will almost inevitably end up with your lowest-performing developers with the least experience in your internal software team. You will have high turnover, and you will have trouble hiring.
What Apple is doing here appears to be their broken attempt at avoiding this issue.
@Lukas That definitely makes sense in a narrow way, but you could also say that good tools are a force multiplier (or bad tools could be a hindrance) and that one of the best things Apple could do for its productivity (and, thus, eventually revenue) would be to invest in tools to help its employees. One might call good knowledge tools a bicycle for the mind. This is especially true in software development where the Mythical Man-Month applies.
It’s likely (but if it’s not I’d love to hear otherwise) that Apple’s HR and other internal systems run on big ERP systems like Oracle, and not on tailor made software.
A system like that typically is tailor-made software. Yes, there’s a set of big shared base modules, but the details involve a ton of custom code. You don’t really buy a module like that, install it, and are done. You hire a vendor-specialized consultant.
Are the feedbackassistant.apple.com website and developer forums IS&T projects?
My understanding is that, yes, Radar (for which Feedback Assistant is a front-end) is maintained by contractors.
That definitely makes sense in a narrow way, but you could also say that good tools are a force multiplier (or bad tools could be a hindrance) and that one of the best things Apple could do for its productivity (and, thus, eventually revenue) would be to invest in tools to help its employees. Kind of like a bicycle for the mind. This is especially true in software development where the Mythical Man-Month applies.
The other point in defense of in-house “consulting” work is dogfooding. If the Cocoa Radar app were built by employees within Apple (which I understand it is not), that would help the feedback loop to further inform in which directions Cocoa needs to improve.
Microsoft seems particularly bad at this aspect — e.g., Teams is built in Electron; Office is largely built in its own UI framework; relatively little is built in WPF (basically just Visual Studio, and Zune back in the day) or Windows Forms. With WinUI, they’re slowly starting to get better at this.
Apple is arguably faring a lot better, being one of the biggest adopters of AppKit and UIKit (and Catalyst) itself. So maybe the dogfooding problem isn’t perceived as big at Apple.
This is 13-15 years ago, but my experience with IS&T and "on-boarding" was that it just kinda worked. IS&T was positioned as support for Apple corporate (engineering, marketing, product, etc) employees. and no, Radar wasn't IS&T. Only the stuff that let these folks get on with their day to day (email accounts, badging permissions, that kind of stuff) was the purview of IS&T. Radar was an engineering thing.
Never heard if "Isn't" pronunciation.
IS&T was just a black box, but I just don't recall much of an issue. Maybe I had low expectations because of other places, but when I had need of things, I got things in unremarkable (meaning reasonable) amounts of times.
People claiming IS&T is exclusively internal-facing are wrong. IS&T builds Apple's online and mobile storefronts.
If Radar is IS&T, that explains a lot. Also explains why Apple may be terrible at services, if the back end is supported by them.
>you could also say that good tools are a force multiplier
Just to be clear, I agree with that. And that's how it works in many smaller companies. You can make these kinds of arguments in a high-trust company. But as the company grows, and more and more people are around, many of which were not hired by the current leadership, and were sometimes hired by people who themselves aren't around anymore, internal trust starts to go down.
At some point, ledership changes happen (e.g. an external CTO is hired to fix the finances), leadership starts to institute "objective" performance measures, and a lot of social dysfunction starts to crop up. At this point, arguments like "good tools are a force multiplier" are much harder to make, because you can't really measure that. That's how you end up with a lot of corporate dysfunction (stack ranking, for example).
At this point, you often end up measuring team value by revenue impact.
If you work in a 10000+ employee company like Apple, it's very difficult to make an argument like "this is a force multiplier, we should invest in it." You'll be asked how you will measure your impact, and that's often difficult to answer with an internal tool. And even if you can get it through one year, a few years later, things will change, and your team will be the first to be cut.
I worked at retail at Apple for 10 years. One of my biggest frustrations as an employee was simply getting vacation time or switching shifts with coworkers. Even getting our schedule was a pain, we had to log in behind the retail firewall to see our shifts and then write them down in our phones or calendar. I kept thinking “I can’t believe they don’t just publish a calendar we can subscribe to” or “why isn’t there a schedule manager app where I can switch shifts with my fellow employees?” Now it makes sense... no talent to be wasted internally. And I guess it worked “good enough” because we were forced to use it. Supposedly they did fix this like a year ago. I just couldn’t believe a company with the best software in the world is using such horrible software for building a schedule.
"You'll be asked how you will measure your impact, and that's often difficult to answer with an internal tool"
True, but how do you measure the impact of working in an enormously expensive giant circle?
So Apple is every big company ever? I would find it more surprising if they were indeed much different.
IS&T will support the Shared Services part of the organisation as well as retail.
They will have Enterprise systems like SAP, SalesForce etc and run on a cloud service like AWS.
Organisations of Apple’s size will use Accenture for SAP Projects. They will use Indian outsourcers for support.
These organisations will compete and step on each other’s toes. It’s the Apple contract managers and governance the should be deployed to ensure that it works smoothly.
However, it is a difficult task as it usually comes down to kpis. Guess what? It’s very difficult to measure quality.
In my experience outsourcing for commodity things like infrastructure and desktop support works but projects are problematic.
Looks like my unflattering comment about the "Cook Doctrine" and IS&T being little India got deleted.
Where's the lie? Apple employees had to train their replacements Tata & Wipro (and other) and all of Apple suffered and continues to do so. It's endemic to Silly Valley, this isn't just $AAPL.
If you see fraud and don't say fraud... bah nm all these "black swans" have some home to roost.
[…] It’s A „Cold War Every Day“ Inside This Group At Apple. Michael Tsai o istoj temi. […]
It’s likely (but if it’s not I’d love to hear otherwise) that Apple's HR and other internal systems run on big ERP systems like Oracle, and not on tailor made software. Typical Apple employees will have neither experience nor affinity for this kind of work, which is often very far removed from modern development techniques. It wouldn’t be surprising that keeping theses consultancies' contract open is a way to stay in their good books, Accenture for instance being one of Apple's partner in pushing Macs to corporate clients.