Apple Card’s Outsourced Algorithm
David Heinemeier Hansson (video):
The @AppleCard is such a fucking sexist program. My wife and I filed joint tax returns, live in a community-property state, and have been married for a long time. Yet Apple’s black box algorithm thinks I deserve 20x the credit limit she does. No appeals work.
It seems like a bit of an overreaction because he’s extrapolating from two data points that are not independent. There could be a reasonable explanation, or simply a bug. But it’s worth asking whether the algorithm is fair in aggregate, and many other customers have chimed in with similar experiences. This story also points to other issues about black box algorithms and AI: whether anyone actually understands the algorithm, the desire for explanations and appeal in individual cases, special treatment for some, and Apple’s role in the card that bears its name.
I care about transparency and fairness. It’s why I was deeply annoyed to be told by AppleCard representatives, “It’s just the algorithm,” and “It’s just your credit score.” I have had credit in the US far longer than David. I have never had a single late payment. I do not have any debts. David and I share all financial accounts, and my very good credit score is higher than David’s. […] But AppleCard representatives did not want to hear any of this. I was given no explanation. No way to make my case.
There wasn’t even a way to check and correct the inputs to the algorithm.
I care about justice for all. It’s why, when the AppleCard manager told me she was aware of David’s tweets and that my credit limit would be raised to meet his, without any real explanation, I felt the weight and guilt of my ridiculous privilege.
If only it were so easy for a manager to snap their fingers and make other computing problems go away. But with cloud syncing and other modern bugs, there’s often no explanation or solution. People accept brokenness because they feel they have no choice. There are no legal requirements for transparency, nor culture of it.
And the best defense of all: THAT’S JUST HOW THE CREDIT INDUSTRY WORKS. Ehh, okay? How is that anything but the most damning charge upon Apple’s pitch with their card? Did the iPhone launch pledging to please carriers and the status quo as its modus operandi? No.
Apple offers a credit card that bases its credit assessment on a black-box algorithm that 6 different reps across Apple and GS have no visibility into. Even several layers of management. An internal investigation. IT’S JUST THE ALGORITHM!
So nobody understands THE ALGORITHM. Nobody has the power to examine or check THE ALGORITHM. Yet everyone we’ve talked to from both Apple and GS are SO SURE that THE ALGORITHM isn’t biased and discriminating in any way.
[…]
The algorithm might discriminate out of biased historical training data, faulty but uncorrectable inputs, programming errors, or malicious intent. You’ll never be able to know.
Sridhar Natarajan and Shahien Nasiripour (tweet, Hacker News):
A Wall Street regulator is opening a probe into Goldman Sachs Group Inc.’s credit card practices after a viral tweet from a tech entrepreneur alleged gender discrimination in the new Apple Card’s algorithms when determining credit limits.
We suspect that the Goldman algorithm was trained on data that included an important bias: that the husband is the primary card holder in traditional credit card approval. This biased the data so the algorithm assigned higher creditworthiness to the primary card holder. This meant that the primary card holder status became the proxy for gender.
The whole situation was made worse by a number of applications coming from a demographic or group that exposed this bias, an AI-enabled product which broke the mental model of Apple family sharing and a total lack of a “human-in-the-loop” recovery combined with unexplainable and non-intuitive AI.
They delegated the most central question of the entire product – CAN I USE THIS? – to a status-quo player with a checkered moral past.
The same thing happened to us. I got 10x the credit limit. We have no separate bank or credit card accounts or any separate assets. Hard to get to a human for a correction though. It’s big tech in 2019.
My @Apple Card got rejected. The reason appears to be that @keybank recently closed a checking account that got automatically opened when I got my mortgage and I never ever used!
Not a great first impression from every party involved in this.
Apple likes to promote that their card was “Created by Apple, not a bank”. So please, stop with the “THIS IS GOLDMAN’S FAULT” shit. The card is called THE APPLE CARD. Sending out GS spokespeople to deflect on their responsibility and ownership is cowardly.
[…]
Imagine if you had to call Foxconn yourself when your iPhone broke? If it says Apple on the box, it’s an Apple product.
“Apple Card completely rethinks everything about the credit card. It represents all the things Apple stands for. Like simplicity, transparency, and privacy.” - I couldn’t make this up! This is literally the top pitch on the @AppleCard marketing page.
Update (2019-11-27): Josh Centers:
Apple Card not offering joint accounts is a real liability that’s blowing up in Apple’s and Goldman Sachs’ collective faces. Goldman Sachs has said that the low-limit problem is due to applicants with a “limited credit history,” like people who piggyback on their spouse’s credit lines. It’s likely this wouldn’t have made the news at all if the Apple Card had simply offered joint accounts from the beginning, like nearly every other credit card.
Update (2020-01-24): Apple Card:
A new kind of credit card. Created by Apple and built for iPhone.
Pretty amazing that Apple still runs this campaign after Goldman Sachs has repeated stated that, no, this is not a card created by Apple. This is a GS card, with a discriminatory black-box algorithm, and they make all the decisions. All Apple provides is their brand and an app.
10 Comments RSS · Twitter
This is not an Apple or GS problem. The US credit (card) market is completely fucked up. This is not Apple can fix.
@Jean-Daniel I completely agree. And I dont understand why there is such as big fuss about it. "My wife and I filed joint tax returns, live in a community-property state, and have been married for a long time.". A Joint Tax Returns means absolutely nothing, a community property state means nothing. Everything that he was complaining about ( And I use Ruby Rails ) has absolutely nothing to do with it. Neither do the Credit Score. The Score only measure how clean and how likely you are going to repay your debt, not how much money they will put onto your line.
There wasn't a mention of his wife's income. Which I am sure DHH will complain it doesn't matter, it does. Unless your Credit Card is opened under a Family or a Joint Account, if you applied it yourself then only your income will be counted. I wonder what if her wife does money laundering, does it mean he is atomically responsible?
And it isn't some black box, someone inside the bank can tell you within all the reasons why they are doing it. But not only do they not have to, and the bank are under no obligation to do business everyone at risk higher than they allow.
If Apple isn't willing or able to make a better product, perhaps they shouldn't have entered that market.
I've always believed that banks are designed to screw you out of your own money and screw people over in general.
"I have never had a single late payment."
That statement is how we'd all like the world to be, but it's not how issuing banks work. One or two (or even a few more) late payments in an otherwise "good" payment record over some years are not even considered to be "bad." On the contrary, they're good — for the bank — because they generate extra (income/late penalty income). And if you're a really good credit card customer — one who pays monthly, but less than the entire balance — you will most likely have a higher limit than those deadbeats like me who pay in full every month. We only generate the 2 - 3% of purchases that banks can suck out of the merchants.
"I do not have any debts."
Now there's the real "problem." Someone who's making monthly mortgage payments, monthly car payments, and monthly credit card payments is a better credit "risk" (depending on their income) and will be given higher limits/lower loan interest rates than someone without that record. And if you used to have those outstanding loans but paid them all of more than five years ago, the credit rating system largely forgets about your "good" record.
If this all sounds as though the consumer is screwed no matter what they do.... well, yeah.
Credit scoring is a black box your that your financial information (credit lines, amounts, outstanding balances (even if you do not carry a balance at the end of the month, payment history, loan amounts and balances, etc) goes into. I don't know if they throw in other factors like insurance claims, driving records, legal issues, etc (all of this stuff is collected by other companies, especially insurance companies). Out of those items and probably others, pops a magic number along with a bunch of other information some of which is used to determine what your credit line should be.
@Nathan Apple is in it to make money as part of a recurring revenue stream, which is the model moving forward. The holy grail of business is the recurring bill (telephone, cable, etc). Same money coming in every month and since they sell items that are recurring (Apple Music, Apple+, Apple Arcade, and their part of the cost of competitors streaming plans purchased in the App Store) why not make a few pennies off of the transaction by offering a credit card knowing customers will sign up for it.
@Jean Daniel you nailed it. Credit Cards are a no good for anyone. Some may not know it, but all of these "rewards" are just your money being returned to you. Businesses factor in the cost of accepting credit cards into their pricing and it can range from less than .50 cents for a debit card to 3% for Visa, MasterCard, and Discover. Some of the speciality cards (high rewards) and American Express cards have even higher percentages. American Express is the worst and that is why many merchants will not accept them. On average it costs me just under 2% to accept a Visa, MasterCard or Discover credit (not debit) card. This varies too as different credit card processors charge different to the accepting merchant. Banks charge the highest merchant rates of all credit card processors which is why there are a lot of alternative processors who run leaner businesses and are happy to get a few less pennies on each swipe.
The monthly merchant statement for accepting credit cards makes phone and cable bills with a few taxes and add-ons look amateur. Merchant statements have dozens and dozens of charges that are a few cents each going to cover various fees and charges. Some are not decodable or explainable and the statement runs about 3 pages of microscopic print and only includes the grand total of charges accepted (the detail of each charge is not part of the statement).
Credit cards and their rewards programs, besides being a money hole for those who carry balances, are responsible for about 2% of the cost of every item purchased with them or with cash. You are getting stuff by using them, but you are paying for it.
Let's not forget that businesses are not legally allowed to charge more for credit card customers (they can only enforce a minimum total that must be met before you can use a card). So the fees get passed onto everyone, as a cost of running the business. So this makes using a credit card with rewards more of a no-brainer, because at least then you're getting something back. Paying cash, you get zero.
Look, this is a Tim Cook problem because Apple decided to go into financial services. Apple and Tim Cook have zero expertise in financial services, and they don’t even know how to do support or PR.
That much is obvious. Apple should stick to Computers.
