{"id":27276,"date":"2019-11-15T20:30:25","date_gmt":"2019-11-16T01:30:25","guid":{"rendered":"https:\/\/mjtsai.com\/blog\/?p=27276"},"modified":"2020-01-24T15:09:48","modified_gmt":"2020-01-24T20:09:48","slug":"apple-cards-outsourced-algorithm","status":"publish","type":"post","link":"https:\/\/mjtsai.com\/blog\/2019\/11\/15\/apple-cards-outsourced-algorithm\/","title":{"rendered":"Apple Card’s Outsourced Algorithm"},"content":{"rendered":"

David Heinemeier Hansson<\/a> (video<\/a>):<\/p>\n

\n

The @AppleCard is such a fucking sexist program. My wife and I filed joint tax returns, live in a community-property state, and have been married for a long time. Yet Apple’s black box algorithm thinks I deserve 20x the credit limit she does. No appeals work.<\/p>\n<\/blockquote>\n\n

It seems like a bit of an overreaction because he’s extrapolating from two data points that are not independent. There could be a reasonable explanation, or simply a bug. But it’s worth asking whether the algorithm is fair in aggregate, and many other customers have chimed in with similar experiences. This story also points to other issues about black box algorithms and AI: whether anyone actually understands the algorithm, the desire for explanations<\/a> and appeal in individual cases, special treatment for some, and Apple’s role in the card that bears its name.<\/p>\n\n

Jamie Heinemeier Hansson<\/a>:<\/p>\n

\n

I care about transparency and fairness. It’s why I was deeply annoyed to be told by AppleCard representatives, “It’s just the algorithm,” and “It’s just your credit score.” I have had credit in the US far longer than David. I have never had a single late payment. I do not have any debts. David and I share all financial accounts, and my very good credit score is higher than David’s. […] But AppleCard representatives did not want to hear any of this. I was given no explanation. No way to make my case.<\/p>\n<\/blockquote>\n\n

There wasn’t even a way to check and correct the inputs to the algorithm.<\/p>\n\n

\n

I care about justice for all. It’s why, when the AppleCard manager told me she was aware of David’s tweets and that my credit limit would be raised to meet his, without any real explanation, I felt the weight and guilt of my ridiculous privilege.<\/p>\n<\/blockquote>\n\n

If only it were so easy for a manager to snap their fingers and make other computing problems<\/a> go away. But with cloud syncing<\/a> and other modern bugs<\/a>, there’s often no explanation or solution. People accept brokenness because they feel they have no choice. There are no legal requirements for transparency, nor culture of it.<\/p>\n\n

David Heinemeier Hansson<\/a>:<\/p>\n

And the best defense of all: THAT’S JUST HOW THE CREDIT INDUSTRY WORKS. Ehh, okay? How is that anything but the most damning charge upon Apple’s pitch with their card? Did the iPhone launch pledging to please carriers and the status quo as its modus operandi? No.<\/p><\/blockquote>\n\n

Apple offers a credit card that bases its credit assessment on a black-box algorithm that 6 different reps across Apple and GS have no visibility into. Even several layers of management. An internal investigation. IT’S JUST THE ALGORITHM!<\/p>\n

So nobody understands THE ALGORITHM. Nobody has the power to examine or check THE ALGORITHM. Yet everyone we’ve talked to from both Apple and GS are SO SURE that THE ALGORITHM isn’t biased and discriminating in any way.<\/p>\n

[…]<\/p>\n

The algorithm might discriminate out of biased historical training data, faulty but uncorrectable inputs, programming errors, or malicious intent. You’ll never be able to know.<\/p>\n<\/blockquote>\n\n

Sridhar Natarajan and Shahien Nasiripour<\/a> (tweet<\/a>, Hacker News<\/a>):<\/p>\n

\n

A Wall Street regulator is opening a probe into Goldman Sachs Group Inc.’s credit card practices after a viral tweet from a tech entrepreneur alleged gender discrimination in the new Apple Card’s algorithms when determining credit limits.<\/p>\n<\/blockquote>\n\n

Sonder Scheme<\/a>:<\/p>\n

We suspect that the Goldman algorithm was trained on data that included an important bias: that the husband is the primary card holder in traditional credit card approval. This biased the data so the algorithm assigned higher creditworthiness to the primary card holder. This meant that the primary card holder status became the proxy for gender.<\/p>\n

The whole situation was made worse by a number of applications coming from a demographic or group that exposed this bias, an AI-enabled product which broke the mental model of Apple family sharing and a total lack of a “human-in-the-loop” recovery combined with unexplainable and non-intuitive AI.<\/p><\/blockquote>\n\n

David Heinemeier Hansson<\/a>:<\/p>\n

\n

They delegated the most central question of the entire product – CAN I USE THIS? – to a status-quo player with a checkered moral past.<\/p><\/blockquote>\n\n

Steve Wozniak<\/a>:<\/p>\n

\n

The same thing happened to us. I got 10x the credit limit. We have no separate bank or credit card accounts or any separate assets. Hard to get to a human for a correction though. It’s big tech in 2019.<\/p>\n<\/blockquote>\n\n

Daniel Vassallo<\/a>:<\/p>\n

My \n@Apple\n Card got rejected. The reason appears to be that \n@keybank\n recently closed a checking account that got automatically opened when I got my mortgage and I never ever used!<\/p>\n

Not a great first impression from every party involved in this.<\/p><\/blockquote>\n\n

David Heinemeier Hansson<\/a>:<\/p>\n

\n

Apple likes to promote that their card was “Created by Apple, not a bank”. So please, stop with the “THIS IS GOLDMAN’S FAULT” shit. The card is called THE APPLE CARD. Sending out GS spokespeople to deflect on their responsibility and ownership is cowardly.<\/p>\n

[…]<\/p>\n

Imagine if you had to call Foxconn yourself when your iPhone broke? If it says Apple on the box, it’s an Apple product.<\/p>\n<\/blockquote>\n\n

David Heinemeier Hansson<\/a>:<\/p>\n

\n

“Apple Card completely rethinks everything about the credit card. It represents all the things Apple stands for. Like simplicity, transparency, and privacy.” - I couldn’t make this up! This is literally the top pitch on the \n@AppleCard\n marketing page.<\/p>\n<\/blockquote>\n\n

Update (2019-11-27): Josh Centers<\/a>:<\/p>\n

\n

Apple Card not offering joint accounts is a real liability that’s blowing up in Apple’s and Goldman Sachs’ collective faces. Goldman Sachs has said that the low-limit problem is due to applicants with a “limited credit history,” like people who piggyback on their spouse’s credit lines. It’s likely this wouldn’t have made the news at all if the Apple Card had simply offered joint accounts from the beginning, like nearly every other credit card.<\/p>\n<\/blockquote>\n\n

Update (2020-01-24): Apple Card<\/a>:<\/p>\n

\n

A new kind of credit card. Created by Apple and built for iPhone.<\/p>\n<\/blockquote>\n\n

David Heinemeier Hansson<\/a>:<\/p>\n

\n

Pretty amazing that Apple still runs this campaign after Goldman Sachs has repeated stated that, no, this is not a card created by Apple. This is a GS card, with a discriminatory black-box algorithm, and they make all the decisions. All Apple provides is their brand and an app.<\/p>\n<\/blockquote>","protected":false},"excerpt":{"rendered":"

David Heinemeier Hansson (video): The @AppleCard is such a fucking sexist program. My wife and I filed joint tax returns, live in a community-property state, and have been married for a long time. Yet Apple’s black box algorithm thinks I deserve 20x the credit limit she does. No appeals work. It seems like a bit […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"apple_news_api_created_at":"2019-11-16T01:30:29Z","apple_news_api_id":"670b6184-05f7-4b8c-b1d4-0c68d53a7363","apple_news_api_modified_at":"2020-01-24T20:09:52Z","apple_news_api_revision":"AAAAAAAAAAAAAAAAAAAABA==","apple_news_api_share_url":"https:\/\/apple.news\/AZwthhAX3S4yx1Axo1TpzYw","apple_news_coverimage":0,"apple_news_coverimage_caption":"","apple_news_is_hidden":false,"apple_news_is_paid":false,"apple_news_is_preview":false,"apple_news_is_sponsored":false,"apple_news_maturity_rating":"","apple_news_metadata":"\"\"","apple_news_pullquote":"","apple_news_pullquote_position":"","apple_news_slug":"","apple_news_sections":"\"\"","apple_news_suppress_video_url":false,"apple_news_use_image_component":false,"footnotes":""},"categories":[2],"tags":[38,1788,1351,1195,16,209,30,1666],"class_list":["post-27276","post","type-post","status-publish","format-standard","hentry","category-technology","tag-apple","tag-apple-card","tag-artificial-intelligence","tag-financial","tag-icloud","tag-legal","tag-mac","tag-macos-10-15"],"apple_news_notices":[],"_links":{"self":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/27276","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/comments?post=27276"}],"version-history":[{"count":6,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/27276\/revisions"}],"predecessor-version":[{"id":27932,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/27276\/revisions\/27932"}],"wp:attachment":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/media?parent=27276"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/categories?post=27276"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/tags?post=27276"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}