Archive for November 15, 2019

Friday, November 15, 2019

Apple Card’s Outsourced Algorithm

David Heinemeier Hansson (video):

The @AppleCard is such a fucking sexist program. My wife and I filed joint tax returns, live in a community-property state, and have been married for a long time. Yet Apple’s black box algorithm thinks I deserve 20x the credit limit she does. No appeals work.

It seems like a bit of an overreaction because he’s extrapolating from two data points that are not independent. There could be a reasonable explanation, or simply a bug. But it’s worth asking whether the algorithm is fair in aggregate, and many other customers have chimed in with similar experiences. This story also points to other issues about black box algorithms and AI: whether anyone actually understands the algorithm, the desire for explanations and appeal in individual cases, special treatment for some, and Apple’s role in the card that bears its name.

Jamie Heinemeier Hansson:

I care about transparency and fairness. It’s why I was deeply annoyed to be told by AppleCard representatives, “It’s just the algorithm,” and “It’s just your credit score.” I have had credit in the US far longer than David. I have never had a single late payment. I do not have any debts. David and I share all financial accounts, and my very good credit score is higher than David’s. […] But AppleCard representatives did not want to hear any of this. I was given no explanation. No way to make my case.

There wasn’t even a way to check and correct the inputs to the algorithm.

I care about justice for all. It’s why, when the AppleCard manager told me she was aware of David’s tweets and that my credit limit would be raised to meet his, without any real explanation, I felt the weight and guilt of my ridiculous privilege.

If only it were so easy for a manager to snap their fingers and make other computing problems go away. But with cloud syncing and other modern bugs, there’s often no explanation or solution. People accept brokenness because they feel they have no choice. There are no legal requirements for transparency, nor culture of it.

David Heinemeier Hansson:

And the best defense of all: THAT’S JUST HOW THE CREDIT INDUSTRY WORKS. Ehh, okay? How is that anything but the most damning charge upon Apple’s pitch with their card? Did the iPhone launch pledging to please carriers and the status quo as its modus operandi? No.

Apple offers a credit card that bases its credit assessment on a black-box algorithm that 6 different reps across Apple and GS have no visibility into. Even several layers of management. An internal investigation. IT’S JUST THE ALGORITHM!

So nobody understands THE ALGORITHM. Nobody has the power to examine or check THE ALGORITHM. Yet everyone we’ve talked to from both Apple and GS are SO SURE that THE ALGORITHM isn’t biased and discriminating in any way.

[…]

The algorithm might discriminate out of biased historical training data, faulty but uncorrectable inputs, programming errors, or malicious intent. You’ll never be able to know.

Sridhar Natarajan and Shahien Nasiripour (tweet, Hacker News):

A Wall Street regulator is opening a probe into Goldman Sachs Group Inc.’s credit card practices after a viral tweet from a tech entrepreneur alleged gender discrimination in the new Apple Card’s algorithms when determining credit limits.

Sonder Scheme:

We suspect that the Goldman algorithm was trained on data that included an important bias: that the husband is the primary card holder in traditional credit card approval. This biased the data so the algorithm assigned higher creditworthiness to the primary card holder. This meant that the primary card holder status became the proxy for gender.

The whole situation was made worse by a number of applications coming from a demographic or group that exposed this bias, an AI-enabled product which broke the mental model of Apple family sharing and a total lack of a “human-in-the-loop” recovery combined with unexplainable and non-intuitive AI.

David Heinemeier Hansson:

They delegated the most central question of the entire product – CAN I USE THIS? – to a status-quo player with a checkered moral past.

Steve Wozniak:

The same thing happened to us. I got 10x the credit limit. We have no separate bank or credit card accounts or any separate assets. Hard to get to a human for a correction though. It’s big tech in 2019.

Daniel Vassallo:

My @Apple Card got rejected. The reason appears to be that @keybank recently closed a checking account that got automatically opened when I got my mortgage and I never ever used!

Not a great first impression from every party involved in this.

David Heinemeier Hansson:

Apple likes to promote that their card was “Created by Apple, not a bank”. So please, stop with the “THIS IS GOLDMAN’S FAULT” shit. The card is called THE APPLE CARD. Sending out GS spokespeople to deflect on their responsibility and ownership is cowardly.

[…]

Imagine if you had to call Foxconn yourself when your iPhone broke? If it says Apple on the box, it’s an Apple product.

David Heinemeier Hansson:

“Apple Card completely rethinks everything about the credit card. It represents all the things Apple stands for. Like simplicity, transparency, and privacy.” - I couldn’t make this up! This is literally the top pitch on the @AppleCard marketing page.

Update (2019-11-27): Josh Centers:

Apple Card not offering joint accounts is a real liability that’s blowing up in Apple’s and Goldman Sachs’ collective faces. Goldman Sachs has said that the low-limit problem is due to applicants with a “limited credit history,” like people who piggyback on their spouse’s credit lines. It’s likely this wouldn’t have made the news at all if the Apple Card had simply offered joint accounts from the beginning, like nearly every other credit card.

Update (2020-01-24): Apple Card:

A new kind of credit card. Created by Apple and built for iPhone.

David Heinemeier Hansson:

Pretty amazing that Apple still runs this campaign after Goldman Sachs has repeated stated that, no, this is not a card created by Apple. This is a GS card, with a discriminatory black-box algorithm, and they make all the decisions. All Apple provides is their brand and an app.

Enpass Moves to a Subscription Model

Enpass:

We are transforming our business model into a seamless subscription model that would make things easier for our users. A subscription would make the on-boarding process easier allowing users to purchase Enpass once anywhere and use the same feature set across all their devices.

A good software costs money and a subscription helps sustain the creator/developer while offering constant value to the customer. Enpass, as a product focused on online security, requires extensive work with security experts as well as regular third-party audits to safeguard your data and digital security. The subscription will allow us to keep improving the experience with the latest platform innovations along with delivering new features and functionalities.

[…]

The transition to subscription will not change how Enpass will work. The core architecture will remain the same – Enpass would continue to be an offline password manager and we would never store your data on our servers in any way.

Paulo Andrade (developer of Secrets):

The competition keeps switching to subscriptions 😏. I’ll be here holding the fort.

Previously:

1Password Takes Accel Investment

Dave Teare (tweet, TechCrunch, Hacker News):

Accel will be investing USD$200 million for a minority stake in 1Password. Along with the investment – their largest initial investment in their 35-year history – Accel brings the experience and expertise we need to grow further and faster.

[…]

Since then, 1Password has become more successful than we ever dreamed. It’s been humbling to watch as we’ve crossed one milestone after another. I still remember with fondness hiring our first employee, planning our first AGConf, recruiting my friend as CEO, opening our first office, and acquiring our first enterprise customer with over 300,000 employees.

[…]

We’ve been turning down Venture Capital firms for as long as they’ve been courting us. We were profitable and didn’t see the value in partnering with someone else. It was fun to grow the company ourselves from 2 to 176, but just like when we hired our CEO, we’ve reached a point where we need expertise and guidance from those who’ve made this journey before.

There’s been a lot of worrying that this is going to hurt the consumer product. This seems like the wrong time to be asking that question because they’ve already been growing rapidly and pivoting to the enterprise. As a consumer, I wish they hadn’t done that. But given that decision—and there are certainly benefits to having thoughtfully designed password software in big businesses—the funding and partnership are tools to help them execute that plan.

1Password has remained a good product, though the parts I care about seem to be on life support. New features are being added to the hosted service, which I don’t want to use. The app’s basic design and focus on browser extensions are not a good fit for the way I want to use it. But these seem to be differences in philosophy, not consequences of the growth strategy.

Accel:

1Password is trusted by millions of users globally, 50,000+ paying business customers, and 25% of the Fortune 100, including leading enterprises such as IBM, Slack, Pagerduty, Dropbox, GitLab and Roche. We’re excited to be investing alongside the Slack Fund, Atlassian co-founders Mike Cannon-Brookes & Scott Farquhar, Atlassian president Jay Simons, and several notable angels.

[…]

At the time of that meeting, they were scaling their consumer-focused product and in the process of releasing a business solution. Now, their Enterprise Password Manager (EPM) is used by tens of thousands of businesses (including 40+ in Accel’s portfolio). They’ve added numerous features broadening the scope of their product beyond password management, including SCIM Bridge and Advanced Protection.

Matthew Panzarino:

1Password was a nice business for a long time but it’s biggest competitor lastpass started pushing hard into enterprise, flooding out the “byot” channel. Makes sense they’d raise to build support and grow enterprise business. I don’t see the need to freak out for personal users.

Ryan Jones:

Ahhh this is important context missing from TechCrunch: Dashlane raised $110M. File this under “you have to plan the game on the field”.

So $200M for an enterprise sales team and enterprise features to avoid getting squashed by capital.

[…]

The @1Password :: Dropbox metaphors could not be more perfect.

Cabel Sasser:

I can’t help but feel there’s a wonderful and often unexplored middle ground between “die” and “grow and grow aggressively”, but that’s extremely easy for me to say without knowing any details, and nobody likes a backseat CEO!!!! I wish you and the team the best! ♥️

Dave Teare:

Fun fact, over the last 14 years we’ve grown log2(174) =~ 7.4, so we’ve been roughly doubling every two years. That to me is already pretty freaking fast but it hasn’t been fast enough to keep up. Hope that gives some context to “aggressively”.

Roustem Karimov:

Feature requests, bug fixes, BugCrowd bounties, SOC2 certification, GDPR requests, pentests, AWS and Google Cloud infrastructure upgrades/monitoring, ...

Customer support -- when your customers have to wait weeks to get a reply because there are 10,000 emails in the queue.

Cabel Sasser:

Yipes!! One (very simplistic) question then: if you have 10,000 e-mails in your queue, you have an incredible number of customers, which means you should have plenty of revenue to cover hiring the people needed to reduce that queue. Right? But maybe not, if you were near death?

Roustem Karimov:

Money was not the deciding factor, we do not need it. We need the people who did this before. Turns out growing is hard and painful -- I had no idea how many things I didn’t know. Sure, we could spend a few more years and figure it out on our own but this felt like an easier way.

Jamie Phelps:

“Doesn’t seem like a business that needs 174 employees to operate.” Let’s unpack that a little[…]

Daniel Jakut:

I’m as skeptical of VC as most people, but I think critics of @1Password’s funding round should be less sure of their predictions of doom. Some companies suffer after funding, and some companies don’t. Change is scary, but not necessarily harmful.

I agree in principle, but I would like to see a list of top-quality consumer apps developed by software companies that took lots of VC money. I don’t see any in my Dock.

Previously:

Update (2019-11-18): John Gruber:

The 1Password founders seem confident that they can expand rapidly into the enterprise world without losing the soul that has made their indie consumer app so beloved (and trusted). Most companies that have tried this, however, have failed. (Dropbox is the one that pops to mind first.)

Dave Teare (tweet):

We kept full control so we could uphold our values and ensure we’d never be forced to compromise on them. Our values are what made us successful over the last 14 years and we plan on building on them for the next 14 years and beyond. Our values really struck a chord in today’s world and I’m tickled pink that they resonated with so many people. 🙌

[…]

For the last 14 years we’ve gone from 2 to 174 people, so we’ve already been doubling every other year. For most businesses this is already “growing aggressively” but we’ve done this for 14 years already and it’s worked quite well. The thing is, even though we’re already growing fast, we need to accelerate the pace just to catch up to today’s tasks, let alone the multitude of other things that we’d like to add to our plate.

[…]

One of the things that gets me most excited is 99% of the time employees at businesses are using the exact same features in 1Password that everyone else is using. The difference in the enterprise is all the tooling that goes around these core features to provide businesses the tools they need. As we improve the user experience for individuals we automatically improve our business offering. It means that we get to continue focusing on what’s made us successful since the beginning. We can have our (gluten-free) cake and eat it, too. 🧁

See also Teare’s 2013 presentation at Çingleton.

Daniel Pasco:

I was also at this talk, which hugely tempered my reaction to the investment news.

Update (2019-11-20): Dave Teare:

Thankfully we’re gonna be fine. There are a lot of incorrect assumptions that those fears are based on and I’d love to clear them up.

Update (2019-11-25): Om Malik:

If you ask me, money won’t ruin 1Password. There are precedents for this sort of thing: Atlassian was a private, self-grown business that thrived for years before it took venture capital and then went public. The capital only helped expand its footprint. It continues to thrive.

Do people love Atlassian products the way they love 1Password?

David Heinemeier Hansson:

We once again proved that if Twitter is great for starting a beef, podcasting is so much better at digesting it. My conversation with @dteare about the $200m in VC for 1pw is going to be out on December 3rd.

Update (2019-11-27): 1Password did an Ask Me Anything.

Update (2020-01-10): Ilja A. Iwas:

Why isn’t @sourcetree 4 notarized?

Maybe don’t aspire to have it work out like Atlassian.