Archive for November 18, 2019

Monday, November 18, 2019

How Google Interferes With Its Search Algorithms and Changes Your Results

Kirsten Grind et al. (via Sam Schechner, Hacker News):

Over time, Google has increasingly re-engineered and interfered with search results to a far greater degree than the company and its executives have acknowledged, a Wall Street Journal investigation has found.


Despite publicly denying doing so, Google keeps blacklists to remove certain sites or prevent others from surfacing in certain types of results. These moves are separate from those that block sites as required by U.S. or foreign law, such as those featuring child abuse or with copyright infringement, and from changes designed to demote spam sites, which attempt to game the system to appear higher in results.


To evaluate its search results, Google employs thousands of low-paid contractors whose purpose the company says is to assess the quality of the algorithms’ rankings. Even so, contractors said Google gave feedback to these workers to convey what it considered to be the correct ranking of results, and they revised their assessments accordingly, according to contractors interviewed by the Journal. The contractors’ collective evaluations are then used to adjust algorithms.


In one change hotly contested within Google, engineers opted to tilt results to favor prominent businesses over smaller ones, based on the argument that customers were more likely to get what they wanted at larger outlets. One effect of the change was a boost to Amazon’s products, even if the items had been discontinued, according to people familiar with the matter.

Barry Schwartz:

The truth is, I spoke to a number of these Wall Street Journal reporters back in both March and April about this topic, and it was clear then that they had little knowledge about how search worked. Even a basic understanding of the difference between organic listings (the free search results) and the paid listings (the ads in the search results) eluded them. They seemed to have one goal: to come up with a sensational story about how Google is abusing its power and responsibility for self gain.

Google is not certainly perfect, but almost everything in the Wall Street Journal report is incorrect.


I know they interviewed me a couple of times, and I told you how that went above. But we reached out to Glenn Gabe, an SEO industry veteran who works extensively with companies that have been impacted by search algorithm updates, who was quoted in the piece. Gabe told us that not only were his conversations with the paper off-the-record but also that he was misquoted. Gabe said he reached out to the reporter who apologized and offered to fix the quote. But later he was told that the quote had to stay as is.

It’s worrisome that something so important is a black box. I’ve been seeing smoke about stuff like this for years, but solid information is hard to come by, and no one has yet put it together in a way that shows there really is a fire. It seems that this report doesn’t, either.



Devran Uenal (via Hacker News):

OpenSwiftUI is an OpenSource implementation of Apple’s SwiftUI DSL (Domain-specific language.

The project’s goal is to stay close to the original API as possible.

The actual rendering of UI elements can then be implemented by other projects for different platforms like Linux, Windows, Embedded, etc.


This project is in early development.

Even if you don’t plan to use the open-source implementation, seeing how it works can help you understand what SwiftUI is doing.


Interesting detail about Square that I learnt on Twitter shortly after SwiftUI was launched… Square sold iPads with their software on as “appliances”, so they can’t deprecate them on a schedule like most software. For this reason the Square iOS codebase (which is a single codebase) must support back as far as iOS 9.

Because of this, they’re considering an internal implementation of SwiftUI so that they can start using it now, rather than in ~5 years time. I wouldn’t be surprised if they open sourced this if they do go ahead with it.

See also: AwfulUI.


Basecamp Personal

Jason Fried (via tweet):

But one complaint we’ve heard is that Basecamp is priced for businesses, not for personal side projects. We felt it was finally time to do something about that.

So today we’re formally introducing Basecamp Personal – a completely free Basecamp plan designed specifically for freelancers, students, families, and personal projects.

It’s limited to 3 projects and 20 users. It’s nice that they allow it for money-making projects, rather than only personal ones as some services do. The business price is $99/month, which is just too much for the occasional freelance project.

David Heinemeier Hansson:

We’ve had a free plan in the past, but it was always way more limited than our new Basecamp Personal. But not everyone can afford our business plans, and it’s a tragedy if they’re forced to trade their privacy with Big Tech tools.


You can delete a project, but no archiving.

Vaping Apps Removed From App Store

Joe Rossignol:

Apple is removing all vaping-related apps from the App Store today, according to Axios, shortly after the Centers for Disease Control and Prevention reported 2,172 lung injury cases linked to e-cigarette or vape products.


Apple had already took a step in this direction in June, when it updated its App Store Review Guidelines to indicate that apps encouraging consumption of vape products are not permitted on the App Store.


The benefits and the health risks of e-cigarettes are uncertain. There is tentative evidence they may help people quit smoking, although they have not been proven to be more effective than smoking cessation medicine. There is concern with the possibility that non-smokers and children may start nicotine use with e-cigarettes at a rate higher than anticipated than if they were never created.

John Gruber:

The stuff about selling cartridges, and sharing news — it’s fine for that stuff to be out of the App Store because you can get it on the web. But Bluetooth stuff where apps were used as the interface for controlling hardware — web apps can’t do that (nor should they be able to). There is no alternative to a native app, and native apps are only available on the App Store. This would be an easy call to make (and would have been made from the get-go by Apple) if vaping were illegal. But it’s not illegal.

Update (2019-11-26): Steve Troughton-Smith:

There are all kinds of modern hardware ecosystems that literally do nothing if not paired with a smartphone app, like some scooter brands. Apple does not get to unilaterally make calls like this, and I hope they get crucified in upcoming antitrust cases

The Macalope:

It’s worth pointing out that the canisters that did contain cyanide were counterfeit. The Macalope just checked his local liquor store and we haven’t banned alcohol sales because prison wine blinded some people. He also checked the App Store and we haven’t banned mixology apps, either. But one of the apps Apple banned actually checked canisters to see if they were counterfeit.

Ben Thompson:

Those apps — and by extension, device functionality — are no longer available to iPhone users— you can’t get this level of functionality in a browser — not because regulators ruled them illegal, or because Congress passed a law, but because a group of technology executives said so. And, what they said held sway because the App Store is integrated with the iPhone: Apple has a monopoly on what apps can or cannot be installed.

John Gruber:

As it stands today, tobacco and marijuana are OK in the App Store if you smoke them but banned if you vape them. That distinction seems impossible to defend, other than by noting that vaping is a hot topic in the news, and cigarettes and weed baroning are not.


Update (2020-10-20): Journal of the American Heart Association:

During peer review, the reviewers identified the important question of whether the myocardial infarctions occurred before or after the respondents initiated e‐cigarette use, and requested that the authors use additional data in the PATH codebook (age of first MI and age of first e‐cigarettes use) to address this concern. While the authors did provide some additional analysis, the reviewers and editors did not confirm that the authors had both understood and complied with the request prior to acceptance of the article for publication.

Post publication, the editors requested Dr. Bhatta et al conduct the analysis based on when specific respondents started using e‐cigarettes, which required ongoing access to the restricted use dataset from the PATH Wave 1 survey. The authors agreed to comply with the editors’ request. The deadline set by the editors for completion of the revised analysis was not met because the authors are currently unable to access the PATH database. Given these issues, the editors are concerned that the study conclusion is unreliable.

The editors hereby retract the article from publication in Journal of the American Heart Association.

The Hotel Cupertino Clause

Alex Russell (Google employee):

There seems to be confusion about how, exactly, Apple keeps the web second-class on iOS. Understandable! It’s the interplay of several interlocking effects. Let’s examine them (thread).

First, no matter how app-like it is, Section 4.2 of the App Store Review Guidelines excludes web experiences from being discovered via the search box where users go to add things to their homescreen.


Next, Apple under-invests in Safari’s engine (WebKit) in ways that cumulatively make it difficult to do anything new and ambitious.


On every other OS, the way we have dealt with laggard browsers is through competition.


Section 2.5.6 is the Hotel Cupertino clause: you can pick any browser you like, but you can’t choose a better web. In fact, iOS prevents other browsers from even replacing Safari as the system default.

2.5.6 caps web progress at the rate that Apple (under) invests.

All of this has been done to preserve the linkage between proprietary OS/APIs, an exclusive software ecosystem, and the hardware sales that software ecosystem supports.

If you’re a web developer, this means that iOS -- the whole OS -- is the new IE6. Your CEO and wealthiest users won’t switch off it, so it taxes everything you do. They also can’t imagine the web being great because, for them, it isn’t.

Russ Bishop (Apple employee):

That’s rich, coming from the company that abused its #1 search position to drive adoption of Chrome, forked Blink from WebKit, and slams new APIs through without considering perf/privacy.

“Open Web” from Google is code for “whatever Is best for Google and ad-tech this week”

You were the ones that decided to fork from WebKit, dividing what had been the largest community of devs working on web technologies.

You did it because YOU want control of the web.

As if best battery life and privacy is “under-investing”.

Tony Arnold:

Google forking WebKit was a thin cover for breaking down standards bodies of the time.

It wasn’t altruistic. It wasn’t about users.

Google are an advertising company that uses their tech to gather data to better sell ads.

Miguel de Icaza (Microsoft employee):

1. This has the upside of preventing the web to have a single provider in the form of Chromium. This is the last bastion of defense for diverse engines.

2. It also slows down the standardization of fringe features that are done purely for Google’s needs and have no reason to be standardized.

Orta Therox (Microsoft Employee):

I’m pretty glad for the App Store rules that keep chrome from monopolizing all web browsers in every context, personally. Never been a Chrome user, but I think Safari do a good balance on user features vs dev wants.

Matt Aimonetti:

Both companies are terrible and both of you have valid points. Apple opts to be conservative and closed (therefore safer but also in a closed ecosystem), Google is pushing features, often for its own benefit, which leads to fragmentation and security threads

Yes, both companies are pursuing policies that benefit them and which they also think are better for users. Users have multiple concerns, which are only partially addressed by each browser.

On the Mac, I choose Safari because I like its emphasis on privacy, and I think it’s a better Mac app. But seemingly everyone I know who is not picky about such things uses Chrome because it’s always compatible. (Or, perhaps, it has such marketshare that sites are forced to adapt to it, but to the user that amounts to the same thing.) I am seeing increasingly many sites that don’t work properly with Safari, so for them I use Chrome or Firefox. The WebKit Goals for 2020 (via Hacker News) don’t have much to say about this. It also annoys me that Safari beachballs for a few seconds every time it auto-fills a login. So I am always tempted to switch, but I don’t like any of the other choices that much.

On iOS, of course, there is no choice. Microsoft famously got in trouble merely for privileging its browser, not actively blocking competing browsers from running.

Kyle Pflug:

There’s a third axis here, which is “good for competition.” It’s difficult to deny (at least, in good faith) that iOS policies have the effect of making it more difficult for compelling alternate device ecosystems to take root (Google Play policies do the same thing.)

The web has remarkable potential to be an equalizer and Apple is strongly disincentivized to invest in capabilities that enable that.

Even when there are also good-faith reasons to prefer a walled garden (and there are many), it’s hard to not notice the more cynical motivations.

Jamie Gaskins:

I really wish people would stop using the phrase “the new IE6”. There is literally nothing in web development today that compares to developing for IE6.

Nathanael Anderson:

Actually I tend to agree it is the new ie6. Been around since pre-ie6. IE6 froze API in that era; so all these cool new API were unusable because of IE6. Safari is doing the same thing now. Lots of new shipping api’s are unusable because of having to support iOS...

Bridger Maxwell:

FWIW, when I port features from my iOS app to the web app version it is often Safari that is missing the APIs. The little examples add up. For example, touch and scroll input APIs don’t support Retina resolutions yet (they are all whole numbers). Other browsers do.

There are lots of people making fun of Google for counting the number of Web APIs implemented. But what other metric would make sense? It’s pretty funny because in the early days Apple bragged about the percentage of Web specs Safari implemented vs. other browsers, and at each WWDC it touts the number of new iOS and macOS APIs that it’s added.


Update (2019-11-20): Apple responds to Congress (via Hacker News):

Users cannot uninstall Safari, which is an essential part of iPhone functionality; however, users have many alternative third-party browsers they can download from the App Store.


iPhone users cannot set another browser as the default browser. Safari is one of the apps that Apple believes defines the core user experience on iOS, with industry-leading security and privacy features.


The purpose of this rule is to protect user privacy and security. Nefarious websites have analyzed other web browser engines and found flaws that have not been disclosed, and exploit those flaws when a user goes to a particular website to silently violate user privacy or security. This presents an acute danger to users, considering the vast amount of private and sensitive data that is typically accessed on a mobile device. By requiring apps to use WebKit, Apple can rapidly and accurately address exploits across our entire user base and most effectively secure their privacy and security. Also, allowing other web browser engines could put users at risk if developers abandon their apps or fail to address a security flaw quickly. By requiring use of WebKit, Apple can provide security updates to all our users quickly and accurately, no matter which browser they decide to download from the App Store.


Google pioneered the out-of-process architecture that Safari now uses, developed the Safe Browsing program that Safari also uses, drove the adoption of HTTPS, put pressure on misbehaving certificate authorities and shepherded certificate pinning and then Certificate Transparency, and found many vulnerabilities in WebKit through security research that Apple was not itself doing.

Moreover, on the desktop, Chrome and Firefox both have automatic update channels that allow them to push out security fixes much more rapidly than Apple’s heavy OS updates. (On iOS, they would be limited by Apple’s App Store approval process.)