Monday, June 20, 2022 [Tweets] [Favorites]

Removed From Bing and DuckDuckGo

Jeff Johnson (Hacker News):

It turns out that my business web site was removed from Bing, which explains why it’s missing from DuckDuckGo.

[…]

According to the link in the results, “Bing limits removal of search results to a narrow set of circumstances and conditions to avoid restricting Bing users’ access to relevant information.” Yet none of these circumstances would seem to apply to my web site, so it’s a mystery.

[…]

A few other people I know, including Jesse Squires, have also seen their web sites mysteriously removed from Bing and DuckDuckGo. Jesse’s site is still missing! Jesse’s blog post links to a blog post by Chase Watts, Affiliate Manager at GoDaddy, who explains an exploit in Bing that allows website owners to deindex competitors, so it’s possible that this is what happened to me.

Jeff Johnson:

A whole thread full of people whose web sites have been inexplicably removed from Microsoft Bing and search engines that rely on Bing, such as DuckDuckGo.

Other people, myself included, are seeing problems where sites are clearly not removed, but neither do they seem to be fully indexed.

Scott Yoshinaga:

@gruber using the DuckDuckGo search in your archive section returns very limited results. I believe it used to be better. I’ve noticed this for my website that uses their search engine as well.

[…]

Also strange: a search term may not have any results return, but if I reload the page multiple times, search results appear eventually.

Previously:

Update (2022-06-24): Jeff Johnson:

Myself and @jesse_squires are both restored to Bing and DuckDuckGo.

No explanation given.

13 Comments

Glad this issue is finally getting some traction. I've personally given up on trying to solve this problem for my website, as my multiple messages to Bing support over the past year or so either went unanswered or consisted of naive unhelpful standard messages. Google doesn't need this kind of free advertising, especially not from their only alleged competitor.

uh oh, now that they're no longer doing it to conservatives or other groups of people that the tech bubble considers "undesirables", it's a problem.

woah. maybe allowing private corporations to instantly censor whoever they wish with no accountability, wasn't such a good idea.

@anon Deciding, as an organisation, that you do not want to publish something is not censorship.
There are plenty of places like Fox News for the ‘undesirables’ to promulgate their conspiracy nonsense.

Old Unix Geek

A hammer is supposed to help one drive in nails, not have an opinion about what kind of nails it likes. In the same way, a search tool is supposed to search the web, not to have an opinion about what it "wants to publish".

Somehow all this bollocks about publishers "having the right to express an opinion" has corrupted what a search tool is. If they want to do that they should clearly state: our website does not search the web, but provides a curated view of what we want you to think. Many newspapers at least have the honesty to state their bias.

Google certainly doesn't say "we have an opinion, a bias" in their blurb about themselves: https://www.google.com/search/howsearchworks/our-approach/ Instead, they claim they care about providing the "most helpful information", which should be the information people are looking for, not the information Google wants them to see. I presume the other search engines say the same.

This has serious implications. If people believe search tools are objective, but they aren't, one is distorting the understanding of the entire population without them knowing. Democracy cannot exist with such shadowy interference.

Furthermore, the fact that people have been trained to think these search engines are objective means actually objective search engines cannot compete.

For instance, WebTop, a British search tool, was out-competed, and is no longer available. https://web.archive.org/web/20000708023501/http://www.webtop.com/index.html (Xapian is derived from that product.) It's dead, and Google ended up victorious.

The real world has a much wider range of valid opinions than American mainstream media or Fox. Some of us want to see that full diversity of views, not just those that our local propagandists want us to hear. This is unfortunately a lot harder than it used to be... knowing multiple European languages is no longer enough, when it used to be.

Oh so its not just me! My site (volitans-software.com) doesn't show up on bing either, but clearly searching for it has related results. Its maddening!

> Google certainly doesn't say "we have an opinion, a bias"

It's that famous liberal bias that reality just can't get rid of, so if Google tries to rank correct results over misinformation, they automatically get accused of having "an opinion." I guess "we want to rank correct results over incorrect results" is now a political opinion.

> neither do they seem to be fully indexed

But this has always been the case, no? Bing's index is just much smaller than Google's, so if you have a website that isn't super popular and do a site search, you're going to get five times as many results on Google than on Bing.

@ Old Unix Geek

I get where you are coming from and in principle would agree were it not for the way the web has become monetised and the algorithms that underlie most of it.

As you know, engagement is king.

It turns out people are more engaged by controversy than level-headedness.
There is also a fine line between publishing and promoting.

If you publish outlandish and provably false opinions it will often turn out they drive the most engagement and become the most popular - so re algorithms those views get presented as having primacy by virtue of engagement - rather than merit or even sanity.
It's a fine line to tread and missing your step can end up facilitating the promotion of wrong-headedness, worldwide.

@Plume If Daring Fireball is not popular enough to be fully indexed, that’s not a good sign for Bing.

Old Unix Geek

@Plume:

> 'reality has a liberal bias'

I don't source my wisdom from mediocre comedians. One might think that reality has a liberal bias in a time where the pendulum is swinging in a liberal direction. One won't think that when it swings back.

Reality itself is neither liberal nor conservative, nor anything else anyone might wish to project on it. It's just reality.

@Niall:

I don't believe the solution is to change the world to coddle people from what is. The solution is to change the people so they can recognize stupidity themselves. Simply because someone is on the mainstream media, or leading a country, doesn't mean they aren't spewing complete and utter rubbish.

We don't cover the world in leather to prevent our feet from being harmed. Instead we cover our feet in leather. In the same way, we should learn to think critically (few people do), rather than expecting our information sources to be "safe for consumption without critical thought". Such beliefs are very helpful to those trying to create a propagandized population.

Proper education is key, and that requires having proper educators. It seems to me that the West is failing on that score, which presages a fall. Ray Dalio has a book about civilizational health. Bad education is a sign of illness.

"A hammer is supposed to help one drive in nails, not have an opinion about what kind of nails it likes."

Another construction metaphor? Sigh. Unfortunately for this argument, different hammers are in fact designed for hitting different things. That is why after thousands of years we haven't all converged on One Best Hammer. Just this week I've used two different kinds of hammers for two different kinds of nails.

"Some of us want to see that full diversity of views ... knowing multiple European languages is no longer enough, when it used to be."

Oh we got both kinds of viewpoints here: Romantic *and* Germanic!

Old Unix Geek

@Ted: :s/what kind of nails it likes/what brand of nails it likes/g

@ Old Unix Geek,

Yeah you're probably right - it's better that there is no interference when information is provided to the public. Perhaps with context provided with the information. As you say, education is key.

It's just problematic as malign influencers will always game the system.

It turns out that being the conduit for 'all' the information is a bit of a poisoned chalice.

"that’s not a good sign for Bing"

Yes. I find Bing's results to be pretty consistently less useful than Google's. My only point is that much of this is probably a matter of them being a worse search engine (and having always been worse), rather than some kind of intentional or recent attempt to remove stuff from their index.

"Reality itself is neither liberal nor conservative"

Obviously. As you pointed out, it was a joke. I'm just saying that "search engines need to rank falsehoods higher than the truth, or else they are biased" is a hell of a take.

"Instead we cover our feet in leather"

Exactly. But your point is that people should wear shoes without soles, and just get thicker skin, because shoe manufacturers shouldn't be protecting people's feet.

Stay up-to-date by subscribing to the Comments RSS Feed for this post.

Leave a Comment