Monday, June 20, 2022

Removed From Bing and DuckDuckGo

Jeff Johnson (Hacker News):

It turns out that my business web site was removed from Bing, which explains why it’s missing from DuckDuckGo.

[…]

According to the link in the results, “Bing limits removal of search results to a narrow set of circumstances and conditions to avoid restricting Bing users’ access to relevant information.” Yet none of these circumstances would seem to apply to my web site, so it’s a mystery.

[…]

A few other people I know, including Jesse Squires, have also seen their web sites mysteriously removed from Bing and DuckDuckGo. Jesse’s site is still missing! Jesse’s blog post links to a blog post by Chase Watts, Affiliate Manager at GoDaddy, who explains an exploit in Bing that allows website owners to deindex competitors, so it’s possible that this is what happened to me.

Jeff Johnson:

A whole thread full of people whose web sites have been inexplicably removed from Microsoft Bing and search engines that rely on Bing, such as DuckDuckGo.

Other people, myself included, are seeing problems where sites are clearly not removed, but neither do they seem to be fully indexed.

Scott Yoshinaga:

@gruber using the DuckDuckGo search in your archive section returns very limited results. I believe it used to be better. I’ve noticed this for my website that uses their search engine as well.

[…]

Also strange: a search term may not have any results return, but if I reload the page multiple times, search results appear eventually.

Previously:

Update (2022-06-24): Jeff Johnson:

Myself and @jesse_squires are both restored to Bing and DuckDuckGo.

No explanation given.

Update (2022-07-26): Jesse Squires:

I originally discovered and wrote about the issue in late March. Jeff published his post a few months later in June. I also received a number of emails and tweets from readers facing the same problem. It really felt like everyone was having this issue. Yet, no one — including DuckDuckGo’s own CEO — seemed to know what the fuck was going on.

[…]

[Bing] confirmed my site committed so-called violations, but refused to tell me what.

[…]

Fast forward almost exactly one month after Bing Support’s response. Ostensibly and miraculously my site is no longer violating Bing’s guidelines.

[…]

However, results on DuckDuckGo still seem incomplete compared to what they used to be, especially compared to Google.

Update (2022-08-02): Relja Novović (via Jeff Johnson):

My websites just got erased from their search results – overnight!

[…]

The first question to ask when something like this happens is: “what did you do/change/update?” The answer is, as far as I remember and can tell – nothing.

[…]

To add insult to injury, when you “Google” the term “BikeGremlin” on Microsoft Bing or DuckDuckGo, SERPs show BikeGremlin social-media accounts and results from a website “bikehow.com” that has literally copied my articles and re-published them!

Update (2022-11-01): Nicolas Magand:

This week, I tried one more time to get in touch with Bing support, I guess this makes it the seventh or eighth time. In a couple of days, I expect to receive a generic email saying that my website doesn’t follow Bing’s guidelines, without saying anything more, even if my website apparently follows every guideline. I have a background in SEO, I write original content, I have a very clean website, I use Bing Webmaster Tools and Google Search Console, and I can’t find anything wrong. I can’t find any reason for this removal. I can’t find any solution to this problem. Clearly, this is a Bing issue, and they are extremely bad at fixing it, or at least point their users to the right direction.

Update (2023-01-18): Dave Rupert (Hacker News):

Why on earth would Bing not index my site at all? To solve this, I took the first step and signed up for Bing Webmaster Tools to try to know what Bing knows about my site and sure enough: zero clicks, zero impressions, and zero indexed pages for my site. Awful.

Jeff Johnson:

Sure enough, https://underpassapp.com is now missing again from both Bing and DuckDuckGo!

My personal site https://lapcatsoftware.com was never removed from Bing or DuckDuckGo. Only my business site is missing.

Update (2023-08-18): Nick Heer (Mastodon):

[This] appears to be a somewhat common issue where Bing — and, consequently, the many small search engines which rely on its results — will completely de-index a website for no apparent reason. Bing still shows zero results for a query for site:techdirt.com, though it does again say “some results have been removed” and links to a generic help page.

16 Comments RSS · Twitter · Mastodon

Glad this issue is finally getting some traction. I've personally given up on trying to solve this problem for my website, as my multiple messages to Bing support over the past year or so either went unanswered or consisted of naive unhelpful standard messages. Google doesn't need this kind of free advertising, especially not from their only alleged competitor.

uh oh, now that they're no longer doing it to conservatives or other groups of people that the tech bubble considers "undesirables", it's a problem.

woah. maybe allowing private corporations to instantly censor whoever they wish with no accountability, wasn't such a good idea.

@anon Deciding, as an organisation, that you do not want to publish something is not censorship.
There are plenty of places like Fox News for the ‘undesirables’ to promulgate their conspiracy nonsense.

Old Unix Geek

A hammer is supposed to help one drive in nails, not have an opinion about what kind of nails it likes. In the same way, a search tool is supposed to search the web, not to have an opinion about what it "wants to publish".

Somehow all this bollocks about publishers "having the right to express an opinion" has corrupted what a search tool is. If they want to do that they should clearly state: our website does not search the web, but provides a curated view of what we want you to think. Many newspapers at least have the honesty to state their bias.

Google certainly doesn't say "we have an opinion, a bias" in their blurb about themselves: https://www.google.com/search/howsearchworks/our-approach/ Instead, they claim they care about providing the "most helpful information", which should be the information people are looking for, not the information Google wants them to see. I presume the other search engines say the same.

This has serious implications. If people believe search tools are objective, but they aren't, one is distorting the understanding of the entire population without them knowing. Democracy cannot exist with such shadowy interference.

Furthermore, the fact that people have been trained to think these search engines are objective means actually objective search engines cannot compete.

For instance, WebTop, a British search tool, was out-competed, and is no longer available. https://web.archive.org/web/20000708023501/http://www.webtop.com/index.html (Xapian is derived from that product.) It's dead, and Google ended up victorious.

The real world has a much wider range of valid opinions than American mainstream media or Fox. Some of us want to see that full diversity of views, not just those that our local propagandists want us to hear. This is unfortunately a lot harder than it used to be... knowing multiple European languages is no longer enough, when it used to be.

Oh so its not just me! My site (volitans-software.com) doesn't show up on bing either, but clearly searching for it has related results. Its maddening!

> Google certainly doesn't say "we have an opinion, a bias"

It's that famous liberal bias that reality just can't get rid of, so if Google tries to rank correct results over misinformation, they automatically get accused of having "an opinion." I guess "we want to rank correct results over incorrect results" is now a political opinion.

> neither do they seem to be fully indexed

But this has always been the case, no? Bing's index is just much smaller than Google's, so if you have a website that isn't super popular and do a site search, you're going to get five times as many results on Google than on Bing.

@ Old Unix Geek

I get where you are coming from and in principle would agree were it not for the way the web has become monetised and the algorithms that underlie most of it.

As you know, engagement is king.

It turns out people are more engaged by controversy than level-headedness.
There is also a fine line between publishing and promoting.

If you publish outlandish and provably false opinions it will often turn out they drive the most engagement and become the most popular - so re algorithms those views get presented as having primacy by virtue of engagement - rather than merit or even sanity.
It's a fine line to tread and missing your step can end up facilitating the promotion of wrong-headedness, worldwide.

@Plume If Daring Fireball is not popular enough to be fully indexed, that’s not a good sign for Bing.

Old Unix Geek

@Plume:

> 'reality has a liberal bias'

I don't source my wisdom from mediocre comedians. One might think that reality has a liberal bias in a time where the pendulum is swinging in a liberal direction. One won't think that when it swings back.

Reality itself is neither liberal nor conservative, nor anything else anyone might wish to project on it. It's just reality.

@Niall:

I don't believe the solution is to change the world to coddle people from what is. The solution is to change the people so they can recognize stupidity themselves. Simply because someone is on the mainstream media, or leading a country, doesn't mean they aren't spewing complete and utter rubbish.

We don't cover the world in leather to prevent our feet from being harmed. Instead we cover our feet in leather. In the same way, we should learn to think critically (few people do), rather than expecting our information sources to be "safe for consumption without critical thought". Such beliefs are very helpful to those trying to create a propagandized population.

Proper education is key, and that requires having proper educators. It seems to me that the West is failing on that score, which presages a fall. Ray Dalio has a book about civilizational health. Bad education is a sign of illness.

"A hammer is supposed to help one drive in nails, not have an opinion about what kind of nails it likes."

Another construction metaphor? Sigh. Unfortunately for this argument, different hammers are in fact designed for hitting different things. That is why after thousands of years we haven't all converged on One Best Hammer. Just this week I've used two different kinds of hammers for two different kinds of nails.

"Some of us want to see that full diversity of views ... knowing multiple European languages is no longer enough, when it used to be."

Oh we got both kinds of viewpoints here: Romantic *and* Germanic!

Old Unix Geek

@Ted: :s/what kind of nails it likes/what brand of nails it likes/g

@ Old Unix Geek,

Yeah you're probably right - it's better that there is no interference when information is provided to the public. Perhaps with context provided with the information. As you say, education is key.

It's just problematic as malign influencers will always game the system.

It turns out that being the conduit for 'all' the information is a bit of a poisoned chalice.

"that’s not a good sign for Bing"

Yes. I find Bing's results to be pretty consistently less useful than Google's. My only point is that much of this is probably a matter of them being a worse search engine (and having always been worse), rather than some kind of intentional or recent attempt to remove stuff from their index.

"Reality itself is neither liberal nor conservative"

Obviously. As you pointed out, it was a joke. I'm just saying that "search engines need to rank falsehoods higher than the truth, or else they are biased" is a hell of a take.

"Instead we cover our feet in leather"

Exactly. But your point is that people should wear shoes without soles, and just get thicker skin, because shoe manufacturers shouldn't be protecting people's feet.

Hi Michael,

I saw this after reading Jesse's article - I'm facing the same problem. In addition to writing a report of the situation, I've used the opportunity to check out the most popular search engines, see which one basically "copy" Bing, and which ones seem to be independent.

Incident report (still unresolved) and the search engine lists are here:
https://io.bikegremlin.com/28530/microsoft-bing-serp-gone-overnight/

I began looking into this in July. We use Duck Duck Go (Bing) for our internal search, but site: searches went from having thousands of pages to between 10 and 55. In other words, they were absolutely useless for our readers. The pages that Bing did have were really, really old (we’re talking early 2000s) for the most part, including some we had not linked since 2005. We had to switch to Mojeek, which had about 3,600 entries for us.

I’ll cut to the chase: Bing has tanked. The index appears to have plunged to fewer than 1,000 million pages. That’s smaller than Inktomi or Alltheweb back in the 2000s. I don’t know why, and I doubt they do.

Here was what I discovered:

https://jackyan.com/blog/2022/07/bing-has-tanked/

Try feeding in different site: searches into Google, Mojeek and Bing and you’ll see just how tiny Bing’s index is.

https://jackyan.com/blog/2022/08/mojeek-shows-more-in-its-search-results-than-google/

For this site, site:mjtsai.com returns the following:

Google: 9,220 claimed
Mojeek: 19,774
Bing: 78

Bing will even repeat results so you can’t believe what it says when it has found x results for your site.

https://jackyan.com/blog/2022/08/more-of-bings-follies-they-just-keep-coming/

[…] on links collected by Michael Tsai, this appears to be a somewhat common issue where Bing — and, consequently, the many small search […]

Leave a Comment