Archive for June 20, 2022

Monday, June 20, 2022

Removed From Bing and DuckDuckGo

Jeff Johnson (Hacker News):

It turns out that my business web site was removed from Bing, which explains why it’s missing from DuckDuckGo.

[…]

According to the link in the results, “Bing limits removal of search results to a narrow set of circumstances and conditions to avoid restricting Bing users’ access to relevant information.” Yet none of these circumstances would seem to apply to my web site, so it’s a mystery.

[…]

A few other people I know, including Jesse Squires, have also seen their web sites mysteriously removed from Bing and DuckDuckGo. Jesse’s site is still missing! Jesse’s blog post links to a blog post by Chase Watts, Affiliate Manager at GoDaddy, who explains an exploit in Bing that allows website owners to deindex competitors, so it’s possible that this is what happened to me.

Jeff Johnson:

A whole thread full of people whose web sites have been inexplicably removed from Microsoft Bing and search engines that rely on Bing, such as DuckDuckGo.

Other people, myself included, are seeing problems where sites are clearly not removed, but neither do they seem to be fully indexed.

Scott Yoshinaga:

@gruber using the DuckDuckGo search in your archive section returns very limited results. I believe it used to be better. I’ve noticed this for my website that uses their search engine as well.

[…]

Also strange: a search term may not have any results return, but if I reload the page multiple times, search results appear eventually.

Previously:

Update (2022-06-24): Jeff Johnson:

Myself and @jesse_squires are both restored to Bing and DuckDuckGo.

No explanation given.

Update (2022-07-26): Jesse Squires:

I originally discovered and wrote about the issue in late March. Jeff published his post a few months later in June. I also received a number of emails and tweets from readers facing the same problem. It really felt like everyone was having this issue. Yet, no one — including DuckDuckGo’s own CEO — seemed to know what the fuck was going on.

[…]

[Bing] confirmed my site committed so-called violations, but refused to tell me what.

[…]

Fast forward almost exactly one month after Bing Support’s response. Ostensibly and miraculously my site is no longer violating Bing’s guidelines.

[…]

However, results on DuckDuckGo still seem incomplete compared to what they used to be, especially compared to Google.

Update (2022-08-02): Relja Novović (via Jeff Johnson):

My websites just got erased from their search results – overnight!

[…]

The first question to ask when something like this happens is: “what did you do/change/update?” The answer is, as far as I remember and can tell – nothing.

[…]

To add insult to injury, when you “Google” the term “BikeGremlin” on Microsoft Bing or DuckDuckGo, SERPs show BikeGremlin social-media accounts and results from a website “bikehow.com” that has literally copied my articles and re-published them!

Update (2022-11-01): Nicolas Magand:

This week, I tried one more time to get in touch with Bing support, I guess this makes it the seventh or eighth time. In a couple of days, I expect to receive a generic email saying that my website doesn’t follow Bing’s guidelines, without saying anything more, even if my website apparently follows every guideline. I have a background in SEO, I write original content, I have a very clean website, I use Bing Webmaster Tools and Google Search Console, and I can’t find anything wrong. I can’t find any reason for this removal. I can’t find any solution to this problem. Clearly, this is a Bing issue, and they are extremely bad at fixing it, or at least point their users to the right direction.

Update (2023-01-18): Dave Rupert (Hacker News):

Why on earth would Bing not index my site at all? To solve this, I took the first step and signed up for Bing Webmaster Tools to try to know what Bing knows about my site and sure enough: zero clicks, zero impressions, and zero indexed pages for my site. Awful.

Jeff Johnson:

Sure enough, https://underpassapp.com is now missing again from both Bing and DuckDuckGo!

My personal site https://lapcatsoftware.com was never removed from Bing or DuckDuckGo. Only my business site is missing.

Update (2023-08-18): Nick Heer (Mastodon):

[This] appears to be a somewhat common issue where Bing — and, consequently, the many small search engines which rely on its results — will completely de-index a website for no apparent reason. Bing still shows zero results for a query for site:techdirt.com, though it does again say “some results have been removed” and links to a generic help page.

Kagi Search and Orion Browser Public Betas

Kagi (Hacker News):

I (Vladimir Prelovac, Founder) started this company when I saw my children starting using the web and realized how unsafe and bad an experience it was for them. All the ads and distractions were not only detrimental to their experience but also were influencing their behavior and habits from a very young age. No alternatives exist or the ones that I tried were not good enough. I have spent the last four years bootstrapping this project, with a small team and building tools to make using the web decent again.

[…]

Today Kagi search and Orion browser enter public beta.

[…]

Kagi search comes as a free version with limited use; and an unlimited use paid option at $10 a month, both completely ad-free, tracking free, and with no search data being retained.

$10/month is not too much for the user to pay if it’s really great, but I don’t see how that can be sustainable for Kagi if it only pays their costs for 2–3 searches per day. Do most users, of the type that would pay for something like this, search far less than I do?

I’m not really sure how to think about the privacy angle. It sounds great, but—as with other search engines—you have no way to know whether they do what they say. The business model doesn’t incentivize them to track you, but how can they improve their algorithms if they truly don’t retain any data? And if they wanted to—or were compelled to—track people, it would arguably be worse than Google because they have your billing information so they know who you are. I’m not saying there’s anything particular to worry about here, just that I would not assume that any Web searches from your own device are fully private.

Anyway, I’m giving Kagi search a try and will report back. So far it definitely seems to have fewer spammy results. Mainly, though, I want to be able to find pages that don’t show up in Bing or Google at all, which is a tall order considering that it doesn’t have its own full index of the Web.

FAQ:

Kagi Search already has many unique features, like personalized results and “Lenses”. And because we depend only on our users for revenue, Kagi can and will always offer a much richer search experience for the user.

[…]

We use heuristics and deep learning to understand query intent, select the best information sources, query them directly using APIs, and rank the results. You can think of Kagi as a “search client,” working like an email client, connecting to indexes and sources to find relevant results and package them into a superior, secure and privacy-respecting search experience for you.

Our searching includes anonymized requests to traditional search indexes like Google and Bing as well as vertical sources like Wikipedia and DeepL or other APIs. We also have our own non-commercial index (Teclis), news index (TinyGem), and an AI for instant answers.

[…]

Kagi features a ‘login token’ which is simply a URL parameter that you can use to automatically log you into your existing Kagi session, from anywhere, including using search within private browser sessions. You can visit Account settings to get your login token.

Vladimir Prelovac:

Kagi doesn’t need a lot of users, because as I said, we are not a Google killer. Google has a huge moat, and a 20 year headstart, and the web is now much harder to crawl than it was 20 years ago, so we will probably never have Google scale, but we don’t need it.

We are trying to build a sustainable business for a small subset of users who think differently. We just need a couple of tens of thousands of such users to be sustainable, and that’s our first milestone.

[…]

The non-commercial indices can have a big impact on the quality of results since it helps us avoid spam, SEO optimized content, and pages riddled with ads. It also allows us to get expert opinions from small websites and personal blogs.

Vladimir Prelovac:

Maybe it will sound strange, but I do not think of our goal as optimizing for growth and number of users. It is to optimize for sustainability and quality of product.

See also: Why Google is so unbearable (and how to fix it) (via Hacker News).

Previously: