DuckDuckGo, youtube-dl, and Bing
Ernesto Van der Sar (via Hacker News):
Privacy-centered search engine DuckDuckGo has completely removed the search results for many popular pirates sites including The Pirate Bay, 1337x, and Fmovies. Several YouTube ripping services have disappeared, too and even the homepage of the open-source software youtube-mp3 is unfindable.
[…]
The most surprising omission, by far, is that the official site for the open-source software youtube-dl is not indexed by DuckDuckGo. This site certainly doesn’t host or link to any copyright-infringing material.
This is quite the predicament. Seemingly, everything is fine. Yet, for some reason my website has been delisted from Bing. I’m perplexed. But to be honest, I’m not that concerned about Bing — I really just want my site indexed by DuckDuckGo again.
[…]
All of this emphasizes the frustrating lack of transparency around search engines and how they work. It is infuriating to be essentially helpless trying to debug and resolve issues with your site not being properly indexed. I will forever lament that search was not a core component of the Internet itself, but was instead a feature that private corporations had to bolt on top.
Gabriel Weinberg (Hacker News):
First, there is a completely made up headline going around this weekend. We are not “purging” any media outlets from results. Anyone can verify this by searching for an outlet and see it come up in results.
Similarly, we are not “purging” YouTube-dl or The Pirate Bay and they both have actually been continuously available in our results if you search for them by name (which most people do). Our site: operator (which hardly anyone uses) is having issues which we are looking into.
However, this doesn’t explain the fact that even a search for my name or website without using the site: operator yields no relevant results.
Interestingly, both YouTube-dl and Thepiratebay.org still don’t show up on Bing.
[…]
A DuckDuckGo spokesperson confirmed to TorrentFreak that the issues are related to Bing data.
It’s understandable that DuckDuckGo wasn’t happy with the coverage. However, the problem was real. And since it’s emanating from Bing, other smaller search engines that rely on that data may be affected as well.
“Since these occurrences originated on Bing, they were passed down to our results, as well as other Bing syndication partners,” Goodman clarifies.
For DuckDuckGo, it may be tricky to resolve the issue permanently as long as it relies on Bing.
[…]
While looking into these issues, we noticed that Bing also affects DuckDuckGo in other ways. From what we can see, the DMCA removals also spill over, including the inaccurate ones.
Previously:
6 Comments RSS · Twitter
My website has be removed from Bing results a few months ago. No clue why. No clear indicators on Bing Webmaster Tools. No answer from the support after my repeated message saying "My site does not appear on the index." Looking for either a solution or an explanation, I found many other users with the same problem, users that are tech-savvy enough to use Bing Webmaster tools, and how SEO works. I think the Bing index is in a much more terrible shape than was believed.
Another reason I switched to Brave Search – it's not based on Bing like DDG. (I've suggested it here before, but no, I'm not astroturfing. Just a happy user. And someone who avoids the Brave browser because of the related cryptocurrency stuff.)
@Alexander Brave say they don’t use other indexes, by which I’m not convinced since their results look exactly like Bing’s, but they say they are using third party APIs — like Bing — for image search. My website disappeared from Bing earlier this year, and disappeared from Brave too at the same time. I wish that StartPage, which uses Google, was available by default in more place.
In general, it seems that Bing just indexes a lot fewer pages than Google. If I use the site: syntax to search company websites for support forum posts, DDG often shows about 10% of the results that Google does. When DDG shows me five matching results, Google shows five pages of results, all of which clearly match the search criteria, and should appear on DDG.
@Nicolas Brave does fallback to results from Bing and, in very rare instances, Google. You can click on the gear icon from the search results page to see their current “Search results independence”, which measures how much of their results come from their own index.
More info on that metric here:
I'm actually really impressed with Brave search. I only use DuckDuckGo occasionally now. Google gets a few hits and even directly to Bing occasionally, but by far I used Brave search day in and day out. DuckDuckGo and their Bing parent have been killing me with missing search results for a while now.
