{"id":35634,"date":"2022-04-20T14:01:19","date_gmt":"2022-04-20T18:01:19","guid":{"rendered":"https:\/\/mjtsai.com\/blog\/?p=35634"},"modified":"2022-05-04T17:14:33","modified_gmt":"2022-05-04T21:14:33","slug":"duckduckgo-youtube-dl-and-bing","status":"publish","type":"post","link":"https:\/\/mjtsai.com\/blog\/2022\/04\/20\/duckduckgo-youtube-dl-and-bing\/","title":{"rendered":"DuckDuckGo, youtube-dl, and Bing"},"content":{"rendered":"

Ernesto Van der Sar<\/a> (via Hacker News<\/a>):<\/p>\n

\n

Privacy-centered search engine DuckDuckGo has completely removed the search results for many popular pirates sites including The Pirate Bay, 1337x, and Fmovies. Several YouTube ripping services have disappeared, too and even the homepage of the open-source software youtube-mp3 is unfindable.<\/p>\n

[…]<\/p>\n

The most surprising omission, by far, is that the official site for the open-source software youtube-dl is not indexed by DuckDuckGo<\/a>. This site certainly doesn’t host or link to any copyright-infringing material.<\/p>\n<\/blockquote>\n\n

Jesse Squires<\/a>:<\/p>\n

\n

This is quite the predicament. Seemingly, everything is fine. Yet, for some reason my website has been delisted from Bing. I’m perplexed. But to be honest, I’m not that concerned about Bing — I really just want my site indexed by DuckDuckGo again.<\/p>\n

[…]<\/p>\n

All of this emphasizes the frustrating lack of transparency around search engines and how they work. It is infuriating to be essentially helpless trying to debug and resolve issues with your site not being properly indexed. I will forever lament that search was not a core component of the Internet itself, but was instead a feature that private corporations had to bolt on top.<\/p>\n<\/blockquote>\n\n

Gabriel Weinberg<\/a> (Hacker News<\/a>):<\/p>\n

First, there is a completely made up headline going around this weekend. We are not “purging” any media outlets from results. Anyone can verify this by searching for an outlet and see it come up in results.<\/p>

Similarly, we are not “purging” YouTube-dl or The Pirate Bay and they both have actually been continuously available in our results if you search for them by name (which most people do). Our site: operator (which hardly anyone uses) is having issues which we are looking into.<\/p><\/blockquote>\n\n

Jesse Squires<\/a>:<\/p>\n

\n

However, this doesn’t explain the fact that even a search for my name or website without using the site: operator yields no relevant results.<\/p>\n<\/blockquote>\n\n

Ernesto Van der Sar<\/a>:<\/p>\n

\n

Interestingly, both YouTube-dl and Thepiratebay.org still don’t show up on Bing.<\/p>\n

[…]<\/p>\n

A DuckDuckGo spokesperson confirmed to TorrentFreak that the issues are related to Bing data.<\/p>\n<\/blockquote>\n\n

Ernesto Van der Sar<\/a>:<\/p>\n

\n

It’s understandable that DuckDuckGo wasn’t happy with the coverage. However, the problem was real. And since it’s emanating from Bing, other smaller search engines that rely on that data may be affected as well.<\/p>\n

“Since these occurrences originated on Bing, they were passed down to our results, as well as other Bing syndication partners,” Goodman clarifies.<\/p>\n

For DuckDuckGo, it may be tricky to resolve the issue permanently as long as it relies on Bing.<\/p>\n

[…]<\/p>\n

While looking into these issues, we noticed that Bing also affects DuckDuckGo in other ways. From what we can see, the DMCA removals also spill over, including the inaccurate ones.<\/p>\n<\/blockquote>\n\n

Previously:<\/p>\n