Over the past few weeks, I’ve seen reports that Microsoft Bing is having its own indexing issues. This means that Bing does not index pages like it previously did for its search engine. In fact, I’m told that Microsoft has confirmed the issue with CloudFlare engineers and is on Microsoft’s side to resolve.
Some thought it had to do with the CloudFlare blocking, but it could be more prevalent than that based on the reports I receive both publicly and privately via email.
I asked Fabrice Canel from Microsoft about this and he responded on Twitter with a set of best practices for Bing to index. 🙂
Best practices in the event that a site is not listed on Bing. Investigate it first through the URL inspection tool https://t.co/InvjmzlbBB to identify possible crawling issues, then read the Bing webmaster guidelines https://t.co / Gbrm4B2XMX Pay attention to make sure your site is playing by the rules
– Fabrice Canel (@facan) October 26, 2020
But let me explain to you what I saw. A CloudFlare community thread received tons of complaints from the other day. Here is the first complaint:
Hello, several people had the same issue as me because their websites hosted by Cloudflare had blocked Microsoft IP addresses from which WebCrawler tried to access web pages.
I just checked my website email and got a notification from Microsoft Bing that their webcrawler could not access my Cloudflare website.
The folks at WebmasterWorld have noticed it too, here are some quotes:
And now for some really cool info: I blocked this IP address in Ubuntu UFW firewall on one of my websites and after about a week it got deindexed from BING. It looks like BING is a lot stricter and if they can’t crawl your website, they are taking some drastic action! It might just be a coincidence of course. So, it’s really not clear if this is a “bad” IP address, as user reviews of “IP abuse” aren’t really proof other than guesswork on their part.
There is an ongoing Bing deindexing update / bug. Many users have reported it on other forums.
I checked on Bing to find that my site had been completely deindexed about a month ago. This is what I get for never bothering to check Bing.
In September, one of my websites was banned from the Bing SERP. On October 23, I noticed my other three websites were banned. Even if I type “site: example.com”, there is no result. Just the link “Some results have been deleted”
CloudFlare confirmed the issue, saying “Microsoft has acknowledged that this is a legitimate crawler and is a problem on their end. They are working to fix it (by configuring to conform to their expected published behavior). “
So it looks like Microsoft Bing is working on a fix.
I’m not sure if this is more than just a CloudFlare and Microsoft Bing issue or not, but there are a lot of complaints about it.
The good news is that it looks like Microsoft Bing is working on a fix.
Forum discussion at the CloudFlare community.
Update: More from Fabrice Canel:
For Cloudflare, some sites reported that a small percentage of Bing crawlers were not resolving DNS properly and that we were crawling outside of the usual IP addresses for a few days. We fixed this yesterday.
– Fabrice Canel (@facan) October 27, 2020