Frédéric Dubut, a senior program supervisor at Microsoft engaged on Bing Search, mentioned on Twitter Wednesday that once you create a particular part in your robots.txt file for its Bingbot crawler, it’s best to make sure that to record all of the default directives in that part.
Specify directives for Bingbot. “For those who create a piece for Bingbot particularly, all of the default directives can be ignored (besides Crawl-Delay),” he mentioned. “You MUST copy-paste the directives you need Bingbot to comply with below its personal part,” he added.
Helpful robots.txt reminder – should you create a piece for #Bingbot particularly, all of the default directives can be ignored (besides Crawl-Delay). You MUST copy-paste the directives you need Bingbot to comply with below its personal part. #web optimization #TechnicalSEO
— Frédéric Dubut (@CoperniX) January 2, 2019
What does it imply? This in all probability signifies that Bing has seen various websites complain that Bingbot is crawling areas of their web sites that they don’t wish to be crawled. It’s seemingly some site owners assumed that in the event that they gave Bingbot some particular directions, it might comply with the remainder of the default directives not listed. As a substitute, you probably have a piece for Bingbot, it’s going to solely comply with the directives you’ve particularly listed in that part of your robots.txt file. For those who should not have a particular part for Bingbot, then Bingbot will comply with the default directives.
Why it issues. Guarantee that once you arrange your robots.txt file, that every one the search engine crawlers can effectively crawl your website. For those who arrange particular directives for blocking, crawl delays or different directives, then make it possible for all of the search engine crawlers are listening to these directives. They might not hear if there are syntax points, if you don’t comply with their protocols or they’ve points accessing such directives.
For extra on establishing a robots.txt for Bing, see the assistance paperwork.
About The Writer
Barry Schwartz is Search Engine Land’s Information Editor and owns RustyBrick, a NY primarily based internet consulting agency. He additionally runs Search Engine Roundtable, a preferred search weblog on SEM matters.