Should robots.txt support a feature for no indexation? Take the survey

Jason September 27, 2021 0 Comments

I saw a discussion on Twitter this morning about the idea of having a feature in Robots.txt that would block both crawling AND indexing. It started with this tweet by Christian Thurston (@ct_oz):

“Hi John [Mueller], has Google considered making it so that the robots.txt file doesn’t just block crawling, but also blocks indexation? To quote @willcritchlow: “I can’t see many situations where I want to block crawling but don’t want to block indexing”.

“That would be a significant change in expectations (and yes, we do think about these things regardless). Do you have some examples where this would cause a visible improvement in search?” Mueller responded. “I’d like to avoid adding more directives. I’m still not aware of common issues caused by this documented functionality … SEOs worry about indexing, but usually these URLs only rank for site:-queries (or if there isn’t other, better content on the site), so it feels artificial?”

With over 20 years of experience in SEO, “I have never encountered a situation where a publisher wanted to have a page indexed that they block for crawling in robots.txt. Not even once have I seen that,” I tweeted in response. “It’s common practice for me to educate people that they have a choice: (1) block crawling, or: (2) prevent indexation, when what they want to do is both. Note: definitely more of an issue for larger sites where crawl budget is an issue.”

Will Critchlow of Search Pilot agreed, “100% agree. I can see the conservatism of not changing a long-standing standard, but I have never seen, and can’t think of a situation where you’d want to block crawling but allow indexation.”

But what do YOU think? Would it be helpful to have a feature in Robots.txt that allowed you to specify that pages you don’t want to have indexed? Take our quick three-question poll below and let us know what you think.

Please keep in mind that if you vote yes for doing this that you would need to accept ALL the risk that at some point your dev team might misapply the capability and NoIndex pages that you don’t want them to.

Loading…

The post Should robots.txt support a feature for no indexation? Take the survey appeared first on Search Engine Land.

AboutJason Richardson

Jason specializes in digital marketing strategies and founded a digital marketing agency Richardson Digital Media™. Richardson Digital Media™ has quickly become a preferred marketing provider for real estate companies and nationwide mortgage lenders. Jason has won 9 President’s Circle of Excellence Awards for loan production over his career and was most recently honored as one of National Mortgage Professional Magazine’s “40 Under 40: The 40 Most Influential Mortgage Professionals.”