.Google's John Mueller answered a concern about why Google.com marks webpages that are forbidden from creeping through robots.txt as well as why the it's safe to disregard the relevant Search Console documents concerning those crawls.Robot Website Traffic To Question Criterion URLs.The individual talking to the concern documented that robots were actually developing links to non-existent question parameter Links (? q= xyz) to webpages with noindex meta tags that are actually also blocked in robots.txt. What cued the concern is actually that Google is creeping the links to those webpages, getting shut out through robots.txt (without watching a noindex robots meta tag) then obtaining reported in Google.com Search Console as "Indexed, though shut out through robots.txt.".The person inquired the observing inquiry:." Yet here's the significant question: why would Google.com index web pages when they can't even find the web content? What's the advantage because?".Google's John Mueller validated that if they can not creep the webpage they can not find the noindex meta tag. He also creates an interesting mention of the site: hunt driver, suggesting to dismiss the results due to the fact that the "common" consumers won't view those results.He created:." Yes, you are actually correct: if our team can not crawl the web page, our team can't find the noindex. That mentioned, if our team can't crawl the webpages, at that point there is actually not a whole lot for our company to mark. Thus while you may see several of those pages with a targeted website:- question, the common consumer will not observe them, so I definitely would not bother it. Noindex is also alright (without robots.txt disallow), it simply suggests the URLs will definitely end up being actually crawled (as well as end up in the Search Console file for crawled/not listed-- neither of these conditions cause issues to the remainder of the internet site). The essential part is actually that you do not produce all of them crawlable + indexable.".Takeaways:.1. Mueller's solution verifies the limits being used the Internet site: hunt accelerated search driver for analysis reasons. Some of those reasons is actually since it is actually not connected to the routine hunt mark, it is actually a distinct factor altogether.Google's John Mueller commented on the web site hunt driver in 2021:." The brief response is actually that a site: question is actually not meant to be complete, nor made use of for diagnostics purposes.A website question is a certain sort of search that restricts the outcomes to a certain website. It is actually basically merely the word web site, a colon, and afterwards the internet site's domain name.This inquiry restricts the results to a particular internet site. It is actually certainly not implied to be a thorough selection of all the webpages from that web site.".2. Noindex tag without utilizing a robots.txt is alright for these type of circumstances where a bot is connecting to non-existent pages that are getting uncovered through Googlebot.3. Links along with the noindex tag will certainly create a "crawled/not listed" item in Browse Console and that those won't have a bad effect on the remainder of the internet site.Go through the question as well as answer on LinkedIn:.Why will Google.com index webpages when they can not also view the content?Included Image by Shutterstock/Krakenimages. com.