Comments on: Can Googlebot See You? Let’s Find Out http://www.ginside.com/2007/1077/can-googlebot-see-you-lets-find-out/?utm_source=rss&utm_medium=rss&utm_campaign=can-googlebot-see-you-lets-find-out Sat, 10 Dec 2011 14:16:52 +0000 hourly 1 http://wordpress.org/?v=3.2.1 By: Jonathan Dingman http://www.ginside.com/2007/1077/can-googlebot-see-you-lets-find-out/#comment-440 Jonathan Dingman Tue, 28 Aug 2007 00:17:11 +0000 http://www.ginside.com/2007/1077/can-googlebot-see-you-lets-find-out/#comment-440 John, I actually went through and verified it, but if there isn't any listing, such as Disallow: <whatever>, it will take whatever it has from above. Such as I have listed, Yahoo!'s slurp will obey both the listings and it actually has done exactly that. Personally, I don't want Googlebot to be the only robot allowed to crawl my site. MSN and Yahoo! still send me a decent bit of traffic which I'm appreciative of. It probably is the case that only the major search engines obey the wildcards, but those are all that I'm concerned with in this case. I'm just as happy not caring about the rest since most people that even search for my site, use one of those three major search engines. But in any case, yes, you would want to use the fully qualified paths if you were wanting to exclude *every* robot that obeys the rules. Thanks for your two cents John. John,

I actually went through and verified it, but if there isn’t any listing, such as Disallow: , it will take whatever it has from above. Such as I have listed, Yahoo!’s slurp will obey both the listings and it actually has done exactly that.

Personally, I don’t want Googlebot to be the only robot allowed to crawl my site. MSN and Yahoo! still send me a decent bit of traffic which I’m appreciative of.

It probably is the case that only the major search engines obey the wildcards, but those are all that I’m concerned with in this case. I’m just as happy not caring about the rest since most people that even search for my site, use one of those three major search engines.

But in any case, yes, you would want to use the fully qualified paths if you were wanting to exclude *every* robot that obeys the rules.

Thanks for your two cents John.

]]> By: JohnMu http://www.ginside.com/2007/1077/can-googlebot-see-you-lets-find-out/#comment-439 JohnMu Mon, 20 Aug 2007 14:56:47 +0000 http://www.ginside.com/2007/1077/can-googlebot-see-you-lets-find-out/#comment-439 I'm fairly certain you don't want your robots.txt like that. By having generic and specified sections, you are telling the Yahoo crawler to apply the crawl-delay but letting it crawl everything, including the feeds. The specific section overrides any setting you have in your generic section. For instance: <blockquote>user-agent: * disallow: / user-agent: Googlebot disallow:</blockquote> would allow the Googlebot to crawl everything while disallowing access to all other crawlers. Similarly, you need to keep in mind that only Google, Yahoo and MSN ("only" probably 99.0%, but anyway :-)) actually use wildcards. This means that all other crawlers who see those disallow-lines will not be able to parse them and will allow those URLs. If you need to keep them out, you would have to put the full URLs into your robots.txt. I’m fairly certain you don’t want your robots.txt like that. By having generic and specified sections, you are telling the Yahoo crawler to apply the crawl-delay but letting it crawl everything, including the feeds. The specific section overrides any setting you have in your generic section.

For instance:

user-agent: *
disallow: /

user-agent: Googlebot
disallow:

would allow the Googlebot to crawl everything while disallowing access to all other crawlers.

Similarly, you need to keep in mind that only Google, Yahoo and MSN (“only” probably 99.0%, but anyway :-) ) actually use wildcards. This means that all other crawlers who see those disallow-lines will not be able to parse them and will allow those URLs. If you need to keep them out, you would have to put the full URLs into your robots.txt.

]]>