Matt Cutts writes, not on this own blog for this one, but on the Webmaster Central blog, about how to verify GoogleBot as real. I actually disagree with the method they are attempting to use as solution for one reason: it will put high stress and load-time issues on the server and GoogleBot. I haven’t tested this to be sure, but I am speaking hypothetically. With over five hundred requests from GoogleBot everyday for my pages everyday, that would start to put a stress on my DNS servers and on the server itself for doing all these lookups.
Telling webmasters to use DNS to verify on a case-by-case basis seems like the best way to go. I think the recommended technique would be to do a reverse DNS lookup, verify that the name is in the googlebot.com domain, and then do a corresponding forward DNS->IP lookup using that googlebot.com name; eg:
> host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.> host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1
I think there needs to be a better way to do authentication when it comes to this, it just still needs to be developed.
