While going through Google’s Webmaster Tools early today for one of my sites, I noticed under Crawl > Blocked URLs, that the number of URLs that Google had blocked was 0, even though the site has a robots.txt with a few rules, and Google has crawled and indexed 10s of thousands of pages so far.

Upon further inspection, I noticed a peculiar thing:

Line 1: User-agent: *     Syntax not understood

Interesting? How can User-agent not be recognized?
Continue reading →