While going through Google’s Webmaster Tools early today for one of my sites, I noticed under Crawl > Blocked URLs, that the number of URLs that Google had blocked was 0, even though the site has a robots.txt with a few rules, and Google has crawled and indexed 10s of thousands of pages so far.
Upon further inspection, I noticed a peculiar thing:
Line 1: User-agent: * Syntax not understood
Interesting? How can User-agent not be recognized?
Continue reading