Webmaster Tools – Robots.txt User-agent: * Syntax not understood

By | May 5, 2014

While going through Google’s Webmaster Tools early today for one of my sites, I noticed under Crawl > Blocked URLs, that the number of URLs that Google had blocked was 0, even though the site has a robots.txt with a few rules, and Google has crawled and indexed 10s of thousands of pages so far.

Upon further inspection, I noticed a peculiar thing:

Line 1: User-agent: *     Syntax not understood

robots-analysis-before

Interesting? How can User-agent not be recognized?

Turns out that one of the things that could be causing this, is if the robots.txt is saved using the UTF-8 encoding, then a BOM (Byte order mark) can be inserted at the beginning of the line, which Google will read, and not like.

Here’s what I had in my robots.txt before:
robots-before

To solve this issue, there were two things that I had to do.

First, was to open the robots.txt file in Notepad, and then do a Save As and selecting ANSI as the encoding

notepad-saveas-ansi

Second thing I had to do, was add a comment on the first line, and then press ENTER so that User-agent went to the second line.
robots-after

After re-uploading it to the site, and running the test again, the message went away.

Hopefully this will help out anyone else who has came across this same problem.

2 thoughts on “Webmaster Tools – Robots.txt User-agent: * Syntax not understood

  1. Jason

    Thanks a lot. i have nevver thought that webmaster tools have Problems with such tvivial Things.

Leave a Reply

Your email address will not be published. Required fields are marked *