Now there are many reasons why this could happen which I will not get into here at the moment. However on March 21st 2006 Shawn wrote in a little detail that Google is not interpretting his robots.txt file correctly. “Google Not Interpreting robots.txt Consistently”
Basically the problem was when you had a generic User Agent section and a Google User Agent section. If you had a Google User Agent section, then Google would ignore the User Agent section instead of merging the two. Please read Shawn’s awesome post to get more info.
Now I’m not one to normally summarize somebody elses blog postings, but I really enjoy Shawn’s random blog. Also consider this a Public Service Announcement to keep an eye on your Robots.txt file.