Robots.txt is a file to keep robots from indexing certain portions of the site. I was looking at trap17's robot.txt, here's a copy: CODE User-agent: * Disallow: /stylesheets/ User-agent: Googlebot Disallow: /stylesheets/ One question. If you specify '*', then why would one need to specify the Googlebot?

Reply

maybe its just redundant, or maybe some bots don't recognize the asterisk

Reply

Oh.. I was restricting google from indexing some files and then there was not any need.
I simply forgot to remove that thing. However, I think I kept it because if I need to add somethign else, then I wont have to search everything back and look for the exact syntax. In short.. I was lazy
But Google does recognise more complex sets of instruction at robots.txt file than other bots. Examples are special wildcards which allows to filter urls .. etc.

Reply

Laziness is good. If you're lazy you'll always keep handy things around

I thought I could post a link to a quite nice robots.txt generator. It's quite simple; you enter whether bots are allowed or not, and then you can enter allowed/refused for a number of common bots if you want them to be different from the default. And then you can enter specific restricted directories.

Nothing complex, but nice. And there are examples and links to pre-made robots.txt files.

 

 

 

Reply

Alright, just checking... I found almost the same thing at quite a few other
sites like cnn.com, microsoft.com, and howstuffworks.com

Are there any robots that don't recognize the asterisk?

Reply

As far as I know Googlebot is the only bot that recognises "*.gif" and such. Other bots don't understand the asterix.

Reply