Robots.txt is a file to keep robots from indexing
certain portions of the site.

I was looking at trap17's robot.txt, here's a copy:



One question. If you specify '*', then why would one need to
specify the Googlebot?

maybe its just redundant, or maybe some bots don't recognize the asterisk

Oh.. I was restricting google from indexing some files and then there was not any need.
I simply forgot to remove that thing. However, I think I kept it because if I need to add somethign else, then I wont have to search everything back and look for the exact syntax. In short.. I was lazy
But Google does recognise more complex sets of instruction at robots.txt file than other bots. Examples are special wildcards which allows to filter urls .. etc.

Laziness is good. If you're lazy you'll always keep handy things around

I thought I could post a link to a quite nice robots.txt generator. It's quite simple; you enter whether bots are allowed or not, and then you can enter allowed/refused for a number of common bots if you want them to be different from the default. And then you can enter specific restricted directories.

Nothing complex, but nice. And there are examples and links to pre-made robots.txt files.

Alright, just checking... I found almost the same thing at quite a few other
sites like cnn.com, microsoft.com, and howstuffworks.com

Are there any robots that don't recognize the asterisk?

As far as I know Googlebot is the only bot that recognises "*.gif" and such. Other bots don't understand the asterix.