Question About Robots.txt File

Mich

Mar 13 2006, 02:02 AM

I ran a search about this file and found a link that took me to a site to generate a robots.txt file. It also told me where to place it in my file manager. However, I just don't understand exactly what this file does. I do get that it prevents some search engines from poking around in all my files. Is this all it does? What are the consequences of not preventing this from happening? My folders are most all graphic sets that I offer for free viewing. I do not use my cgi-bin nor my mail. I am only using my public_html folder.

I do see in my errors log that someone or something has been looking for a robots.txt file.

keysmaker

Mar 13 2006, 02:15 AM

The file name utilized by the robot exclusion protocol. Web robots download this file from the server’s document root and parse it for instructions on what to index and not to index. The case of the file name does not matter, but it must exist in the document root.

for example:

CODE

User-agent: Googlebot
Disallow : /
User-agent: *
Disallow: /forum/

Where user-agent is the name of the robot or spider, in this case this instruction disallow to Googlebot (google robot) to index all the site.

In the next instruction, the "wildcard" belongs to ALL the robots or spiders, disallowing the indexing only to "/forum/" directory.
Is a good idea to set up the robots.txt file, avoiding that search engines crawdle into your gallery, anyway, if you want to share the content, simply put a flat .txt file with this inside:

CODE

User-agent: *
Disallow :

Mich

Mar 13 2006, 02:41 AM

QUOTE(keysmaker @ Mar 12 2006, 09:15 PM)

OK, so define "Parse". What is considered the "document root"? My public_html folder? Or the one up from there that appears in my ftp window when I connect to upload?

So if I put this in my robots.txt
CODE
User-agent: Mediapartners-Google*
Disallow: /cgi-bin/
Disallow: /_*/
User-agent: *
Disallow: /

Google cannot get into my cgi-bin or any folder that starts with an underline and any other robot cannot get in any folder or file?

jlhaslip

Mar 13 2006, 02:51 AM

Mich,
The most likely reason you are getting the robots.txt file in your error logs is if you do not have one. It would be an error of type "404" if that were the case. To avoid that error, place at least an empty robots.txt file in the directory "the one up from there that appears in my ftp window when I connect to upload".

As to the correctness of the robots.txt file you list, I will let others answer because I am not up-to-date on them.

parse= read and interpret the instructions contained in the file

http://www.gnc-web-creations.com/creating_robotstxt_file.htm

kylelnsn

May 10 2006, 10:30 PM

what

Recent Queries:-

robots.txt raw matter - 135.76 hr back. (1)

Similar Topics

Keywords : robots txt file

A Trap17 How-to Guide For Beginners - Something for those who can't get enough of the Trap17 Readme file (12)
I am not entirely sure if this the right place to post about this, but here goes anyway... For
those who find themselves slightly (or considerably more than slightly) clueless about where to get
started after getting a hosting account, here is an user guide that might potentially help you
out. I wrote it out of three hours, so forgive me if it's clumsy, but I'm hoping that a few
people here may find it useful. Feedback is greatly appreciated /smile.gif"
style="vertical-align:middle" emoid=":)" border="0" alt="smile.gif" /> I'm planning to build on
a FAQ...

Phpbb File Missing - (4)

Something's Is Wrong With My Install File - (5)

I uploaded all of my files and I am sure that they all went through correctly. But when I went to
the install file it didnt work seem to work so I checked it out and the file got uploaded but there
is no text in the text file and I tried to delete it and then reuploaded but the same thing
happened. I even tried editing the file and then put the text in manually but afterwards it always
says page cannont be displayed then I reload the page and it says it saved but it doesn't. I
dont know if I am doing something wrong or not...

File On Host - (3)

Can i put file *.rar ??? And what is a limit it ? I need 20mb limit moved as per report ...

A Question About Zip File! - Upload to unzip it on my host???? (3)

QUOTE Don't get carried away by other companies who claim to give UNLIMITED SPACE AND
BW! Just upload a couple of MP3's and they will Delete your accounts right away! Even
though We generally do NOT allow hosting of ZIP and MP3 files, You still can. But if you do, Your
account will not be upgraded beyond 1GB Bandwidth. So we still allow you to SPAM your Account. Now
no other competitor would offer that! I wonder if I can upload my zipped website in a
single file and unzip it after uploaded. Is this allowed? I mean that 'can my account b...

Ftp File Size Limit - (2)

hey I am wondering what the FTP file size limit is. Does anybody know?...

Word Wrap - File editor (4)

Is there anyway of disabling the word wrap that happens when editing files? It has been getting
annoying and confuses me sometimes. And i had to modify a script i made cause word wrap somehow made
it create parse errors....

Cpanel Log File - Question... Problem ? - log file not logging traffic- ADMIN help requested (1)

Hi all, Have my Cpanel "Log File Manager" settings set to "backup" the logfile daily(ie. I've
checked the first checkbox). QUESTION: This should give me a "cumulative" backup listing of all the
traffic to my webpages in a sequential date (ie. day by day) format..Right?? If this is correct..
then there is a problem.. PROBLEM: I'm only getting partial (many days of traffic are missing)
from the raw log files. The monthly file only contains a couple of days of logs. Can anyone clarify
(or help)?? RGPHNX...

File Type - (2)

Does this hosting service have file type restriction? /huh.gif' border='0'
style='vertical-align:middle' alt='huh.gif' /> Are there any restrictions, such as a file type
whose upload is impossible? ...

How To Upload File,so I Can Link It For Download? - (2)

To which folder,etc....

File Uploading Problem - (0)

I checked the php info file, and I found out that there is no temp directory for the files to be
uploaded to (I don't mean the uploader on the server, I mean when you ask a user to upload
something from your website). My code is also correct, so the files should be uploading to my
server. Is anyone else experiencing this problem?...

Cron Help. It Says File Not Found. - (4)

Help. I just got the cron jobs working and now its saying file not found. Here is a image of what I
enter. Image I'm getting an email saying: QUOTE /bin/sh: line 1:
http://www.xero.trap17.com/xore/cron/rewards.php: No such file or directory Can anyone help me
on this? I really need these cron jobs. The times I have it set on are: 0 Minutes, 12:00 Hours,
Every Day, Every Month....

Looking for robots, txt, file

ADD REPLY / Got an Opinion!	a humble request :-)	RAPID SEARCH!	Free Hosting	[X]
Express your Opinions, Thoughts or Contribute your information that might help someone here. Ask your Doubts & Queries to get answers.. "Together, We enlight each other!"	Register FREE for AD-FREE forum, Create your own topics, Ask Questions, track topics, setup subscriptions & notifications and Get a Free Website w/ Email and FTP.		500MB Space No Ads, CPanel, FTP, PHP, MySQL, EMails - 100% FREE

Question About Robots.txt File - What exactly does it do?

Question About Robots.txt File - What exactly does it do?