Yeah, the first day and my Drupal CMS warning showed up this while I was setting up my website:



So, I executed a tracert on that mysterious IP as NO ONE had an account on my site except ME.



Well, if you aren't experienced, that guy is the Googlebot, which basically is just a bot(duh) that indexes keywords and similar stuff and then puts it into google's database so it shows up when it querys that keyword or similar keyword.

I am sure it will be wroking on by tomorrow, but if you search my name on google you will get a lot of gaming profiles and a lot of spanish/english stuff. ( Stalk, Stalk!)

What hinted me was the IP(I remembered some octets) and that it was looking for robots.txt (A file to restrict certain bots programmed for indexing or whatever else like translation engines to enter your website).

Well, I am happy by now. See ya!

 

 

 

Reply

Congratulations on getting 'botted'.

As you mentioned above, all bots seek out a file named robots.txt which contains the permissions and restrictions that most Bots adhere to, but it isn't 'required' that Bots even stop to read the file. Bot visits can be good for indexing your site on the search engines, but you run the risk of a Bot picking up information which you don't want made public. One trick is to store your 'personal files' in a folder "above" the public_html folder on your site. Apache servers don't allow unauthorised access to these files if the user isn't specificly allowed access.

Reply

How about making a robots.txt tutorial? (You know more) I don't know if there is one already made. I will go check.

Reply

Robots.txt is very easy to understand and you can write it with notepad.

Visit http://www.robotstxt.org/wc/faq.html for more information. Basically it tells which bot agent to view or not to view (a.k.a. indexing) certain directories and files.

This is useful when you hold files/filenames that may change or deleted in the future. You don't want bots to report back as "broken link." This will cause search engines to flag your site/content as "unreliable."

Reply