Welcome Guest ( Log In | Register)



 
Reply to this topicStart new topic
> How Searchengines Makes Their Bots?
itssami
post Mar 29 2006, 03:44 PM
Post #1


Super Member
*********

Group: Members
Posts: 258
Joined: 13-November 05
Member No.: 14,234



Where as i know, when we submit a website to a search engine, they usually send a BOT to our website , which index all the pages and then adds it to the website.
i have a stupid question that how do they make the spider bot ohmy.gif i was just thinking about it smile.gif Is there any other way to add website without a bot ?? Are there some search engines which dont use bots ? and they use some other way ??
Go to the top of the page
 
+Quote Post
jlhaslip
post Mar 29 2006, 04:44 PM
Post #2


A computer once beat me at chess, but it was no match for me at kick boxing.
Group Icon

Group: [MODERATOR]
Posts: 3,994
Joined: 24-July 05
From: In Trouble Again... still?
Member No.: 9,787
Spam Patrol



To find the ways to make a 'bot', use Google. (Google is your friend) I have seen instructions on php programming sites, BUT, they can be quite resource heavy, so I wouldn't use them on a Shared Server such as the Trap17 Hosting. They are also prone to having lots of errors, so they can't be depended upon. This is something best left to professional coders. An amatuer Bot will be trouble waiting to happen.

As for other methods to getting the available information off the increasing number of Web Locations and indexing them into a Searchable Database, I would think that if you discover a new method, Google would love to hear about it. Good luck in your search.
Go to the top of the page
 
+Quote Post
Lyon2
post Mar 30 2006, 07:53 PM
Post #3


The Ethical Hacker
***********

Group: [HOSTED]
Posts: 1,154
Joined: 27-May 05
From: Portugal (Europe)
Member No.: 7,566



itssami, that's not a stupid question at all, very pertinent i may say so.

Actually, i'm allways reading seo and search engines ebooks, at least the best ones, and i can't find a resource, ebook or something that can help me learn how to build a spider/boot to crawl the web for pages or links to pages and so on.

I would really like help on this issue like itssami, so if someone has good information about this, i would apreciate it very much.
Go to the top of the page
 
+Quote Post
anwiii
post Mar 31 2006, 05:10 AM
Post #4


Privileged Member
*********

Group: [HOSTED]
Posts: 572
Joined: 5-March 06
From: Warsaw, MO
Member No.: 19,564



QUOTE(itssami @ Mar 29 2006, 09:44 AM) *

Where as i know, when we submit a website to a search engine, they usually send a BOT to our website , which index all the pages and then adds it to the website.
i have a stupid question that how do they make the spider bot ohmy.gif i was just thinking about it smile.gif Is there any other way to add website without a bot ?? Are there some search engines which dont use bots ? and they use some other way ??


search engines normally use bots so you're out of luck there, but directories like yahoo, or dmoz, etc....don't they have live people looking over your website to make sure you found the right directory and your site is up and running with no under construction page.

now although normally all search engines and directories take time to list because when you submit your site, you are last in the que for the bot to spider which takes up to 3 months now i hear since there are more than 3 billion webpages out there now(not sure on exact figure). 10 years ago, it was alot easier and faster to get listed.

now you have two options if you don't want to wait. one is to pay for submission like yahoo(although yahoo isn't a search engine) OR you want to use some strategy in allowing the search engines to find you easily. google is great about listing sites that never was submitted since the bots like to float around alot and find websites they never saw before.

how do you accoplish this second alternative? find someone who is already linked to google. and ask them to link to you. whether a big or small site, doesn't matter as long as that websites links are bot friendly. i recomend everyone to consider google site maps(even though i never used it before). i believe it's created as an .xml file extention? not sure of that so don't quote me. or creat your own site map that is mirrored in tree order according to how you set up your pages and make sure all your pages are linked together.

what this will do is when the bots revisit the website that is now linked to you, it will see your link and follow it. if it follows it and your site hasn't already been submitted, then google will automatically submit your site for you. now some website put the revisit metatag in their head. i believe this is a useless tag as google will revisit when it guesses how often you update for new content. so make sure you link to a page already indexed in google that always has new content every week or two.

i've heard of people getting listed this way the same day. but usually 1-2 weeks if the site linking to you has good content. ALSO....the even better news about this way of getting your site listed is that google loves new web pages it finds and some say(a rumor that i believe is true) will get you a higher listing than if you submitted your site directly to google either by hand or with an autosubmitter.

now trap17 is indexed by google. if you post a link to your site and use your keywords and a good description within your post in the appropriate sub forum, then you wont have to submit your site at all. the search engines will find you just like they will find your other posts that you post on trap 17. i've already seen my posts from her in the search engines accidently....

anyway, this information was given to the best of my knowledge right now. i haven't updated myself in the last 1-2 months with search engine placement and seo, but i'm sure if you take to heart what i've said, you'll have no problem getting listed....BUT! with a bot....but quicker than you would think and faster than alot of other people get listed wink.gif if you really want to get listed without a bot though, go to yahoo and submit your site.....dmoz or aol. i think inktomi powers aol....but again, i know i might be off in some of my facts here.

wish ya luck with whatever you're trying to accomplish bud.
Go to the top of the page
 
+Quote Post
Lyon2
post Mar 31 2006, 07:28 AM
Post #5


The Ethical Hacker
***********

Group: [HOSTED]
Posts: 1,154
Joined: 27-May 05
From: Portugal (Europe)
Member No.: 7,566



anwiii, sorry but i must correct you in somethings you're saying:

Yahoo doesn't pay their team to look up the website, that's done with scripts, and in google is the same thing, imagine to visit more then 10 billion web pages, can you imagine how many workers will you need to look up the web page one by one?

It matter who your site exchange or even get links from.
It's better having 3 links from high popular websites in google that 100 links from not popular websites.

Go to the top of the page
 
+Quote Post
amhso
post Apr 3 2006, 02:32 AM
Post #6


Super Member
*********

Group: Members
Posts: 426
Joined: 24-September 05
Member No.: 12,212



as someone said before, bots are really resource hogs. They search millions of sites a day, using a lot of CPU/mem. There isnt much point to making one. Except i remember on the news a few years ago, a kid in college made a more efficient bot/server than google had for a search engine...so i think google hired him.
Go to the top of the page
 
+Quote Post

Reply to this topicStart new topic

Collapse

> Similar Topics

Topics Topics
  1. Apple Makes Everything Safari(0)
  2. Sony Vaio Cd/dvd Rom Drive Makes So Much Noise!(12)
  3. Beheading Suspect Makes Court Appearance.(6)
  4. J2me Somebody?(6)
  5. Arrogant People! Arg(10)
  6. Google Talk Translation Bots(2)
  7. Smoking Makes You Ugly And Unnatractive(36)
  8. Bot's Cs Source ! Need Help !(5)
  9. How Do You Know When You Don't Love A Person Anymore?(13)
  10. My Chemistry Teacher Makes Me Mad!(4)
  11. Share What Makes Your Pet Unique Here(10)
  12. Lucky Pic Of My Dog(2)
  13. Funny Flash Quiz.. It Makes Me Laugh.. Lol.. (for Boys Only)(11)
  14. What To Look In Antivirus(6)
  15. Global Warming(16)
  1. Email Address Protection From Spam Bots(15)
  2. Encode Your Email Address(5)
  3. Saint Michael Makes A Movie Review Comback With The Movie The Comebacks(0)
  4. Motorola Makes Its Midp 3.0 Open Source(0)
  5. What Makes The Web So Great?(10)
  6. The Thing That Made My Today(1)
  7. War Makes Sense(18)
  8. The Little Things...(7)
  9. The Basis Of Justification(9)
  10. Jealousy Makes You Nasty...(7)
  11. It Makes Money From House And In Internet Without Investing Nor To Dolar(1)
  12. I Thought I'd Do A Sig...(5)
  13. Unedible Food(24)
  14. 301 Redirect And Bots(6)
  15. Wordpress Upgrade Makes Increase In Disk Space(3)


 



- Lo-Fi Version Time is now: 6th September 2008 - 06:29 PM