List of search engine spider IP addresses

List of search engine spider IP addresses

Post by Larr » Sun, 04 Apr 1999 04:00:00



Is a list of search engine spider IP addresses available?

I am wondering if a list is maintained anywhere.

I imagine they might be pretty dynamic and change often.

Any help would be appreciated.

Larry

 
 
 

List of search engine spider IP addresses

Post by George Girar » Sun, 04 Apr 1999 04:00:00



>Is a list of search engine spider IP addresses available?

>I am wondering if a list is maintained anywhere.

>I imagine they might be pretty dynamic and change often.

>Any help would be appreciated.

>Larry

I think these are correct/current/ok/????:

209.24.48.12 .netmind.com NetMind, URL-Minder (robot) - East Meadow, NY.
206.132.186.131 .green.alexa.com ALEXA, The Crawl (robot)
198.3.99.64 .info.webcrawler.com WebCrawler (robot)
198.3.99.101 .webcrawler.com WebCrawler (robot)
209.1.12.44 .inktomisearch.com Slurp -- Inktomi's Web Robot - San Mateo, CA
209.1.32.126 .inktomi.com Slurp -- Inktomi's Web Robot - San Mateo, CA
204.123.2.67 ww2.altavista.digital.com Alta Vistsa, Scooter (robot)
204.123.9.76 add-url.altavista.digital.com Alta Vistsa, Scooter (robot)
204.123.9.20 scooter.pa-x.dec.com Digital Research, Scooter (robot)
195.101.94.120 alcorak.x-echo.com X-Echo (robot) - Sophia Antipolis, FR.
204.62.245.166 crawl1.atext.com E* - Spider
206.181.238.130 .wisewire.com WiseWire - Spider
207.82.104.100 interpix.com Image Surfer - Excalibur Internet Spider
207.77.91.187 .spider7.srv.pgh.lycos.com Lycos - (Spider)
209.67.228.51 .lycos.com Lycos - (Spider)
209.67.228.54 .lycos.com Lycos - (Spider)
209.67.228.55 .lycos.com Lycos - (Spider)
209.67.228.56 .lycos.com Lycos - (Spider)

George Girard, Programmer/FiddlerArounder
I See All.. I Know All.. I Am The Wizard...
http://www.veryComputer.com/~georgegg/The.Wizard's.Mysterious.Page


 
 
 

List of search engine spider IP addresses

Post by Neil Oost » Mon, 05 Apr 1999 05:00:00


: Is a list of search engine spider IP addresses available?

Just wondering why you would want one....? If you want to prevent those
IP addresses from accessing your site, you need not do it that way.

If you place a robots.txt file in your html root directory, it will
prevent most from indexing your site. Of course, you need to have the
proper syntax for the lines in the file. Sorry, I don't have that on me.
But I'm sure if you do a search on the web you'll come up with something.

HTH


+------------------------------------+
| Thanks to you, my mom has no hair! |
+--->  http://www.hair.ica.net/  <---+

--

 
 
 

1. Need search engine/spider with proximity function

hi,

i am looking for a site indexer/spider/... with the functionality of
proximity searching. i am using htdig now, which is an excellent tool,
but alas..no proximity. (btw, i am talking about a apache/linux
platform)

does anyone know of such a tool?

thanx,
/steven
-
"The day Microsoft makes something that doesn't suck is the day they
start making vacuum cleaners..."

2. NEED HELP: NS Enterprise Suitespot Group?

3. search engines spider pages:200 OK aaaaarrrhg please help

4. Memory tuning?

5. Home Page Listing For Search Engines

6. FTP problem with data connection

7. Search Engine for URL Lists?

8. EPSON stylus COLOR driver!

9. how do i get top priority search engine listing

10. FreeBSD.org: Mailing list search engine

11. Top Search Engine Listing Guide

12. Search engine listing

13. Internet Rank Engine-the next generation of search engines