Is a list of search engine spider IP addresses available?
I am wondering if a list is maintained anywhere.
I imagine they might be pretty dynamic and change often.
Any help would be appreciated.
Larry
I am wondering if a list is maintained anywhere.
I imagine they might be pretty dynamic and change often.
Any help would be appreciated.
Larry
>I am wondering if a list is maintained anywhere.
>I imagine they might be pretty dynamic and change often.
>Any help would be appreciated.
>Larry
209.24.48.12 .netmind.com NetMind, URL-Minder (robot) - East Meadow, NY.
206.132.186.131 .green.alexa.com ALEXA, The Crawl (robot)
198.3.99.64 .info.webcrawler.com WebCrawler (robot)
198.3.99.101 .webcrawler.com WebCrawler (robot)
209.1.12.44 .inktomisearch.com Slurp -- Inktomi's Web Robot - San Mateo, CA
209.1.32.126 .inktomi.com Slurp -- Inktomi's Web Robot - San Mateo, CA
204.123.2.67 ww2.altavista.digital.com Alta Vistsa, Scooter (robot)
204.123.9.76 add-url.altavista.digital.com Alta Vistsa, Scooter (robot)
204.123.9.20 scooter.pa-x.dec.com Digital Research, Scooter (robot)
195.101.94.120 alcorak.x-echo.com X-Echo (robot) - Sophia Antipolis, FR.
204.62.245.166 crawl1.atext.com E* - Spider
206.181.238.130 .wisewire.com WiseWire - Spider
207.82.104.100 interpix.com Image Surfer - Excalibur Internet Spider
207.77.91.187 .spider7.srv.pgh.lycos.com Lycos - (Spider)
209.67.228.51 .lycos.com Lycos - (Spider)
209.67.228.54 .lycos.com Lycos - (Spider)
209.67.228.55 .lycos.com Lycos - (Spider)
209.67.228.56 .lycos.com Lycos - (Spider)
George Girard, Programmer/FiddlerArounder
I See All.. I Know All.. I Am The Wizard...
http://www.veryComputer.com/~georgegg/The.Wizard's.Mysterious.Page
Just wondering why you would want one....? If you want to prevent those
IP addresses from accessing your site, you need not do it that way.
If you place a robots.txt file in your html root directory, it will
prevent most from indexing your site. Of course, you need to have the
proper syntax for the lines in the file. Sorry, I don't have that on me.
But I'm sure if you do a search on the web you'll come up with something.
HTH
+------------------------------------+
| Thanks to you, my mom has no hair! |
+---> http://www.hair.ica.net/ <---+
--
1. Need search engine/spider with proximity function
hi,
i am looking for a site indexer/spider/... with the functionality of
proximity searching. i am using htdig now, which is an excellent tool,
but alas..no proximity. (btw, i am talking about a apache/linux
platform)
does anyone know of such a tool?
thanx,
/steven
-
"The day Microsoft makes something that doesn't suck is the day they
start making vacuum cleaners..."
2. NEED HELP: NS Enterprise Suitespot Group?
3. search engines spider pages:200 OK aaaaarrrhg please help
5. Home Page Listing For Search Engines
6. FTP problem with data connection
7. Search Engine for URL Lists?
9. how do i get top priority search engine listing
10. FreeBSD.org: Mailing list search engine
11. Top Search Engine Listing Guide
13. Internet Rank Engine-the next generation of search engines