Quote:>> I have a client that wants to allow their site to be searched &
>> indexed by robots, but who want users visiting the site to fill out a
>> survey form before being allowed access.
The BrowserMatch directive should let you set an environment variable
to say if they're a spider or not. You then make index.html an alias
for a cgi script that will pass back a different Redirect header. So
spiders end up at "realindex.html" and others end up at "survey.html".
It would be nice if we could define directives that say when someone
asks for index.html, show indexn4.html if Netscape 4, indexn3.html if
Netscape 3, etc. Then we could make top pages that suit the browser
features. I couldn't see anything in Apache that can do this without
resorting to cgi/modules however.
Quote:>There is no absolute way to know the difference between a real user and a
>robot. Users of either type can easily fake being the other.
Spiders are normally honest, and if not they're probably not a real
search engine, so who cares where they go. Almost all users have
browsers where the user agent can't be changed, and the rest are very
unlikely to pretend to be a spider.