Wrap your brains around this little item:
Where will it end? This is an old article, but relevant
Read it and weep (be sure to highlight the whole url, which will likely
Now, for the next 20 years, anyone who uses the simple, obvious "spider"
approach for indexing and/or searching the Internet must pay Lycos a
royalty, or suffer the consequences.