Building a search engine

[Don Logan]

Hi

I have to build a specialised search engine and want to know the best way to process large volumes of random pages from the web. I have no prior SE building knowledge and therefore little idea as how to proceed. Basically I need to index pages in a particular way which seems to be the easy part, I just don’t know how to get the pages off the web in the first place. I have looked on the web and most of the info there refers to intranets or large sites where the location of pages is assumed to be known. A vague and naïve question but hopefully you get he nub of my gist.

[SpectateSwamp]

Hi

I have built one.

First I had it do text search, with hiliting and line wraps etc.

Then I had it display a picture when a match was found and
the next line was the path to a picture.

It could easily be altered to have a url instead of a picture.
every line of text in the website would have a matching line
pointing to the page it came from.

It strictly works from text files. To speed thing up and eliminate
all the directory overhead. I merge all the files I want to search
into a large file. Then search that file, listing the file name it came
from. (When merging, I place a break between files with the file name in it)

All my emails, notes and net clippings since 1999 is 75MB
At 20,000,000 characters per second, it only takes 3 seconds
to search everything. No indexing required and the data is
portable.