aspSpider by anorakgirl

aspSpider is a basic site indexer, designed for datadriven sites where links might be generated automagically from a database (e.g. productdetail.asp?productid=4). The spider is supplied with the url of a start page,and then crawls through all the pages in the site following links and requesting the pages via http. The page url, keywords and description are added to an access database. This data can then be used to drive a search engine for the site. Because the indexer follows links within the site, there is no need to maintain a list of pages in the site. The indexer can be configured to include only certain file types and to ignore certain directories. At present the whole thing is fairly basic, so some ASP coding skills may be required to incorporate the search engine into your site. aspSpider is written in ASP (javascript) and uses an Access 2000 database to store data.