An Adaptive, Selective and Incremental Web Crawler

Pallavi, Rajiv Sharma

Abstract: The large size and the dynamic nature of the Web make it necessary to continually maintain Web based information retrieval systems. Crawlers facilitate this process by following hyperlinks in Web pages to automatically download new and updated Web pages.The expansion of the World Wide Web has led to a chaotic state where the users of the internet have to face and overcome the major problem of discovering information. For the solution to this problem, many mechanisms were created based on crawlers who are browsing the www and downloading pages. In this paper we will describe a crawling mechanism which is created in order to support data mining and processing systems and to obtain a history of the web?s content.

Keywords: Web, web crawler, URL, Crawling policies, Incremental and Adaptive Crawler, Search Engine.