Under the present study, Web Crawler simulator has been designed than analyze the different web crawling algorithm to evaluate their performance. Web crawler is a computer program or software. Web crawler is an essential component of search engines, data mining and other Internet applications. Scheduling Web pages to be downloaded is an important aspect of crawling. Previous research on Web crawl focused on optimizing either crawl speed or quality of the Web pages downloaded. While both metrics are important, scheduling using one of them alone is insufficient and can bias or hurt overall crawl process. This paper is all about the comparative study of scheduling algorithm for Web Crawling using VB.NET Technology

Download PDF

References

1. http://en.wikipedia.org/wiki/Web_crawler#Examples_of_Web_crawlers
2. http://www.chato.cl/papers/castillo04_scheduling_algorithms_web_crawling.pdf
3. http://ieeexplore.ieee.org/iel5/2/34424/01642621.pdf
4. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.1.9569&rep=rep1&type=pdf.
5. http://dollar.biz.uiowa.edu/~pant/Papers/crawling.pdf
6. Marc Najork, Allan Heydon SRC Research Report 173, “High-Performance Web
7. Sergey Brin and Lawrence Page, ”Theanatomy of a large-scale hyper textual Web search engine”, In Proceedings of the Seventh International World Wide Web Conference, pages 107–117, April 1998
8. . [Ard¨o A]. (2005). “Combine Web crawler,” Software package for general and focused Web-crawling. http://combine.it.lth.se/.

Back