The number of web pages is increasing into millions and trillions around the world. To make searching much easier for users, web search engines came into existence. Web Search engines are used to find specific information on the World Wide Web. Without search engines, it would be almost impossible to locate anything on the Web unless or until a specific URL address is known. This information is provided to search by a web crawler which is a computer program or software. Web crawler is an essential component of search engines, data mining and other Internet applications. Scheduling Web pages to be downloaded is an important aspect of crawling. Previous research on Web crawl focused on optimizing either crawl speed or quality of the Web pages downloaded. While both metrics are important, scheduling using one of them alone is insufficient and can bias or hurt overall crawl process. This paper is all about design a new Web Crawler using VB.NET Technology.

Download PDF

References

1. http://en.wikipedia.org/wiki/Web_crawler#Examples_of_Web_crawlers
2. http://www.chato.cl/papers/castillo04_scheduling_algorithms_web_crawling.pdf
3. http://ieeexplore.ieee.org/iel5/2/34424/01642621.pdf
4. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.1.9569&rep=rep1&type=pdf.
5. http://dollar.biz.uiowa.edu/~pant/Papers/crawling.pdf
6. Marc Najork, Allan Heydon SRC Research Report 173, “High-Performance Web Crawling”, published by COMPAQ systems research center on September 26, 2001.
7. Sergey Brin and Lawrence Page, ”Theanatomy of a large-scale hyper textual Web search engine”, In Proceedings of the Seventh International World Wide Web Conference, pages 107–117, April 1998.
8. [Ard¨o A]. (2005). “Combine Web crawler,” Software package for general and focused Web-crawling. http://combine.it.lth.se/.

Back