Question for mahesh

Question

I am planning to create my own search engine .for that i have idea to create a crawler/bot that indexes the web but don't have idea where to host it so it can index web 24/7 and add new pages automatically to database ..
I prefer c# as language to code it .
also i want guidance on which type of database will be better for crawl engine Queue .
any idea about till which level I have to crawl any link ?

Mahesh Chand · Answer

Besides the crawling technology I can tell you about technology to build your crawler. You can definitely use MySQL as a database. If you use SQL Server you will have to pay thousands $$$ to buy a production license. As per language goes I think you will have to use more than C#. You definitely will need lot of JavaScript C++ or some other languages. I am not sure just alone C# can do this work. You will also need network and socket programming. Did you look at these samples? http://www.codeproject.com/KB/IP/Crawler.aspx How To: Write a Web Crawler in C# - The Coding Humanist Cheers!

Kirtan Patel · Answer

thanks jinge .. some how it helped me but Lucene is not integrating with sql server no doubt lucene is good in query etc ... is there any better crawler that can do that ??

jinge · Answer

Maybe you can learn sth. from lucene.net from http://lucene.apache.org/lucene.net/.

Kirtan Patel · Answer

Ok thanks for replying :)

Mahesh Chand · Answer

Kirtan To be frank I have not done this kind (crawling) of work and would not have any idea. I have built many commerial applications and websites but have not touched this topic.