How many threads can we run at a time.

Question

I am scraping a website and want to fetch data from it by multiple threads.... I want as many threads as to get data but after few pages it says 404-page not found OR Connection Time Out. Any solution please help!!!

Sandeep Singh Shekhawat · Answer

We are using thread for fetching data. Sometimes it is possible that one thread depends on another so that situation second thread can't work without first thread output data. So this state is called deadlock. AS per my understanding you are facing deadlock situation and it may be possible that your thread is sleep. So you must sure that your thread is running when you are accessing data. I will suggest to avoid timeout and page not found that make a thread asynchronous call of thread. or check lock condition in thread so when this thread working on data then another thread can't work on same data lock( this ) { _calcState = CalculationStatus.NotCalculating; FireStatusChangedEvent( _calcState ); }