4
Answers

Web Scrapping in c#

navneet  kaur

navneet kaur

7y
303
1
-I am using selenium and PhantomJSDriver for web scrapping.I am facing the issue for scrapping list of items for wbsite.The result is not consistent,I get different number of records each time.To overcome this,I use explicit and implicit wait for webdriver.But it through timeout error for that case.
 
driver.FindElementsByClassName("Tile").ToList()
 
retrive different number of records.and if I use debugger and wait for some time then it retrive  all records.
  1. driver.Url = uri.ToString() ;  
  2. driver.Navigate();  
  3. if (uri.PathAndQuery == "/")  
  4. {  
  5. //This is synchronous, it will not go to the next line until the crawl has completed  
  6. driver.FindElementsByClassName("Tile").ToList().ForEach(x =>  
  7. {  
  8. objUPCProcess = new UPCProcessingModel();  
  9.   
  10. try  
  11. {  
  12. objUPCProcess.Description = x.FindElement(By.ClassName("Tile-heading")).Text;  
  13. objUPCProcess.Price = x.FindElement(By.ClassName("Price-display")).Text;  
  14. listOfProductDetails.Add(objUPCProcess);  
  15. if (i == 0)  
  16. {  
  17. log.Item = objUPCProcess.Description;  
  18. i++;  
  19. }  
  20. }  
  21. catch (OpenQA.Selenium.NoSuchElementException ex)  
  22. {  
  23. log.Message = ex.Message;  
  24. log.Status = "Error";  
  25. }  
  26. });  
  27. }  
 
Answers (4)