Website Recursive Url Parser

In this article I am trying to share a piece of code that might be useful to some of the developers.

We can find a lot of code in C# that will parse the http urls in given string. But it is difficult to find a code that will:

• Accept a url as argument, parse the site content
• Fetch all urls in the site content, parse the site content of each urls
• Repeat the above process until all urls are fetched.

Scenario

Taking the website http://valuestocks.in (A Stock Market Site) as example I would like to get all the urls inside the website recursively.

Design

The main class is SpiderLogic which contains all necessary methods and properties.



The GetUrls() method is used to parse the website and return the urls. There are two overloads for this method.

The first one takes 2 arguments. The url and and a Boolean indicating if recursive parsing is needed or not.

E.g.: GetUrls(http://www.google.com", true);

The second one is 3 arguments, url, base url and recursive Boolean.

This method is intended for usage like the url is a sub level of the base url. And the web page contains relative paths. So in order to construct the valid absolute urls, the second argument is necessary.

E.g.: GetUrls("http://www.whereincity.com/india-kids/baby-names/ ", http://www.whereincity.com/ , true);

Method Body of GetUrls()

public IList<string> GetUrls(string url, string baseUrl,
bool recursive)
{
if (recursive)
{
_urls.Clear();
RecursivelyGenerateUrls(url, baseUrl);

return _urls;
}
else
return InternalGetUrls(url, baseUrl);
}

InternalGetUrls()

Another method of interest would be InternalGetUrls() which fetches the content of url, parses the urls inside it and constructs the absolute urls.

private IList<string> InternalGetUrls(string baseUrl, string absoluteBaseUrl)
{
IList<string> list = new List<string>();

Uri uri = null;
if (!Uri.TryCreate(baseUrl, UriKind.RelativeOrAbsolute, out uri))
return list;

// Get the http content
string siteContent = GetHttpResponse(baseUrl);

var allUrls = GetAllUrls(siteContent);

foreach (string uriString in allUrls)
{
uri = null;
if (Uri.TryCreate(uriString, UriKind.RelativeOrAbsolute, out uri))
{
if (uri.IsAbsoluteUri)
{
if (uri.OriginalString.StartsWith(absoluteBaseUrl)) // If different domain / javascript: urls needed exclude this check
{
list.Add(uriString);
}
}
else
{
string newUri = GetAbsoluteUri(uri, absoluteBaseUrl, uriString);
if (!string.IsNullOrEmpty(newUri))
list.Add(newUri);
}
}
else
{
if (!uriString.StartsWith(absoluteBaseUrl))
{
string newUri = GetAbsoluteUri(uri, absoluteBaseUrl, uriString);
if (!string.IsNullOrEmpty(newUri))
list.Add(newUri);
}
}
}

return list;
}

Handling Exceptions

There is an OnException delegate that can be used to get the exceptions occurring while parsing.

Tester Application

A tester windows application is included with the source code of the article.
You can try executing it.

The form accepts a base url as the input and clicking the Go button it parses the content of url and extracts all urls in it. If you need a recursive parsing please check the Is Recursive check box.



Next Part

In the next part of the article, I would like to create a url verifier website that verifies all the urls in a website. I agree after doing a search we can find free providers like that. My aim is to learn & develop a custom code that could be extensible and reusable across multiple projects by community.