This blog assumes you have
a basic understanding of automation testing using NUnit and running them from
command line. For starters, NUnit is the most preferred tool for writing unit
tests for .NET based projects. Unit testing is about writing code to test your
own or other’s code.
Acceptance Tests and Behavior Driven Development (BDD) is fast catching up with Agile teams where apart from developers and tests, business analysts (BA) are also part of feature teams; and SpecFlow is becoming the preferred choice of writing such tests. In SpecFlow you write tests in Gherkin, an English-like language that BAs can understand and contribute in reviewing and writing. For writing automated tests in SpecFlow for web-based applications developed in ASP.NET (or any other platform for that matter), we can use browser automation tools like Selenium. The problem faced is the slowness of execution of such browser based automation tests.
With the advent of NUnit 3 parallel test execution feature, executing tests within the same test binary in parallel has become possible. However when the test output is fetched to SpecFlow Reporting, the reporting engine is not able to transform it into expected HTML output. This is primarily because when tests are executed in parallel, the text output from individual tests is not in a sequential order. The issue has been raised with SpecFlow project on GitHub.
Now I'll focus on the solution for this issue while the maintainers of SpecFlow fix the issue and release it for download. There are two parts of the problem. First, with NUnit 3, the text output begins with '=>', whereas in lower version of NUnit it begins with "***". Latest version of SpecFlow i.e. 2.0 at the time of writing, doesn't work when input line begins with '=>'. The fix is trivial and is available here in GitHub. To apply the fix, you need to download the source code of SpecFlow from GitHub and apply the fix on the code. The binaries you build will have the fix in there.
The second issue is bit trickier. As mentioned above, the text output from NUnit 3.0 parallel execution makes it impossible to identify the start and end of a specific test. To overcome this issue, we can mark each output line with the name of the test case. While rendering the report, the entire text file is scanned for matching entries, thus clubbing text output for particular tests. The fix sounds logical, but implementation has another problem. NUnit 3.0 does no longer support console output. The new approach given by NUnit team is to use TestContext.Out. When SpecFlow engine runs tests, calls to Console.WriteLine are replaced with TestContext.Out. Fix is also available in GitHub here.
After applying both the fixes, and marking your assembly with
[assembly: Parallelizable(ParallelScope.Fixtures)]
as suggested in SpecFlow documentation, you will see your tests are running in parallel. Tests in a SpecFlow feature run in parallel with other features. Tests within the same feature will run sequentially. The degree of parallelism is limited by the number of cores in your machine. In a Continuous Integration (CI) environment the build agent can run the test in parallel leading to a faster feedback cycle. To take maximum advantage of all the cores in the build agent, you should balance out tests among features. When your test suite is small you may not feel the need for parallel execution however as you increase them you will realize the need for running them in parallel. In environments where automated tests run as part of CI builds and concepts like gated check-in are in place, running tests quickly will always be desirable, without compromising the extent of code coverage.
To summarize, SpecFlow BDD combined with Selenium based tests written in NUnit are a good choice for maintaining high code quality and faster iterations in Agile for web based applications. Clubbed with parallel execution, this is even better!