Testing Flaky .NET Tests with Historical Failure Analysis

A flaky test is one of the most frustrating problems in a software delivery pipeline.

The test passes locally. It passes in CI five times. Then it fails once without an obvious code change.

Developers rerun the pipeline.

It passes.

The team moves on.

Over time, this creates a dangerous pattern:

Code

Test fails
   |
   v
Rerun
   |
   v
Test passes
   |
   v
Ignore failure
   |
   v
Flaky test becomes normal

Eventually, developers stop trusting the test suite.

That is much more serious than a single unreliable test. Automated tests are supposed to provide confidence. If failures are routinely dismissed as "probably flaky," genuine regressions can be hidden among the noise.

The solution is not simply to rerun failed tests more often.

A better approach is to collect historical test results and analyze failure patterns over time.

This article explains how to identify flaky .NET tests using historical execution data, calculate useful metrics, build a failure history, and turn recurring patterns into actionable engineering work.

What Is a Flaky Test?

A flaky test is a test whose result is nondeterministic even when the relevant code and environment have not intentionally changed.

For example:

Code

Run 1 -> Pass
Run 2 -> Pass
Run 3 -> Fail
Run 4 -> Pass
Run 5 -> Pass

The test is not consistently broken.

Its result depends on something that is not fully controlled.

Common causes include:

Timing assumptions
Race conditions
Shared test state
Database state
Network dependencies
Parallel execution
File-system behavior
Random data
Time-zone assumptions
External services
Async synchronization problems
Resource exhaustion
Environment differences

The important distinction is:

Code

Consistently failing test
        !=
Flaky test

A consistently failing test is usually easier to diagnose.

A flaky test requires historical evidence because individual executions can look perfectly normal.

Why Historical Analysis Matters

Suppose a test has failed twice.

That alone does not tell you much.

Now consider its history:

Code

Last 100 executions

Pass: 94
Fail: 6

The failure rate is:

Code

6 / 100 = 6%

That is meaningful.

Now compare another test:

Code

Last 100 executions

Pass: 99
Fail: 1

The second test may still deserve investigation, but the operational priority is different.

Historical analysis lets teams prioritize based on evidence rather than anecdotes.

Start With a Test Result Dataset

A useful historical record should contain more than pass/fail.

For each execution, capture information such as:

Field	Example
Test name	OrderServiceTests.CreateOrder_ShouldPersist
Project	OrderService.Tests
Result	Passed
Duration	1.42 s
Timestamp	2026-08-17T08:15:00Z
Commit	a82f31c
Branch	main
Build ID	18452
Agent	windows-runner-03
Framework	net10.0
Failure message	Timeout
Retry number	0

The more context you retain, the easier it becomes to identify patterns.

For example, a test might appear flaky globally but actually fail only on:

Code

Linux
+
Parallel execution
+
Specific test runner

That is a much more actionable finding.

Calculate a Basic Flake Rate

The simplest metric is the failure rate over a defined number of executions.

Code

Flake Rate =
Number of unexpected failures
/
Total executions

For example:

Code

Executions: 500
Failures:     15

Flake rate = 15 / 500
           = 3%

Do not interpret every failure as a flaky failure automatically.

A test that consistently fails after a known code change is more likely to be a genuine regression.

Historical analysis should classify failures before assigning them to the flake-rate metric.

Track Failure Streaks

Failure rate alone can hide useful information.

Consider:

Code

P P P F P P P P F P

versus:

Code

P P P F F F P P P P

Both contain three failures if the window is larger, but the second pattern may indicate a persistent environmental or code problem.

Track:

Consecutive failures
Consecutive passes
Maximum failure streak
Maximum pass streak

A sudden failure streak often deserves more immediate investigation than isolated historical failures.

Track Failure Recency

Old failures and recent failures should not necessarily receive equal weight.

Consider:

Code

100 executions
Failures:
- 90 runs ago
- 80 runs ago
- 75 runs ago

versus:

Code

100 executions
Failures:
- 3 runs ago
- 5 runs ago
- 8 runs ago

The second pattern is much more relevant to the current state of the system.

A practical dashboard should therefore expose:

Code

Last failure
Failures in last 10 runs
Failures in last 50 runs
Failures in last 100 runs

This provides both long-term and recent context.

Analyze Test Duration

Flakiness is not always represented by failure alone.

A test that normally takes:

Code

200 ms

but occasionally takes:

Code

20 seconds

may have a timing or resource problem even if it eventually passes.

Track duration statistics:

Average duration
Median duration
p95 duration
Maximum duration
Duration variance

For example:

Metric	Value
Median	210 ms
p95	340 ms
Maximum	18.4 s

That is a useful investigation signal.

The test may be sensitive to a slow database, thread scheduling, contention, or an external dependency.

Correlate Failures With Execution Environment

One of the strongest benefits of historical analysis is finding environmental correlations.

Suppose the history contains:

Environment	Runs	Failures
Windows	500	2
Linux	500	18

The overall failure rate hides the important detail.

Linux is clearly worth investigating.

Continue breaking the data down by:

Code

Operating system
.NET version
CPU architecture
Runner
Database version
Container runtime
Browser version
Test category
Parallelism

The relevant dimensions depend on the test.

Analyze Parallel Execution

A common source of flaky tests is shared state.

For example:

public class OrderTests
{
    [Fact]
    public async Task CreatesOrder()
    {
        await database.ClearAsync();

        // Test...
    }
}

If another test is simultaneously modifying the same database, the result may depend on execution order.

The test may pass in isolation:

Code

Test A
  |
  v
Pass

but fail when the complete suite runs:

Code

Test A ----+
           |
Test B ----+----> Shared database
           |
Test C ----+

Historical data can reveal whether failures increase when test parallelism is enabled.

Capture Failure Categories

Do not store only the complete exception message.

Classify failures where possible.

For example:

Code

Timeout
Assertion
Connection
File system
HTTP
Database
Concurrency
Serialization
Environment
Unknown

A simple classification makes trends easier to identify.

Suppose a test fails 20 times:

Code

Timeout:      15
Database:      3
Assertion:     1
Unknown:       1

That strongly suggests investigating timeout and database behavior first.

Normalize Failure Messages

Raw exception messages often contain dynamic values.

For example:

Code

Expected 42 but received 41

and:

Code

Expected 43 but received 42

may represent the same underlying failure category.

Likewise:

Code

Timeout after 5000 ms connecting to 10.0.1.20

and:

Code

Timeout after 5000 ms connecting to 10.0.1.21

may represent the same network problem.

Normalize dynamic values before grouping failures.

The goal is to identify recurring failure signatures.

Example Historical Storage Model

A simple relational table could contain:

SQL

CREATE TABLE TestExecutionHistory
(
    Id BIGINT PRIMARY KEY,
    TestName NVARCHAR(500) NOT NULL,
    ProjectName NVARCHAR(200) NOT NULL,
    Result NVARCHAR(20) NOT NULL,
    DurationMs BIGINT NOT NULL,
    ExecutedAtUtc DATETIME2 NOT NULL,
    CommitSha NVARCHAR(64) NULL,
    BranchName NVARCHAR(200) NULL,
    EnvironmentName NVARCHAR(100) NULL,
    Framework NVARCHAR(50) NULL,
    FailureCategory NVARCHAR(100) NULL,
    FailureSignature NVARCHAR(500) NULL
);

This is intentionally simple.

A production implementation can add:

Build identifiers
Test suite
Agent information
Retry count
Pull request ID
Container image
Database version
Runtime version

The schema should reflect the questions the engineering team actually needs to answer.

Query Tests With High Failure Rates

Once historical data exists, start with simple queries.

For example:

SQL

SELECT
    TestName,
    COUNT(*) AS TotalRuns,
    SUM(
        CASE
            WHEN Result = 'Failed'
            THEN 1
            ELSE 0
        END
    ) AS Failures
FROM TestExecutionHistory
GROUP BY TestName
HAVING COUNT(*) >= 30;

You can calculate the failure rate in application code or in SQL depending on the database.

The important point is to avoid ranking tests based on only one or two executions.

A test with:

Code

1 failure / 2 runs

should not automatically outrank:

Code

20 failures / 1000 runs

The second test has much stronger historical evidence.

Add a Minimum Sample Size

A useful rule is to require a minimum number of executions before classifying a test as flaky.

For example:

Code

Minimum history = 30 executions

The exact threshold is a policy decision.

The purpose is statistical sanity.

Without a minimum sample size, a newly added test can immediately appear highly flaky after one failure.

Build a Flakiness Score

A single failure rate is useful but incomplete.

A more practical score can combine:

Code

Failure frequency
+
Recent failures
+
Failure streak
+
Execution volume
+
Duration anomalies

For example:

Code

Flakiness Score =
Weighted Failure Rate
+ Recent Failure Weight
+ Duration Anomaly Weight

The exact formula should be validated against actual engineering outcomes.

Do not pretend that a score is mathematically objective if its weights are arbitrary.

The score is primarily a prioritization mechanism.

Detect Tests That Pass Only After Retry

Retries are particularly useful historical signals.

Consider:

Code

Attempt 1 -> Fail
Attempt 2 -> Pass

If this happens frequently, the test deserves investigation.

Track:

Code

Initial failures
Recovered on retry
Failed after all retries

For example:

Test	Runs	First Attempt Failures	Retry Recoveries
Test A	500	18	16
Test B	500	4	1

Test A is a much stronger flakiness candidate.

Retries should not be used to hide this information.

Do Not Use Retries as the Fix

A retry can be useful operationally:

Code

Test fails
   |
   v
Retry
   |
   v
Pipeline continues

But this does not make the test reliable.

If the test fails 5% of the time and retries make the pipeline green 99% of the time, the underlying reliability problem still exists.

The correct interpretation is:

Code

Retry = mitigation
Not = root-cause fix

Track the initial failure even when the retry passes.

Identify Time-Based Failures

Some flaky tests correlate with specific times.

For example:

Code

Failures:
09:00-10:00 -> 1
10:00-11:00 -> 2
12:00-13:00 -> 17

That may indicate:

Scheduled infrastructure activity
Resource contention
Database backups
Network congestion
Time-zone problems
External service limits

Time-based analysis is especially useful for tests that interact with shared environments.

Identify Commit Correlations

A test may be stable for months and become flaky after a particular change.

For example:

Code

Commit A
100 runs -> 0 failures

Commit B
100 runs -> 12 failures

That is a valuable signal.

The test may have been affected by:

New asynchronous behavior
Changed database access
Modified timeouts
Parallel execution
Shared state
New dependency versions

Historical test data should therefore be correlated with commit history.

Detect Test Order Dependencies

Order-dependent tests are particularly difficult to diagnose.

Imagine:

Code

Test A -> Pass
Test B -> Pass
Test C -> Pass

but:

Code

Test C
Test A
Test B

causes Test B to fail.

Run-history data should capture enough information to reconstruct execution order when parallelism is relevant.

If possible, record:

Code

Test group
Worker
Execution sequence
Parallel worker ID

This can turn a seemingly random failure into a reproducible ordering problem.

Separate Product Failures From Test Failures

Not every intermittent failure is a flaky test.

Suppose an integration test calls an external service.

If the service returns:

Code

HTTP 503

the test may be correctly reporting a real dependency outage.

Historical analysis should therefore distinguish:

Code

Application defect
Test defect
Environment defect
External dependency
Infrastructure failure

This classification prevents teams from rewriting good tests to accommodate unreliable infrastructure.

Build a Flaky Test Dashboard

A useful dashboard can show:

Top Flaky Tests

Code

Test Name                         Flake Rate
------------------------------------------------
OrderServiceTests.CreateOrder        5.2%
PaymentTests.TimeoutHandling         3.8%
CacheTests.ConcurrentRefresh         2.9%

Recent Flaky Tests

Code

Test                     Failures / Last 20
--------------------------------------------
CreateOrder                    4
RefreshCache                   3
ExportReport                   2

Environment Correlation

Code

Environment       Failure Rate
--------------------------------
Windows           0.4%
Linux             3.7%

This provides a much more useful view than a simple "build passed" indicator.

A Practical CI Workflow

A robust workflow can look like:

Code

Run test suite
      |
      v
Collect results
      |
      v
Store execution history
      |
      v
Classify failures
      |
      v
Calculate flakiness metrics
      |
      v
Identify recurring patterns
      |
      v
Create engineering work

The important part is the last step.

Historical analysis should result in action.

Examples include:

Code

Fix race condition
Isolate test data
Remove external dependency
Improve synchronization
Fix time handling
Correct test cleanup
Replace unstable environment

Common Mistakes

Looking Only at Current CI Results

One build rarely provides enough evidence.

Treating Every Failure as Flaky

A genuine regression can be mislabeled as flaky if failures are not investigated.

Ignoring Retry Results

A retry that passes is often one of the strongest flakiness signals.

Using Only Average Failure Rate

Recent failures, streaks, environment, and duration patterns matter.

Keeping Only Test Names

Without commit, environment, framework, and timing context, historical analysis becomes much less useful.

Automatically Quarantining Every Flaky Test

Quarantine can reduce pipeline noise, but it can also allow important failures to disappear from normal development.

Use it deliberately.

Deleting Historical Data Too Quickly

A short history makes it difficult to determine whether reliability is improving or getting worse.

Troubleshooting a Suspected Flaky Test

When a test is suspected of being flaky, start with its recent history.

Check:

Code

1. Last 100 executions
2. Failure rate
3. Last failure
4. Retry recovery rate
5. Failure signatures
6. Execution duration
7. Environment
8. .NET version
9. Parallelism
10. Related commits

Then reproduce it deliberately.

For example:

Bash

dotnet test \
    --filter "FullyQualifiedName~CreateOrder" \
    --no-restore

Run the test repeatedly under the same conditions.

If it only fails under parallel execution, test the parallelism hypothesis.

If it only fails on one operating system, investigate environment-specific behavior.

Historical data should narrow the search space.

It should not replace actual debugging.

A Practical Flaky-Test Classification

After analysis, classify tests into categories such as:

Classification	Meaning	Action
Stable	No meaningful intermittent failures	Keep
Suspected flaky	Limited evidence	Monitor
Flaky	Repeated intermittent failures	Investigate
Environment-dependent	Failures correlate with environment	Fix environment/test
External dependency	Failure depends on external service	Isolate or control dependency
Consistently failing	Reproducible failure	Treat as defect

This prevents the word "flaky" from becoming a generic label for every difficult test.

Frequently Asked Questions

How many runs are needed to identify a flaky test?

There is no universal number. A reasonable minimum history should be large enough to distinguish a recurring pattern from a one-off failure. Many teams can start with dozens of executions and increase the window for low-frequency tests.

Should failed tests automatically be retried?

Retries can reduce pipeline disruption, but they should never erase the original failure from historical analysis. A test that fails and then passes on retry is an important flakiness signal.

Should flaky tests be removed from CI?

Usually not as a first response. Removing a test eliminates its signal rather than fixing the underlying reliability problem. Temporary quarantine can be appropriate when a test seriously blocks delivery, but it should have ownership and follow-up.

Can historical analysis find race conditions?

It can provide strong clues. Correlations with parallel execution, test ordering, timing, CPU load, or specific environments can point toward concurrency problems, but the race condition still needs to be reproduced and diagnosed.

Should test duration be tracked?

Yes. Intermittent latency spikes can reveal resource contention, timeouts, synchronization issues, or unstable dependencies even when the test ultimately passes.

Is a low flake rate acceptable?

It depends on the test's importance and the cost of failure. A 1% failure rate can still be unacceptable for a test executed thousands of times in CI if it creates significant developer disruption.

Conclusion

Flaky tests become manageable when teams stop treating them as isolated CI annoyances and start treating test execution as historical data. A single failure rarely explains why a test is unstable, but a history of hundreds of executions can reveal failure frequency, retry recovery, timing anomalies, environment correlations, execution-order dependencies, and relationships with recent code changes.

For .NET teams, the practical goal is not to create a complicated analytics platform. Start by storing test results with enough context to answer basic questions: when did the test fail, how often does it fail, does it pass on retry, where does it fail, and what changed around the failure? From there, use historical evidence to prioritize investigations and fix the underlying reliability problem. A trustworthy test suite is not one that never fails; it is one where failures have a predictable meaning and intermittent failures are actively identified and removed.