At nine in the morning, the build fails. Someone checks the error, sees that the records came back in a different order than expected, reloads the page, finds nothing wrong, and decides the test is broken. They relax the test, turn the build green, and move on.
That decision might be correct, but it hasn’t really been proven, just assumed.
A red test is a disagreement, not a verdict
Researchers have a term for the tricky part of this situation, and it isn’t just “testing”. It’s called the oracle problem: running a program is easy, but deciding if the output is correct needs a source of truth, like a specification, contract, known relationship, or an expert who understands the domain.
So, a failed test only tells you one thing: two answers don’t match. Whether the problem is in the code or the expectation, the test result alone doesn’t explain why.
Three stories usually fit the evidence:
- The code broke a promise somebody was entitled to rely on;
- The test froze a detail nobody ever promised;
- The promise was never put in writing, and at present two reasonable courses of action are contesting the empty chair.
It’s important to keep these two terms separate. A brittle test fails after a small change because it relies on something accidental. A flaky test gives different results even when the code hasn’t changed, often because of timing, ordering, or shared state. A test can be brittle, flaky, both, or neither, and the solutions stay the same.
Flaky tests aren’t harmless. In a study of 201 flaky test fixes across 51 open source projects, about a quarter of the fixes changed the code being tested, and 94% of those changes fixed real bugs. So, if a rerun passes, that’s actually a sign something might be wrong, not proof that everything is fine.
To whom can one rely on this?
The main question is: who should be able to rely on this behavior? If it’s a user, a caller, a published standard, or a documented rule, then you’re dealing with a contract. But if the only reason is that ‘the test saw it once’, then you’re treating an accident as something important. Google’s approach is different: a good test changes when requirements change, but stays the same during a refactor that doesn’t affect behavior. If a simple refactor makes you change the test, that’s a sign.
Two rows tie, and the test crowns a winner
Regarding that morning failure, the results are arranged according to a certain score, and when two records are tied, the database returns them in the order determined by the query plan. PostgreSQL is clear on this point: when there is no explicit sorting, the order of the rows is not specified, and the documentation for LIMIT and OFFSET suggests the use of a unique ordering since otherwise the slice you end up with is unpredictable.
There are two solutions, and you can’t swap one for the other. If the order doesn’t matter to users, don’t pretend there’s a sequence. Instead, compare the set of identifiers, check that each row matches the filter, and make sure there are no duplicates or missing rows.
Order only matters when it’s important, like for ranking, pagination, or lists that shouldn’t change when someone moves them. In those cases, production should set a total order, not just a main sort key. You need to define how ties are broken, if the tie-breaker is stable, how case, collation, and nulls are handled, the direction for each key, and whether a new row can push another across a page boundary.
Picking the wrong solution can be subtle but expensive. If you swap an exact sequence for ‘same set’, the build will pass, but a ranking bug might slip by. The other fix has its own risk: once a tie-breaker makes the output stable, clients may start depending on that stability, even if you never promised it. Hyrum’s Law applies to your own API too, so be clear about whether the tie-break is guaranteed, best-effort, or just internal.
Exact answers from data you don’t own
The second type involves a test calling a live service and asserting a specific value obtained from live data. As Google states in its section on test doubles, a test that is reading an external web page fails if the server is busy or the content changes. Bazel treats this as a rule in that a test must only interact with its declared dependencies, otherwise the results will no longer be reproducible.
A typical test involves being asked four questions together.
- Is the algorithm correct? By means of using fixed fixtures and specifying the expected outputs.
- Does the adapter understand the terms that it has agreed to? Those concerned and the fixed elements; the schema, the error routes, the variants.
- Does the live service still respect it? This is a periodic contract test which is outside the gating path. Fowler’s view is that matching does not have to mean identical data, only that the essential structure is the same.
- Is the user’s flow now healthy? This is a synthetic probe based on a user-visible outcome and has its own severity.
Choose assertions that match the product’s promise, not just what you’re used to. When the value is part of a contract and all inputs are controlled, like cryptographic vectors, money, or protocol status codes, use exact equality. If you can allow some wiggle room, set a tolerance. Pytest notes that strict float and timing checks often cause flaky tests, and PEP 485 says there’s no one-size-fits-all tolerance. If the promise is ‘A is closer than B’ instead of a fixed string, use a relative order.
If getting an exact answer is expensive or unreliable, it’s better to check the relationships. For example, if the coordinates are the same, the distance should be zero. Swapping the two endpoints shouldn’t change the result. If you move a point a little along a straight line, it shouldn’t drop in rank. These relationships should be reviewed by someone who knows the domain, since even a reasonable check might be wrong if the real system doesn’t match your mental model. But these checks are valid when using real data.
Instead of just guessing, you can widen the tolerance until the test suite meets the requirement. This removes the need for unnecessary approximations.
A cache changes the path, not the promise
The system becomes modal because its behaviour varies depending on whether or not the item has been cached. According to RFC 9111, a cache must invalidate the target URI after carrying out an unsafe request, that is, after a PUT, POST, or DELETE, so that the next time it is read it will follow a different path. If the cache is warm, you might be viewing a representation that has remained unchanged for several hours; but if it is cold, the query is executed once again, including the ordering.
Which is why timing so often fingers the wrong culprit:
The fact that cache invalidation can show a defect without itself being that defect.
When cache behavior is involved, you need to test all the different states: cold, warm, revalidated, invalidated after a write, concurrent fill, and partial failure. Compare the actual result in each case, ignoring things that are allowed to change, like age, trace IDs, and timestamps. If a cached path gives a different meaning, it should be because of a clear product decision, not an accidental change.
Running in production doesn’t make it a health check
The Google SRE book regards symptom-based, black-box monitoring as applicable only to those signals which result in a human being alerted, since such alerts must be actionable and have to be kept to a minimum. AWS, on the other hand, includes a complementary measure: if a health check is too detailed it leads to a number of false positives that correlate with each other and causes healthy servers to be identified.
A script that checks for a specific order or a current data value isn’t automatically a health check just because it runs in production. It’s probably one of the other types, and each has its own rules.
| Check | Question it answers | When it fails |
|---|---|---|
| Journey synthetic | Is it possible for a user to complete something that matters? | The page where real impact is experienced |
| Contract investigation | Does the dependency still return the agreed-upon shape? | Inform the person responsible for the integration; it’s not always an outage |
| Perform a regression test | Did the code breach a controlled invariant? | Stop the change |
| Diagnostic | Did an internal signal move? | Record it; don’t report it as impact |
Neither side of that argument works. Loading the page yourself doesn’t necessarily mean there’s no red probe; it might just be a different region, data set, or cache situation. But a red probe also doesn’t prove anyone was affected.
Working the disagreement
- Freeze the evidence. Expected and actual, repeat runs of the same build, cache state, fixture version, locale and clock, live dependencies versus doubles.
- Write the contract as one sentence. “Ordered by relevance, then by a stable identifier for ties.” No agreement on that sentence? The missing spec is your bug, and neither the code nor the test gets the deciding vote.
- Name the disagreement. Production defect, test defect, missing contract, external drift, broken environment, or a mislabeled check.
- Change one variable at a time. Same build twice. Warm against cold. Fixture against live. Duplicate sort keys. You’re hunting the dimension that flips the verdict.
- Fix the narrowest cause you can defend. This includes a real race condition, even if today’s user impact seems small.
- Get the trust back. Quarantining limits the damage while you fix things, but it doesn’t solve the problem. Rerunning until the build passes just hides the signal the test suite is supposed to give.
The judgment is the job
Set or sequence. Exact or approximate. Warm or cold. Page or log. Each of these choices is someone deciding what needs to stay true, at what layer, under what conditions, and how precisely. None of this is just about typing. Once you’ve settled the argument, writing the assertion only takes a few seconds.
So, the next red build isn’t just a bug report. It’s really a question: was this ever a promise? Figuring that out takes more time than just relaxing the test, but that answer is what really matters in the long run.
