All that a check shows is that an assertion has been executed and that the value it saw matched the value someone expected when they wrote it; it does not tell us whether the change is safe to merge, whether the release is ready to be deployed, or whether the requirement has actually been met. Each of these is a separate question that requires its own kind of evidence, yet most test suites fail to make it clear which of these questions they are addressing. In many cases, the check is merely the last step in a decision that was taken earlier, often in an informal way.
I call this missing piece a “verification contract.” This isn’t an official industry term. It’s just what I use for something that other engineering fields already document: a brief agreement connecting a requirement to the decision that relies on it. It covers what must remain true, under which conditions, what evidence is acceptable, what counts as enough, who interprets the result, and what the result doesn’t prove.
NASA treats that agreement as a requirement of its own. Its software engineering handbook says the project manager “shall plan software verification activities, methods, environments, and criteria for the project”, and that satisfaction criteria “may be given in numerical form (a specific value, a minimum value, a range of values)” or “in a pass/fail or true/false format.” The ordering matters more than the formality. Criteria come first, so the check implements a decision that has already been made about what correct means.
What the contract contains
| Part | The question it settles |
|---|---|
| Claim | What behavior or property has to stay true? |
| Context | Which version, actor, inputs, dependencies, and environment does it include? |
| Method | Test, analysis, inspection, demonstration, review, or exploration? |
| Threshold | What constitutes a sufficient observation? |
| Decision | What does a pass permit — merge, release, rollback, or acceptance? |
| Owner, limits | Who interprets the result, and what does it leave unproven? |
This table is my own summary, not an official standard. A typical test has an input and an expected output. A contract goes further by adding the reason for the check and the decision that depends on it. This helps prevent a passing test from being seen as stronger evidence than it really is.
Testing is one method, not the method
NASA’s product verification process names four methods: analysis, demonstration, inspection, and test. Analysis is “the use of mathematical modeling and analytical techniques to predict the suitability of a design.” Demonstration shows that using the product achieves the requirement, “differentiated from testing by the lack of detailed data gathering.” Inspection is “the visual examination of a realized end product.” Test produces “data at discrete points for each specified requirement under controlled conditions,” and the handbook calls it “the most resource-intensive verification technique.”
Reading a permission policy or a config diff is inspection. A capacity calculation is analysis. Walking through the workflow with the person who asked for it is demonstration. Testing is for when you need controlled execution and numbers you can compare. Teams routinely write an automated test for all four situations, paying the most expensive method on the list to answer a question a ten-minute inspection would close.
Acceptance criteria are a boundary, not proof of value
NASA separates two questions software teams tend to merge. Verification answers “Was the end product realized right?” Validation asks whether the product “accomplishes the intended purpose in the intended environment — that it meets the expectations of the customer and other stakeholders”, or as the handbook puts it, “Are we building the right product for our users and other stakeholders?”
Acceptance criteria belong to verification. They can be clear, observable, and automated, but still not be the right criteria. When “the acceptance tests pass,” it just means the build meets a boundary set earlier. It doesn’t guarantee the feature is valuable. If the tests pass but the people who asked for the feature are still unhappy, that’s a problem with the requirements, not something more tests can fix.
A good criterion should state the important property, the context, the observation, the threshold, and what happens if the result is a pass, a failure, or unclear. The last part is often missed, but it’s what turns a result into a real decision.
The interface is where this gets concrete
Contract tests are the smallest version of the idea. Martin Fowler describes them as a separate set of checks confirming “that all the calls against your test doubles return the same results as a call to the external service would.” The double isn’t the service. The contract is the claim that it still behaves like one.
Consumer-driven contracts go further. Ian Robinson and Fowler wrote in 2006 that consumer contracts are “open and incomplete with respect to the business functionality available to the system”, consumers depend on a subset of what a provider offers, and collecting those expectations gives the provider “the fine-grained insight and rapid feedback we require to plan changes and assess their impact on applications currently in production.” That’s a verification contract at an interface: the consumer defines the evidence before the provider writes the change.
For a single endpoint, this could mean a consumer contract that covers the fields consumers actually read, provider-side checks of the documented responses and error cases, a demonstrated user journey, and a stated limit.
None of this covers how the system behaves under load. Load is a separate issue, with its own method and threshold, based on operational needs, not invented to fill a row.
Coverage makes a poor headline
Fowler’s note on test coverage is clear: it’s “a useful tool for finding untested parts of a codebase” and “of little use as a numeric statement of how good your tests are.” If you can hit a number with weak tests, people will do just that. Saying “coverage is ninety percent” isn’t a contract. A real contract is saying, “These behaviors and failure modes have evidence for this release decision, and here’s the uncertainty we’re accepting.”
DORA makes a related argument higher up. In its 2023 report, it treats metrics as “a starting point” and recommends that teams concentrate on learning and adapting rather than simply meeting targets. The same study, which was based on over 36,000 professionals, showed that teams that focus on users see 40% higher organizational performance, and 30% higher organizational performance for teams with a generative culture. That’s survey correlation and it proves nothing about the practice I’m describing; I use it for the narrower point that evidence should connect to user and delivery outcomes, not just to figures that look good on a graph. DORA’s 2019 report says the same thing: continuous delivery “reduces risk and cost”, and teams should move away from “heavyweight external change approval processes” and instead use peer review during development combined with automation. Taking feedback earlier does not mean that judgment can be omitted from release decisions.
Where the contract is weakest
You can’t plan for all the evidence ahead of time, and that’s the main criticism of this approach. Andy Tinkham and Cem Kaner adopt a definition of exploratory testing as “any testing to the extent that the tester actively controls the design of the tests as those tests are performed and uses information gained while testing to design new and better tests.” The learning happens as you go. A contract doesn’t mean you can predict every defect; it’s a commitment to the decisions you know you’ll need to make, while leaving room to discover things nobody thought to specify.
The concern about too much process is valid, and NASA addresses it in its guidance: small projects “may wish to consolidate their verification planning into the Software Development Plan (SDP) or document it as part of a verification or traceability matrix.” For a low-risk change, it gets a row. For something expensive to change later or a boundary your team will manage for years, a more detailed approach is needed. Cut down on ceremony, but not on making good decisions.
Contracts can go out of date; for example, if you discover a defect, alter a dependency, or realize that a requirement was incorrect only after users have tried the system, the evidence plan might then be no longer valid. The same NASA page carries a lesson from the Mars Polar Lander: unit and integration testing should “at a minimum, test against the full operational range of parameters.” When database parameters that affect logic decisions change, “the logic should be re-tested.” Saying you ran tests isn’t enough; showing the plan covered the full range and the path of change is what matters.
None of this will prevent every defect, and I am not claiming that it will. What it does is reveal missing requirements, weak evidence, and hidden uncertainties early enough to address them. So before you write the next check, ask yourself: what decision should this evidence authorize, and what will it still leave me unable to claim?
