A completed vendor security questionnaire only proves one thing: on a specific date, someone at the vendor answered your questions. Anything beyond that, like whether the control exists in the environment you are buying, whether it applies to the service instead of just the company brochure, or whether it was true last quarter and is still true now, all require separate evidence.
I don’t have a problem with questionnaires. For most supplier relationships, they are the right first step and often the only one that fits the budget. The problem comes after the questionnaire is completed. People treat it as a final decision, file it under “vendor approved,” and forget to note which questions still need answers. I’ve made a similar point about passing tests: a green check only shows that an assertion ran, not that the change is safe to ship. A returned questionnaire is the procurement version of a green check.
The question it answers
NIST is rather direct about the position of the form. According to SP 800-161r1, due diligence questionnaires are included in the market analysis section, alongside requests for information, “for the initial screening and collection of evidence from potential suppliers.” The following sentence is more important: “Enterprises should not treat the initial C-SCRM due diligence risk assessment as exhaustive.” NIST also states that the results of this research “can also be helpful in shaping the sourcing approach and in refining the requirements.” This is a screening stage which influences what you ask for afterwards, not a final conclusion.
The Cloud Security Alliance is careful in the same way. CAIQ v4 is a set of yes/no questions “a cloud consumer and cloud auditor may wish to ask of a cloud provider to ascertain their compliance to the Cloud Controls Matrix,” so that buyers can “gauge the security posture of prospective cloud service providers.” The operative verb is gauge, and CSA files the questionnaire at STAR Level 1, its self-assessment tier, which tells you what its own authors think the artifact weighs.
So the question a questionnaire answers is more specific and useful than just asking, “is this vendor secure?” It really asks: what does this vendor say about these controls, for this service, on this date, and what should we check next?
A yes is not one kind of fact
Two vendors might both answer yes to “do you enforce multi-factor authentication?” but mean completely different things. For one, it could just mean they have a policy document. For another, it could mean multi-factor authentication is actually enforced on the admin console of the tenant you are about to buy, with break-glass accounts named and monitored. The gap between these answers is not dishonesty. The question simply did not ask for that detail.
Scope is what matters most here. You are not assessing “the vendor” as a whole. You are looking at a specific product, in certain regions and environments, with certain subprocessors, handling certain types of data, and with a clear line between what they manage and what you configure. A certificate in the sales deck may not cover the deployment on your order form. Diligent’s guidance on these questionnaires makes the other limitation clear: a questionnaire “only gives a perspective on third-party risk at a point in time,” and it “is completed by the vendor themselves and relies on their ability to assess risk and report candidly on it.” Since Diligent sells continuous monitoring, keep that in mind. The limitation is about structure, not about honesty.
| The questionnaire tells you | You still have no evidence about |
|---|---|
| MFA is required by policy | Whether it’s enforced on the tenant and roles you’re buying |
| An incident response plan exists | Whether it has been exercised, or when you would be told |
| Subprocessors are “managed” | Which ones touch your data, in which regions, under what terms |
| Backups run and recovery objectives are set | Whether a restore has been tested against those objectives |
Saying it, showing it, testing it
NIST SP 800-53A separates three ways of obtaining evidence, and that separation is this whole argument in miniature. To examine is “checking, inspecting, reviewing, observing, studying, or analyzing assessment objects.” To interview is “conducting discussions with individuals or groups within an organization to facilitate understanding, achieve clarification, or lead to the location of evidence.” To test is “exercising one or more assessment objects under specified conditions to compare actual with expected behavior.”
A questionnaire is a structured self-report that directs you to things you should examine and people you should interview. It is not the examination itself, and it is definitely not the test. Filling out the form does not exercise anything under specified conditions, which is why a full spreadsheet of green cells can sit alongside an unpatched service and contradict nothing.
The next artifact up the stack is different in type, not just in quality. The AICPA describes a SOC 2 report as an examination of controls at a service organization, created because customers and business partners “usually need information about the design, operation, and effectiveness of controls within the service organization’s system.” This report is much stronger evidence for the system, criteria, and time period it covers, but it does not say anything about areas outside that scope. That is why you should read the scope and period pages first.
Making the answers do something
NIST’s own guidance is about being proportional, not doing the most possible: “There are a variety of acceptable validation and revalidation methods, such as requisite certifications, site visits, third-party assessments, or self-attestation,” and “the type and rigor of the required methods should be commensurate with the criticality of the service or product being acquired.” This means you should decide on the tier before sending the questionnaire, not after. Figure out what the vendor can access, data, privileges, operational dependency, or what would break if they are unavailable, and let that determine how much evidence you need.
One of NIST’s three opening examples of supply chain risk is “a store chain that experiences a massive data breach tied to an HVAC vendor with access to the store chain’s data-sharing portal.” The interesting variable in that sentence is access, not the vendor’s control maturity. A supplier with a modest security program and no path to your data is a smaller problem than a well-run one holding an admin credential.
Make sure each answer leads to something concrete. A yes should include the artifact, the time period, the owner, and any exceptions, so you end up with a list of follow-ups instead of just a feeling. Put the claims you depend on into the contract, just as NIST recommends: “the periodic revalidation of supplier adherence to security requirements,” and clear processes for “the reporting of information about vulnerabilities, incidents, and other business disruptions.” Set the trigger for reassessment based on changes, not just on a calendar. NIST says you should review after product updates, supplier operational or structural changes, internal system changes, or geopolitical or environmental events. These changes do not wait for December just because that is when you planned your review.
None of this affects your own responsibilities. Even if a vendor answers every question honestly, it does not cover how you set up identity, access, and logging for the integration. In 2022, CISA, NSA, and ODNI published customer guidance for procurement and deployment for this exact reason: a lot of the risk comes from areas the supplier cannot describe on your form.
The objections that hold up
Questionnaires are the only practical way to do an initial review at scale, and that is a valid point. EY’s 2023 third-party risk survey of over 500 institutions found that 54% sent one combined questionnaire and 46% sent multiple questionnaires from different risk areas, with 90% moving toward centralized risk management. This shows what organizations are doing, not how accurate it is, no one in the survey showed that a questionnaire actually caught any issues. Centralizing the process makes assessments faster and easier to compare, but it does not turn a self-report into an independent review.
Asking every supplier for an audit is expensive and often not worth it for smaller vendors, which is a good reason to use tiering instead of applying the same strictness to everyone. External ratings do not solve the problem either. They are timely and come from outside, but they cannot see your contract, your tenant setup, or the responsibilities you and the vendor each assumed the other would handle.
A vendor risk program does not improve just by collecting more questionnaires. It improves when every answer leads to a real decision and a follow-up that someone is responsible for. If a returned questionnaire does not change what you would have done, you did not run an assessment. You just collected a document.
