Guide · Procurement
What an external scan can and cannot evidence for a security questionnaire
A customer sends you a security questionnaire. Some of it you can answer with evidence, some of it only with your word, and some of it needs an auditor. Knowing which is which before you start is most of the work.
This is written from the outside in: what is observable about a running application by someone with no access to it. That turns out to be a small share of a standard questionnaire — and being clear about the size of that share is the point of the page.
Three kinds of answer
Every question on a questionnaire falls into one of three buckets, and they are not interchangeable.
Verifiable
Someone outside your company can check it themselves, today, without your cooperation. TLS configuration, security headers, whether a debug endpoint answers. You do not need to be believed.
Attestable
True, and not externally observable. Whether you require MFA, how you offboard staff, whether backups are tested. You are asserting it, and the buyer is deciding whether to accept your assertion.
Audited
A third party has examined it and put their name to the result. SOC 2, ISO 27001 certification, a penetration test report. Expensive, slow, and the only thing some buyers accept.
The mistake worth avoiding is answering an attestable question as though it were verifiable — or letting a buyer believe a scan report is an audit. Both get discovered, and they get discovered at the worst moment.
What is genuinely observable from outside
Concretely, without any access to your systems, a scan of a running application can establish:
| Transport security | Certificate validity and expiry, whether plain HTTP redirects to HTTPS, whether HSTS is set, whether the certificate covers both apex and www. |
|---|---|
| Browser-facing controls | Content-Security-Policy, X-Content-Type-Options, frame protection, referrer policy, and the Secure / HttpOnly / SameSite flags on the cookies you set. |
| Accidental exposure | Whether files that should never be served are being served — a .git directory, a .env, source maps, a directory listing, a framework debug page. |
| Email anti-spoofing | SPF, DKIM and DMARC records, and whether the DMARC policy is set to monitor or to enforce. |
| Published disclosure route | Whether a security.txt exists, which is how a researcher who finds something knows where to send it. |
| Domain and certificate history | Registration age via RDAP, and the hostnames that have appeared in public certificate-transparency logs. |
If your code is available to check as well — because you provide it, not because it was found — two more become verifiable: whether a credential is committed anywhere in the repository or its history, and whether your dependencies carry published vulnerabilities with fixes available.
What is not observable, and never will be
This is the longer list, and pretending otherwise is how a supplier loses a deal at the diligence stage rather than the questionnaire stage.
Nothing about your organisation. Background checks, security training, who has access to production, how quickly you revoke it when someone leaves, whether there is a named person responsible for security. An external scan cannot see a single one of these.
Nothing behind the login. Whether an endpoint enforces authorisation correctly, whether one customer can reach another's data, whether your admin panel checks roles. Establishing that means calling your endpoints as different users and interpreting the responses — which is a penetration test, done with permission, by someone who is not us.
Nothing about data at rest. Whether the database is encrypted, where backups live, how long you keep logs, whether a deleted record is actually deleted.
Nothing about process. Incident response, change management, vendor review, business continuity. These are documents and habits, and the only external evidence for them is an audit.
On a standard organisation-wide questionnaire, the externally-observable share is small — a minority of questions, concentrated in the application-security and encryption-in-transit sections. On a short technical form about a specific web application, the share is much larger. Which form you have been sent matters more than which tool you use.
A way to work through one
- Sort before you write. Go through the whole form once and mark every question verifiable, attestable or audited. Most of the panic comes from treating a 200-question form as one task rather than three.
- Answer the verifiable ones with evidence, and say where the evidence came from and when. A dated result someone can re-run beats a "Yes" every time.
- Answer the attestable ones plainly, and date them. "Yes — MFA is required for all administrative accounts, as of March" is a stronger answer than "Yes", because it tells the reader you know what you are asserting.
- Say so when the answer is no. A supplier who writes "we do not have a SOC 2; here is what we do instead" is more credible than one whose form is entirely green. Buyers read a lot of these.
- Do not let a scan be described as an audit. If your answer references a scan report, say it is an automated external assessment. The buyer's security team will work it out, and it is better that they learn it from you.
Where this product fits
Preflight produces a dated, independently verifiable record of the externally-observable items above — the first bucket, and only that bucket. Each finding states what was observed, how, what it does not prove, and the change that fixes it. The record is hashed and timestamped by an independent authority, so a buyer can check it against them rather than against you.
It is not an audit, it is not a penetration test, and it will not fill in a questionnaire for you. It answers a minority of one honestly and says so — which is the only way a partial answer is worth anything.
This page describes what is and is not observable from outside a running application. It is not legal or compliance advice, it does not interpret any particular regulation or framework, and the requirements of a specific questionnaire are set by whoever sent it to you.