Where Static Analysis Stops: What Automated Scanning Can and Cannot Prove

On This Page

1. What a Scanner Is Actually Doing
2. Limit 1: It Matches Shapes, Not Consequences
3. Limit 2: It Cannot See a Guard It Doesn't Model
4. Limit 3: It Cannot Tell You Whether a Finding Is Reachable
5. Limit 4: It Cannot Supply Context
6. Limit 5: It Cannot Find an Absence
7. The False Negative Problem
8. The Manual Toolchain, and What It Is For
9. Reading a Scanner Report Critically
10. Layering the Two Properly
11. Frequently Asked Questions
12. Conclusion

1What a Scanner Is Actually Doing

A static analysis tool parses your code — or decompiles your binary — into a structure it can query, and then asks that structure a fixed set of questions. Does this string match a pattern that looks like a credential? Does this manifest declare a permission on a known list? Is this dependency version named in an advisory database? Does this call pass user-controlled data into a function on a dangerous-sinks list?

Within that set of questions, a scanner is better than a person: it is fast, exhaustive, perfectly consistent, and it never gets bored on file four hundred. That is a genuine and underrated advantage, and it is why automation should run on every commit rather than once a quarter.

The limits are not defects to be patched out in the next release. They are structural — consequences of the fact that the tool is answering questions somebody wrote down in advance. Understanding them is what lets you read a report correctly instead of reading it as a verdict.

2Limit 1: It Matches Shapes, Not Consequences

A tool can reliably tell you that a string literal appears in a comparison. What it cannot tell you is what that means for your remediation plan. Consider two lines that a scanner treats almost identically:

# (a) A credential read from configuration, with a committed default.
API_TOKEN = os.getenv("VENDOR_TOKEN", "tok_live_9f3c")

# (b) An endpoint comparing a header against a copy of the key.
if request.headers.get("X-Api-Key") != "tok_live_9f3c":
    return unauthorized()

Both are “a hardcoded secret,” and a good tool flags both. But (a) is fixed by setting an environment variable, while (b) survives rotation entirely — that endpoint will keep accepting the old credential after you have rotated at the provider, silently, with no error and no log entry. The remediation for (a) is configuration. The remediation for (b) is a code change, and if you miss it, your rotation did not close the exposure.

Nothing in the scanner’s output distinguishes them, because the distinction is not about the shape of the code. It is about what the code does and what your fix will and will not reach.

3Limit 2: It Cannot See a Guard It Doesn’t Model

Authentication detection is the clearest case. A tool that identifies protected routes has to encode what “protected” looks like, and frameworks offer many spellings of the same idea:

# 1. A decorator -- the form most tools model.
@requires_auth
def handler(request): ...

# 2. A dependency declared in the signature.
def handler(request, user = Depends(current_user)): ...

# 3. A shared-key header checked inline.
def handler(request):
    if request.headers.get("X-Internal-Key") != expected:
        return unauthorized()

# 4. A helper function called for its side effect.
def handler(request):
    _check_webhook_auth(request)   # raises on failure

A rule written around form 1 reports forms 3 and 4 as unauthenticated. That is a false positive, and it is the benign direction — a reviewer opens the handler, sees the guard, and moves on. The same limitation runs the other way too: a tool that treats the presence of any auth-looking call as proof of protection will accept a guard that returns early on one branch, or one that is checked after the sensitive work has already happened.

4Limit 3: It Cannot Tell You Whether a Finding Is Reachable

A dependency scanner matches your lockfile against an advisory database. It is accurate about exactly that. It does not know whether the vulnerable package is present in the artifact you deploy, whether your code calls the specific feature the advisory describes, or whether anything an attacker controls can reach that feature.

The result is a list where a critical-severity advisory in a package your production image does not contain sits above a moderate one on your unauthenticated sign-in path. Sorting by severity gets that ordering exactly backwards, and no amount of tuning fixes it, because the missing information is about your build rather than about the package.

This is a judgement task with a repeatable method rather than an unanswerable question — it just is not one the scanner can perform. It is covered in full in Is That CVE Reachable?

5Limit 4: It Cannot Supply Context

A great deal of what determines severity is knowledge that exists outside the repository. A tool cannot know:

• Whether a value is real. A credential-shaped string in a test fixture and the same shape in a deployment script are the same to a pattern and not remotely the same to you.
• Which data actually flows through a path. A function that processes conversation transcripts is handling training material or regulated personal data depending entirely on which identifier is passed to it — and nothing in the code distinguishes the two.
• Who can reach a system. A finding’s severity depends on whether the repository has three collaborators or three hundred, and on whether an internal service is reachable from the public internet. That is organizational fact, not code.
• What the organization has promised. A rule in a policy document, a contractual commitment, or a regulatory obligation determines whether a behaviour is acceptable. A scanner has never read them.

This is why the most useful findings in an assessment often come from reading the client’s own documentation alongside the code. The gap between what an organization has written down and what its code enforces is frequently the finding — and it is invisible to any tool that only reads one of the two.

6Limit 5: It Cannot Find an Absence

This is the most important limit and the least discussed. Scanners find things that are present and match a pattern. A great many serious findings are not things that are present — they are things that are missing, and no pattern matches an absence.

The last row generalises: a whole class of real findings is about inconsistency rather than any single location. A control correctly applied in thirty places and skipped in three is the normal shape of a security problem in a mature codebase, and it is only visible to something that can hold all thirty-three in view at once and notice which are different.

7The False Negative Problem

Tool evaluation tends to focus on false positives, because they are visible and annoying. The asymmetry runs the other way:

• A false positive costs a reviewer five minutes. They open the file, see the guard, dismiss the finding.
• A false negative tells you the repository is nearly clean and stops anyone looking. It costs nothing visible, forever.

False negatives are also structurally harder to discover, because nothing in the output hints at them. A report saying “3 secrets found” looks exactly the same whether it is correct or whether it missed seventeen.

What exposes them, in practice, is asymmetry — two things that should behave the same being treated differently:

# Line 433 -- a credential, NOT reported by the scanner.
client_secret = os.getenv("RC_CLIENT_SECRET", "<value>")

# Line 434 -- a credential, reported correctly.
jwt_token = "<value>"

# Two secrets, adjacent lines, one found. That asymmetry is not a
# property of the code -- it is a property of the detection rule,
# which anchored on `name = "value"` adjacency and could not see
# the name sitting inside the first argument of a two-arg call.

The generalisable habit: when a number seems low for the codebase in front of you — three secrets in an application that integrates a dozen third-party services — treat the number as a hypothesis and check a sample by hand. A count is only as complete as the rules that produced it, and nothing in the output tells you what those rules could not see.

8The Manual Toolchain, and What It Is For

For compiled mobile applications, the standard tools are well established. What matters is understanding what each one gives you that an automated report does not.

Automation covers the deterministic subset of this well: parsing the manifest, extracting strings, checking whether the binary is signed and whether hardening flags are set. Those are exactly the checks that should be automated, because they are mechanical and you want them on every build.

What manual work adds is the questions that require following logic across the application: what this obfuscated routine actually computes, whether a certificate-pinning implementation can be bypassed by the path the developer did not consider, whether a client-side check is the only thing standing between a user and a paid feature, and whether a credential recovered from the binary is live or a leftover from a staging environment. Each of those is a chain of reasoning, and chains of reasoning are what automation does not do.

9Reading a Scanner Report Critically

A short list of signals worth treating as prompts to look harder:

• A secret count that seems low for the integration count. If the application talks to a dozen services and the report names three credentials, check a sample by hand before accepting it.
• Findings that did not change after a significant refactor. Either the refactor genuinely did not touch security-relevant code, or the rules are anchored on something that survived the change without meaning what it used to.
• Everything in one category. A report that is entirely dependency findings usually means the other analyses found nothing to say, which is different from there being nothing to find.
• No findings about absent controls. If nothing in the report concerns something that should exist and does not, that is expected from a scanner and a gap in your assessment.
• Severity assigned without reachability. A list sorted purely by CVSS has not been triaged, whatever the summary says.

None of these mean the tool is bad. They mean a scan is an input to an assessment rather than the assessment itself.

10Layering the Two Properly

The two approaches fail in different directions, which is what makes them worth combining rather than choosing between.

A working arrangement: automation runs continuously and owns coverage and regression — it is what stops a known class of problem from reappearing after it has been fixed. Manual review happens at decision points — before a launch, after a significant architectural change, when entering a regulated market, or when someone needs a defensible answer rather than a list. The manual pass should start by reading the automated output, including asking what the rules that produced it could not have seen.

11Frequently Asked Questions

12Conclusion

Static analysis answers, quickly and exhaustively, the questions somebody wrote down in advance. That is genuinely valuable and it should run on every commit. What it cannot do is tell you what a finding means for your remediation plan, recognise a guard nobody modelled, establish whether a vulnerability is reachable in your build, supply context that lives outside the repository, or notice that something which should exist does not.

The last of those is the one worth remembering. A large share of the findings that change how an organization operates are absences — no second factor, no revocation, no retention period, a policy written and never enforced — and no pattern matches an absence. Finding them requires arriving with a model of what should be there and naming the difference.

A scan is an input to an assessment, not a substitute for one. See APK Static Analysis, Explained and IPA Security Testing, Explained for what the automated layer covers on mobile, and Is That CVE Reachable? for the triage method behind limit three.