AI-generated code monitoring — why a one-off scan isn't enough
Code that nobody held in their head
An application assembled with heavy AI assistance has a property a hand-written project doesn’t share to the same degree: nobody — not the person, not the model — holds every decision that went into it in their head at once. The model suggests a dependency at whatever version it learned during training, not the latest one. It generates a config file from a common pattern that never gets read line by line, because it “works.” It creates a test endpoint that was supposed to disappear before deployment and didn’t. None of this is individual carelessness — it’s a consequence of code getting generated faster than anyone can actually read and judge it.
Why this behaves differently from a hand-written project
A few mechanisms repeat across AI-assisted development:
- Dependencies pinned to whatever the model suggested — library versions reflect the model’s training data, not the current state of security patches. Without an explicit check, those versions sit in the project for months.
- Generated config copied without review — CORS rules, security headers, sample
.envfiles. They work, so nobody looks again. - Preview and staging deploys left running — faster iteration means more deployments, more subdomains, more things forgotten a few weeks later.
- Secrets and debug endpoints that survive into production — a generated debug panel or a test key that was only meant to be temporary.
- Changes ship faster than anyone reviews them — the review cycle that used to slow down every change shortens, or disappears, with AI assistance.
None of this is an exotic bug class. It’s a consequence of speed — and speed is exactly why people reach for AI assistance in the first place.
Why a one-off scan isn’t enough
A one-off security scan is a statement about the state of the code at a single moment. For a project that changes week to week — often with more deployments in a day than an older team shipped in a month — that statement stays true for a short time. The question worth answering isn’t “is it safe now?” It’s “once it stops being safe, how long until I know?”
A one-off scan can’t answer that — it has nothing to compare against. The fix isn’t “scan more thoroughly,” it’s scanning repeatedly and comparing the result against yesterday’s.
What daily watching changes
On the first scan, Watch stores the domain’s state as a baseline — which headers the server sends, whether HTTPS runs, which files are reachable, what the DNS and email authentication records look like. It then re-runs the same checks every day and compares the result against the previous one. When something changes in a way worth noticing, an alert goes out immediately — not at the next manual scan somebody might never run. Once a month, a summary digest arrives regardless of whether anything fired that month.
The difference against a one-off scan isn’t that Watch sees more things. It’s that it sees the same things repeatedly and can say when they changed.
What this actually catches
A few realistic examples of what the daily comparison can flag:
- A newly exposed file. A generated
.env, a forgotten config backup, or an open directory listing that appears after a deploy and wasn’t there yesterday. - A leaked AI-provider API key in client-side code. An OpenAI, Anthropic, or Google API key pasted directly into the frontend — a common artifact of fast prototyping, where the key gets set on the client “temporarily” and stays there.
- A certificate that starts expiring, or already has. Common on preview and staging domains nobody touches once the iteration ends.
- Weakened email authentication. A DNS change (even an incidental one, while touching unrelated infrastructure) can disable or weaken SPF/DMARC — Watch catches this as a regression against the previous state.
- An injected malicious script. When foreign code lands on the site, the next daily check sees it as a new finding, not a month later.
- A jump in overall finding severity, even when it can’t be pinned to one specific check.
What Watch doesn’t do
Watch is passive, external, black-box observation — it reads only what the server itself publishes to an ordinary visitor: HTTP headers, the TLS handshake, publicly reachable files, DNS and email records. It doesn’t log in, it doesn’t walk the source code, it’s not static analysis (SAST) or code review. It doesn’t see inside the application — it can’t check the logic the model wrote, or dependencies that aren’t visible from the outside.
That has a concrete consequence: today’s daily check set doesn’t cover everything that hurts most in AI-assisted development. Outdated JavaScript libraries on the frontend, a login form submitted over unencrypted HTTP, or a dangling CNAME left behind by a decommissioned preview deploy (subdomain takeover) are exactly the kind of risk continuous monitoring is meant to watch for — and right now they aren’t part of Watch’s daily checks. We’re extending the check set over time; until then, we’d rather say so plainly than let it look like Watch watches more than it actually does.
The short version
The value of continuous monitoring for AI-generated code isn’t that it replaces code review or covers every risk class. It’s that it turns the open-ended anxiety of “I don’t know what’s in there” into a bounded question: how long until I find out that something Watch can see just got worse.