Late on a Friday, a developer asks an AI assistant for a login handler. Seconds later it’s sitting there, neat, with comments and everything. It runs. It works. Up goes the pull request. The reviewer has six other pull requests open and, honestly, a lunch to get to. Thumbs-up. What neither of them sees: the SQL query is glued together from strings, the textbook way to get an injection bug (CWE-89, for the collectors). It heads to production on Monday.
Nobody in that story did anything strange. The code compiled, the tests passed, and it was Friday. If your developers use AI coding tools, and most do by now, announced or not, slips like that are worth planning for.
What actually goes wrong
The assistants learned from huge piles of existing code, and existing code is full of bad habits, so the old classics come straight back. Queries built from strings. User input trusted too early. An API key typed straight into a config file. Error messages that spill stack traces to the browser. These are the same mistakes that fill the OWASP Top 10, just produced faster. The confidence is what makes it sneaky. A teammate who isn’t sure will usually say so, whereas an assistant almost never does.
Dependencies are the next trap. Picture the assistant suggesting a library nobody has checked. It could be years out of date, or it could have known vulnerabilities, or the model might have invented the name because it sounded about right. What follows has a name too, slopsquatting. Somebody registers that invented name on npm or PyPI, and then it’s a waiting game until a developer runs the install without looking. Licensing is the quiet one. Generated code can resemble open-source code under terms your company can’t live with, and nobody finds out unless somebody looks.
Then comes plain over-trust. Odd but true: tidy-looking code gets a lighter look than a colleague’s messy draft, when it should be the reverse. Volume makes it worse. Hundreds of lines show up in sixty seconds. A tired reviewer starts skimming. You would too. A simple fix some teams use: limit the size of a generated change, so somebody can read every line.
What a sensible rollout looks like
Now imagine forty developers who’ve just been given an assistant. Week one has one rule. Generated code gets handled like a pull request from a contractor nobody’s met. A person reviews it. Passing tests don’t amount to a review. Whoever approves should be able to explain what the code does and why it’s safe, and it helps when authors mark which bits came from the assistant, since that shows where to slow down.
By week two the platform engineer has wired three checks into the pipeline. Static analysis (SAST) comes first: Semgrep, CodeQL or SonarQube looking for risky patterns. Then a dependency check. Snyk, Dependabot or OWASP Dependency-Check compare your libraries with known vulnerabilities and flag licenses nobody approved. Last, secret scanning, where Gitleaks or TruffleHog spots a password or key that wandered into a commit. A serious finding fails the build. A red pipeline on a Thursday evening is annoying. Finding out in production is worse.
Week three has a surprise. The assistant keeps recommending a package nobody has heard of. Someone searches npm and finds nothing, because it never existed. From then on every new package is looked up by hand and pinned to an exact version in the lockfile before the merge, which is about two minutes of work. Logins and permissions, payments, encryption and anything that takes input from outside also get a senior engineer’s eyes every time, no matter who or what wrote the code.
One more habit: watch the tests. When the same assistant writes them, they tend to prove the code does what the code does. So somebody adds the ugly cases: junk input, missing permissions, a request ten times too big. Anything facing the internet gets a few real security tests as well.
What goes into the assistant matters too
A month in, a developer pastes a stack trace into a personal chatbot account. The trace contains a customer’s email address. Nothing bad happens. The security lead finds out anyway, and two short documents follow. One lists the assistants the company has approved, meaning the ones whose terms spell out what happens to your data. The other lists what never goes into a prompt: secrets, customer data, anything under NDA. Access to the approved tool also gets faster, since people grab a personal account when the official route takes two weeks.
Who wrote this line?
At some point an auditor, or someone running an incident, will ask which code was written by a machine. If commits and pull requests carry a simple AI-assisted label, there’s an answer. Months later you can also see whether labeled changes have more bugs than the rest. Write down which tool and version, if you can. Skip it and you’ll be guessing on the worst possible day. There’s a side benefit as well. When a bug turns up in generated code, people talk about the process instead of hunting for someone to blame.
A last look before release
Before a change with generated code goes out, a short conversation settles it. Has a person who understands the code approved it? Did the scans come back clean? Was any new package verified and pinned? Do the tests cover what happens when things fail? Did the risky areas get senior review, and can the change be traced back later? If someone answers no, wait a day. A day’s delay is a lot cheaper than an incident.
Here’s a cheap experiment worth trying. Take one small generated change from last month, hand it to three reviewers, and don’t tell them it came from an assistant. Compare what they flag. If nobody notices the weak spot, you’ve learned how your reviews really work, for the price of an hour.
Where this leaves you
A fair worry is that all this review eats the time the assistant saved. On the risky parts it sometimes does. Boilerplate is different. Test scaffolding, data mappers, routine endpoints: the time saved mostly survives a light review. So spend your careful reviewing where a mistake would really hurt, and let everything else move.
Please don’t read any of this as “avoid the tools.” The tools do save time on routine work. And the guardrails? Mostly things good teams do already. What moves is the care. It goes into unhurried review, into automated checks for the repetitive stuff, and into firm rules about data. That’s why our AI application services come with human review gates and security checks built in. Comparing platforms? Choosing an AI coding platform for enterprise lists what to ask, and how an AI code debugger finds and fixes issues shows one way to handle the review side.
Modernize with confidence. Build for what's next.
Talk to our team about embedding AI into the core of your business.