Are Vibe-Coded Repos Less Secure?
As part of the larger Fictivize project, we have been working extensively with vibe coders. One thing we have found is that many of them don’t have a good plan for maintaining and measuring the security of their SaaS sites. This leads to the obvious question: is it actually possible to create secure code with AI tools? Or are AI coded sites just inherently insecure?
To find out, we used the WhiteRabbit security scanning tool that we provide to our students (it’s also open source). We then ran that tool against 510 GitHub repos scraped from Reddit.
We broke those repos down into three categories. “Indie” sites are public sites on GitHub that don’t have multiple maintainers, have less than 100 stars and less than ten forks. In short, these are just regular repos maintained by a single person. “Vibe-Coded” sites are those with AI coding signals. This includes things like a Claude.md file, references to an AI tool in the comments, etc. The third bucket was the control, these are “Established OSS” repos. These are sites with hundreds or thousands of stars, thousands of downloads, multiple maintainers, etc..
When you look at these repos, you find that the larger repos have more security issues. And, since AI repos tend to be larger, they tend to have more issues. Of course, the OSS repos are larger still and thus have even more issues. However, the normal metric for code security quality is “weighted issues per 1k lines of code.” What “weighted” means is that critical vulnerabilities are weighted higher than low vulnerabilities. These total vulnerabilities are then assigned, scored, totalled up and then divided by 1,000 lines of code. Thus, the numbers above show a relative score that allows you to compare a large repo to a smaller one. A repo with three “low” issues will score better than one with two “high” issues.
Looking at the data this way, you see that AI generated code is actually MORE SECURE than hand independent repos but less secure than peer reviewed OSS repos. Which is reassuring, if OSS repos were massively less secure than AI, we would all have a problem. But, they’re not.
What’s even more interesting is to look for signals that tell us if an AI repo will be more or less secure. And we found one: CI/CD.
Put simply, if an AI coded repo had any sort of CI/CD, it would be roughly half as insecure as if you don’t use any at all. It’s also interesting to note that AI generated is the most secure and the most insecure code by this measure, simply by controlling for the presence of CI/CD or not. The difference between the indie repos with or without CI/CD is small enough that it could just be a fluke with the sample size, but the difference between the two types of AI repos is very stark and unlikely to be a fluke. That’s a sample of almost 200 repos, which is a large enough sample to convince us.
We also looked at the type of agent used:
Cline, Windsurf and Aider all have less than ten results each so you probably should not draw conclusions from that. Sticking to the statistically relevant samples, ChatGPT is safer than Cursur, but only by 27%. While this is a notable number, it’s not close to the gap that CI/CD creates. It should also be noted that Cursor supports multiple models as does Copilot. The two main competitors in this space, Anthropic and OpenAI have essentially identical scores. Net-net, the tool usage doesn’t seem to be the deciding factor.
Language, on the other hand did have a significant impact:
Security by Language
| Language | Vibe n | Vibe w/K | Indie n | Indie w/K | Est. n | Est. w/K | Δ |
|---|---|---|---|---|---|---|---|
| JavaScript | 144 | 0.813 | 85 | 1.511 | 40 | 0.431 | -0.70 |
| TypeScript | 98 | 1.151 | 68 | 2.021 | 32 | 0.434 | -0.87 |
| Python | 115 | 0.615 | 67 | 0.876 | 37 | 0.468 | -0.26 |
| CSS | 115 | 0.884 | 68 | 1.510 | 32 | 0.450 | -0.63 |
| Shell | 114 | 0.639 | 52 | 0.965 | 34 | 0.411 | -0.33 |
| Rust | 35 | 0.501 | 15 | 0.491 | 18 | 0.329 | +0.01 |
| Go | 13 | 0.690 | 8 | 2.114 | 10 | 0.447 | -1.42 |
| Swift | 26 | 0.460 | 10 | 1.752 | 11 | 0.375 | -1.29 |
| C | 14 | 0.348 | 14 | 0.255 | 13 | 0.313 | +0.09 |
| Kotlin | 8 | 0.438 | 12 | 1.848 | 8 | 0.474 | -1.41 |
| C# | 14 | 0.804 | 5 | 0.437 | 6 | 0.353 | +0.37 |
| SQL | 30 | 0.878 | 12 | 1.962 | 10 | 0.502 | -1.08 |
Typescript for example is much more unsafe than other languages such as JavaScript (which is ironic since TypeScript was created to give a safer alternative to JavaScript). However, the difference between indie and AI doesn’t really change based on language. If you ignore the languages with N under 50, you see that in all cases AI is more secure. This means that this isn’t a language artifact, but actually tied to the AI tooling and methodology.
The bottom line: True “vibe coding” where you simply tell AI to build without any structure or process is not secure. However, AI code factories with simple CI/CD are more secure than coding by hand.