September 30, 2026
4 min read

Five Models Walk Into an App Builder. Security Gets Interesting.

Eran Machlev
Security Researcher
Interactive Testing: Lead a Team of AI Hackers with Tenzai

‍TL;DR: We built the same three apps with five models on Base44 and ran Tenzai's AI hacker against all 15. Tenzai uncovered 254 vulnerabilities across the 15 applications. While every model introduced flaws, their severity and type varied significantly. Base1 delivered the strongest results - it was the only model with zero high or critical findings and no access-control vulnerabilities, performing on par with or better than the leading frontier models tested.

‍

Earlier this year, we tested popular AI coding agents, including Cursor, Claude Code, Codex, Replit, and Devin, and found exploitable vulnerabilities in code generated by every single one.

This time, our focus shifted. AI app builders place a harness around the underlying LLM to translate prompts into functioning software. However, different models can run within that same framework.

While output naturally varies by model, we wanted to answer a practical question:

When keeping the surrounding platform constant, how much does the choice of model dictate the security of the end product?

Base44 provided an ideal environment to find out.

What we tested

We defined three distinct application specifications on Base44. Each specification was generated using five different backends:

  • Base1
  • Opus 5
  • DeepSeek v4 Flash
  • GLM 5.3 Flash
  • GPT-5.6 Sol

This yielded 15 total builds. We then tasked Tenzai's autonomous security agent with probing each application and comparing the findings.

The experiment design was straightforward: identical builder, identical prompts, different underlying model.

Substantial variation in security outcomes

Across the test suite, Tenzai identified 254 total vulnerabilities. Although no model produced entirely flaw-free code, their overall risk profiles diverged quite significantly.

The most prominent contrast lies in issue severity: Base1 stood out as the sole model with zero high or critical findings across all three builds. By contrast, Opus 5 and GPT-5.6 Sol each introduced multiple high or critical issues, while DeepSeek v4 Flash and GLM 5.3 Flash each accounted for three.
‍


This demonstrates that model selection is more than an implementation detail - it directly governs the level of risk carried over into the final application.

Divergence in access controls

Categorizing issues by vulnerability class revealed even sharper distinctions around access control. Except for Base1, every model generated at least one critical or high flaw allowing unauthenticated access to sensitive data.

Base1 avoided high/critical access-control vulnerabilities entirely.

Except for Base1, every model generated at least one critical or high flaw allowing unauthenticated access to sensitive data.

These operational differences extend beyond standard LLM benchmarks, translating directly into real-world attack vectors.

Different models, different failure modes

Injection flaws highlighted another divide. Opus 5 and GPT-5.6 Sol registered zero high/critical injection findings, whereas Base1 yielded a few low-severity issues. On the other hand, DeepSeek v4 Flash and GLM 5.3 Flash both introduced high-severity NoSQL injection vulnerabilities.

The key takeaway is not just that code formatting differed, but that model choice resulted in fundamentally different runtime attack surfaces.

The Base1 result stands out

Our goal was to measure how much of an LLM's inherent security behavior permeates through the developer platform into deployed software. 

Base1 proved particularly noteworthy. It rivaled or surpassed top-tier frontier models across all tested metrics, reducing both the frequency and business impact of identified flaws. That said, even Base1 was not entirely immune to vulnerabilities - a reality shared across every model tested.

The model matters - but the output matters more

Holding platform parameters and application requirements constant while swapping models yielded software with distinctly different risk profiles. Consequently, model selection is a critical security decision that shapes the vulnerabilities exposure of the final product.

However, choosing a stronger model merely establishes a cleaner baseline; it does not guarantee a secure application. In practice, threat actors target deployed application logic rather than the underlying generative model.

Selecting an appropriate model is essential, but rigorous security testing of the generated application remains indispensable.

‍

‍

Talk to us. We're listening.

See Tenzai run against an app you own.

Get a Demo

Media inquiries and interviews

[email protected]

Hiring elite hackers and more

Open roles
Get a demo

See Tenzai in action! Leave us your details and we'll get back to you.

Modern Bank web application security dashboard showing 10 findings with details on OWASP categories, days open, and status. Listed findings include issues like Stored XSS, injection, insecure design, and data integrity failures with endpoints and last seen dates. Status mainly open. Sidebar displays session management options, test result analysis, and suggested follow-ups.