Testing and documentation
Coverage and living docs are first-class outputs.
Tests and docs evidence
3 services · 11 files · 0 shared schemas
Checks green. Staging URL live. Spend recorded on the same thread.
The category
Generation without checks is a liability. Arvad attaches suites and docs to the project so the shareable link includes green checks, cited files, and docs that moved with the change.
Suites run on the same project as the build
Share view shows suite status and cited files
Living docs sync when code changes on the project
Failed checks block a shareable green status
The gap
Most AI coding tools optimize for generation, not quality. You get more code, more bugs, more security issues, and docs that never appear. The 2024–2025 data is clear.
10×
Apiiro's September 2025 study of Fortune 50 enterprises found AI-assisted developers generate 10× more security issues than non-AI peers, including a 322% increase in privilege escalation paths and 153% increase in architectural design flaws.
Apiiro, September 202524.8%
An FSE 2024 study found only 24.8% of AI-generated unit tests pass execution. 57.9% fail to compile entirely, and of those that do compile, 85.5% of failures are caused by incorrect assertions, the tests are wrong, not the code.
FSE 2024 / arXiv59%
Clutch's 2025 survey of 800 software professionals found 59% of developers use AI-generated code they do not fully understand. Sonar reports 96% don't fully trust AI output, yet only 48% verify it before committing.
Clutch & Sonar, 20258×
GitClear's 2025 analysis of 211 million changed lines found code duplication increased 8× during 2024. Code refactoring dropped from 25% to under 10% of changes. For the first time, copy/paste exceeded refactoring.
GitClear, 2025Independent benchmarks show assistants often fail at tests that catch bugs.
5-29%
Copilot Test Coverage
Diffblue's 2025 benchmark found GitHub Copilot achieves only 5-29% code coverage when generating tests. Claude Code managed 7-17%. 12% of Copilot's generated tests fail to compile entirely.
Diffblue Benchmark, 20254%
Mutations Caught Despite 100% Coverage
The MutGen study showed LLMs can generate tests with 100% line coverage that catch only 4% of mutations, tests that execute every line but miss 96% of potential bugs. Coverage is a vanity metric.
MutGen / arXiv, 202545%
AI Code Has OWASP Top 10 Vulnerabilities
Veracode's July 2025 report across 100+ LLMs and 80 coding tasks found 45% of AI-generated code introduces OWASP Top 10 security vulnerabilities. Java showed a 72% security failure rate.
Veracode, July 20252,500%
Predicted Defect Increase
Gartner predicts prompt-to-app approaches will increase software defects by 2,500% by 2028, driven by "context-deficient code" that lacks system architecture awareness.
Gartner Predicts, Dec 2025Quality is rarely a first-class output beside the generated code.
“University of Turku research found Copilot test generation "inconsistent and frequently requires human intervention" with common test smells including Magic Number Tests and Lazy Tests.”
“Cursor only generates code, it does not validate whether tests run, fail, or are high quality. Requires constant developer oversight for any testing workflow.”
“Cognition's own November 2025 review described Devin as "senior-level at codebase understanding but junior at execution." They acknowledge humans must check test logic after Devin takes the first pass.”
“Amazon Q launched unit test generation in December 2024, but operates on a single file at a time with no cross-file architectural awareness or integration test capability.”
“Started as a test generation tool but pivoted primary focus to code review. Their open-source test generation tool (Qodo Cover) is no longer actively maintained.”
“No current AI coding tool generates production-quality tests and documentation alongside code as first-class outputs. Testing is always manual, always secondary, always an afterthought.”
Current tools generate code. Arvad keeps tests, docs, and deploy in the same workspace as the change.
Arvad sits where the thread keeps the plan, staging, spend, and checks.
| Capability | Code-only AI tools | Arvad |
|---|---|---|
| Test Generation | Manual prompt required, 5-29% coverage | Automatic with every build |
| Test Quality | 24.8% execution rate, 4% mutation catch | Validated tests with mutation-aware generation |
| Integration Tests | Not supported (single-file only) | Cross-service API and database tests included |
| E2E Test Suites | Not generated | Critical user flows tested automatically |
| Documentation | Not generated or quickly outdated | Docs stay attached and update with the change |
| Architecture Docs | Manual or nonexistent | Generated from codebase analysis, kept current |
| API Documentation | Separate tool (Swagger/OpenAPI) | Auto-generated specs, examples, and guides |
| Code Intent / "Why" | Never documented by AI | Business logic and design decisions captured |
| Security Testing | 45% OWASP vulnerabilities introduced | Security patterns enforced, OWASP scans built-in |
| Quality Gates | Manual CI/CD setup required | Built-in coverage thresholds, mutation scores |
Treat testing and documentation as first-class outputs, automated with each build.
Unit tests aimed at real defects, not vanity coverage.
Stays in the workspace with plan and checks
Cross-service, database, and API contract tests with the build.
Stays in the workspace with plan and checks
E2E suites for critical user flows before deploy.
Stays in the workspace with plan and checks
Security scans in the generation path.
Stays in the workspace with plan and checks
OpenAPI specs and examples generated from code and updated with changes.
Stays in the workspace with plan and checks
Docs cover what the code does and why the design chose it.
Stays in the workspace with plan and checks
Tests and docs update as code changes.
Stays in the workspace with plan and checks
Coverage thresholds, mutation scores, and scan results gate the shareable green status.
Stays in the workspace with plan and checks
Quality practices run with every build cycle, not as a later cleanup.
Quality practices run with every build cycle, not as a later cleanup.
Bugs found in production cost far more than bugs caught in development. Third-party research below.
$2.41T
Annual Cost of Poor Software Quality
CISQ estimated the cost of poor software quality in the U.S. reached $2.41 trillion in 2022, with accumulated technical debt at $1.52 trillion. AI-generated code without tests is accelerating this crisis.
CISQ / Synopsys, 202242%
Developer Time on Maintenance
Stripe's Developer Coefficient found developers spend 42% of their work week (17.3 hours) on maintenance, debugging, and refactoring, representing $85 billion in annual opportunity cost globally.
Stripe Developer Coefficient, 201840-90%
Fewer Defects With Test-Driven Development
Microsoft Research and IBM's landmark study found TDD reduces pre-release defect density by 40-90% across industrial teams, at a cost of only 15-35% more initial development time.
Nagappan et al., Microsoft Research100×
Cost Multiplier for Production Bugs
The IBM Systems Sciences Institute finding: fixing a bug costs 1× during requirements, 6× during implementation, and up to 100× in production. AI-generated code without tests skips straight to production risk.
DORA research across large samples ties automated testing and docs to performance.
3.7×
More Likely to Use Continuous Testing
DORA found elite teams are 3.7× more likely to leverage continuous testing and 5.8× more likely to leverage continuous integration. Manual testing accounts for only 10% of their effort.
DORA / Google, Multi-year2.4×
More Likely to Meet Reliability Targets
Teams with quality documentation are 2.4× more likely to meet reliability targets and 3.8× more likely to implement security practices. Documentation isn't nice-to-have, it's a performance multiplier.
DORA Report, 202450%
Faster Developer Onboarding
Organizations with comprehensive documentation report 50%+ reduction in developer onboarding time and 75% less senior-developer time needed for support. Google identified poor docs as a top-3 onboarding hindrance.
DX Research / Google78-93%
Cost Reduction From Test Automation
Organizations implementing comprehensive automated testing report 78-93% reduction in quality-related expenses, 40-75% faster release velocity, and 50-80% fewer production defects.
Industry AnalysisThe gap
Vibe coding (Collins Dictionary Word of the Year 2025) means accepting AI output without understanding or testing it. That habit is moving into enterprise workflows.
36%
A 2025 arXiv grey literature review found 36% of vibe coders accept AI output without any validation. Only 29% perform manual testing. The speed-first mindset is creating untested, undocumented production code at scale.
arXiv Grey Literature Review, 202569
Tenzai tested Claude Code, OpenAI Codex, Cursor, Replit, and Devin across 15 applications and found 69 total vulnerabilities, with approximately half a dozen rated "critical." A Wiz study found 20% have serious security issues.
Tenzai Assessment, Dec 202545%
Stack Overflow 2025: 45% of developers say debugging AI-generated code is more time-consuming than debugging their own. Trust in AI accuracy fell to just 29%. Positive sentiment dropped from 70%+ to 60%.
Stack Overflow Developer Survey, 202519%
The METR RCT found experienced developers using AI were 19% slower yet believed they were 20% faster. The perception gap is dangerous, teams think they're shipping quality when they're shipping debt.
METR Randomized Controlled Trial, 2025
Solo or team. Upgrade when usage grows.
Compare plansPick the plan that matches how you build. | FreeStart free | StarterChoose plan | ProfessionalChoose plan | EnterpriseChoose plan |
|---|---|---|---|---|
| Credits included | 120 credits included | 240 credits Included | 500 credits Included | |
| Projects per week | 1 mini project | 2 mini projects | 4 mini + 2 medium projects | |
| Documentation access | Essential | Standard | Full | |
| Planning & task breakdown | ||||
| Dependency resolution | ||||
| Queue-based processing | ||||
| Daily questions | Limited | Up to 10 / per day | Up to 25 / per day | |
| Upgrade options | ---- | Large projects | Large-scale projects | |
| Price | $0.0 | $25 | $50 |
Stats on this page cite published research. Read the sources.
Copilot: 5-29% coverage, 12% compile failures. Claude Code: 7-17%. Most rigorous independent comparison of AI test generation quality.
Only 24.8% of AI-generated tests pass execution. 57.9% fail to compile. 85.5% of runtime failures from incorrect assertions.
LLMs achieve 100% line coverage but catch only 4% of mutations. Proves coverage is a vanity metric without mutation testing.
Comprehensive evaluation finding all studied LLMs underperform traditional tools like EvoSuite in test coverage and bug detection.
Fortune 50 study: 322% more privilege escalation paths, 153% more architectural flaws. 10,000+ new findings/month by June 2025.
211M lines analyzed. 8× duplication increase. Refactoring dropped from 25% to <10%. Code churn nearly doubled.
45% of AI code introduces OWASP Top 10 vulnerabilities across 100+ LLMs. Java at 72% failure rate.
Predicts 2,500% defect increase from prompt-to-app. Identifies "context-deficient code" as new defect class.
Elite teams 3.7× more likely to use continuous testing. Documentation quality is AI's "biggest growth opportunity."
TDD reduces defect density 40-90% at 15-35% initial time cost. Landmark study across 4 industrial teams.
Developers spend 42% of time on maintenance. $85B annual opportunity cost. $300B GDP lost to developer inefficiency.
$2.41 trillion in quality costs. $1.52 trillion accumulated technical debt. Testing is the highest-leverage intervention.
Limits stated plainly. No demo theater.
One question
If you cannot, you still have a chat log. Arvad keeps that work in one workspace.
Keep exploring
Tests and docs ship with the change, not as an afterthought.