New
View the LLM Leaderboard for Code Quality, Complexity, and Security
State of Code report series

The Coding Personalities of Leading LLMs

Make smarter AI adoption decisions with Sonar's latest report in The State of Code series. Explore the habits, blind spots, and archetypes of the top five LLMs to uncover the critical risks each brings to your codebase.

Download the reportExecutive summary

Key findings

Our deep analysis of LLM-generated code goes beyond standard benchmarks.

developer

Coding personalities

Each LLM has a distinct style that impacts your production environment.

handshake

Shared strengths

All models consistently produce valid and create viable solutions for well-defined problems.

warning

Shared blind spots

All models have a fundamental lack of security awareness and a bias for messy code.

arrows pointing up on a diagonal

Upgrades increase risk

Newer models can generate bugs that are almost twice as likely to be of the highest severity.

What our analysis uncovered

%
more likely for new Claude model to be of 'BLOCKER' severity than its predecessor.
%
of all issues found in LLM-generated code create long-term technical debt.
%
of the vulnerabilities for one LLM are of ‘BLOCKER’ severity.
%
of all bugs from one popular LLM are control-flow mistakes.

Methodology

Our analysis is based on 4,442 identical programming tasks performed by each LLM. We measured their output across multiple dimensions to create a comprehensive profile of each model's coding personality and risk profile.

feedback

Verbosity Measurement

Verbosity quantifies the sheer volume of code each model generates to solve identical tasks.

  • Lines of Code (LOC)
    Total number of lines of code generated across all 4,442 tasks, including blank lines and comments. This metric reveals whether a model tends toward concise or elaborate implementations.
  • Token Count
    Total tokens generated in the code output, providing a language-agnostic measure of code volume that accounts for the actual content density.
  • Code Density
    Ratio of executable statements to total lines, indicating how compact or spread out the code structure is.
code merge

Complexity Measurement

Complexity metrics evaluate the structural and logical intricacy of the generated code using industry-standard software engineering measures.

  • Cyclomatic Complexity
    Measures the number of linearly independent paths through the code. Higher values indicate more complex control flow with multiple branches, loops, and decision points.
  • Cognitive Complexity
    Evaluates how difficult code is for humans to understand, accounting for nested structures and non-linear control flow. This metric better reflects maintainability concerns than cyclomatic complexity alone.
  • Nesting Depth
    Maximum depth of nested structures (loops, conditionals, try-catch blocks). Deeper nesting typically correlates with harder-to-maintain code.
pdf

Communication & Documentation

Documentation metrics measure how thoroughly each model explains its code through comments and inline documentation.

  • Comment Density (%)
    Percentage of lines containing comments relative to total lines of code. Calculated as (comment lines / total lines) × 100.
  • Docstring Coverage
    Percentage of functions, classes, and modules that include docstrings or header comments explaining their purpose and usage.
  • Inline Explanation Quality
    Qualitative assessment of whether comments add meaningful context or merely restate obvious code operations.
secure

Software Quality Analysis

Quality metrics identify issues across three critical dimensions using static analysis tools.

  • Reliability (Bugs)
    Code defects that would cause runtime errors, incorrect behavior, or system failures. Includes control-flow mistakes, resource leaks, and concurrency issues.
  • Security (Vulnerabilities)
    Weaknesses that could be exploited by attackers, such as injection flaws, path traversal vulnerabilities, and insecure cryptographic practices. Severity ranges from INFO to BLOCKER.
  • Maintainability (Code Smells)
    Design weaknesses that increase technical debt and make code harder to modify. Includes duplicated code, overly complex functions, and poor naming conventions.
automatic

Functional Performance

Pass rate measures each model's ability to produce functionally correct solutions.

  • Pass Rate (%)
    Percentage of the 4,442 tasks where the generated code produces correct output for all test cases. This establishes the baseline capability before considering code quality.
  • Issue Density
    Total number of reliability, security, and maintainability issues per thousand lines of code (issues/KLOC). Lower values indicate cleaner code.

The coding archetypes of leading LLMs

Our analysis shows that each LLM has a unique and measurable coding personality. Which one have you "hired" for your team?

The "trust but verify" mandate for AI

AI is now a core part of software development, but performance benchmarks alone are misleading. They can lead to LLMs that solve difficult challenges but fail to write good, secure, and reliable code. To harness these powerful models responsibly, you must look beyond the benchmark. Our report provides the critical insights needed to choose the right models and use them safely.

Download the report

The three qualities of software source code

Sonar classifies the issues found in every project or codebase across three deeply interconnected software qualities: reliability, security, and maintainability.

smily

Reliability

Bugs that would affect the software's capability to maintain its level of performance under promised conditions, potentially compromising its reliability and operational effectiveness.

Advanced Security demo
lock

Security

Vulnerabilities and security hotspots. Vulnerabilities are code weaknesses that could be exploited for attacks, while hotspots are security-sensitive code requiring manual review.

SonarQube demo
code

Maintainability

Code smells, which could indicate weaknesses in design that can increase technical debt, slow down development, or increase the risk of bugs or failures down the line.

AI Code Assurance demo

Security Vulnerability Analysis

Security vulnerabilities in AI-generated code pose significant risks. Our analysis reveals distinct patterns in how each LLM handles security-critical code, with some models producing vulnerabilities of BLOCKER severity at alarming rates.

warning

Severity Distribution

Newer models show alarming increases in BLOCKER-severity vulnerabilities, with some models producing critical security flaws in over 70% of cases.

secure

Common Vulnerability Types

Injection flaws, path-traversal, and insecure cryptography are prevalent across all models, indicating fundamental gaps in security awareness.

arrows pointing up on a diagonal

Model Upgrades = Higher Risk

Claude Sonnet 4 is 93% more likely to produce BLOCKER vulnerabilities compared to its predecessor, showing that newer isn't always safer.

Ready to release secure, reliable, and maintainable software?

Request demoStart for free


TRUSTED BY OVER 7M DEVELOPERS WORLDWIDE
Mercedes Benz
Nvidia
Santander