Static Analysis Signals as Predictors of Post-Release Defects: An Empirical Study of 1.2 Million Commits

1.Introduction

Static analysis occupies a strange position in software practice. Its value as a bug-finding tool is well established for specific defect classes. Its value as a measurement instrument — a dashboard number that tells an organisation whether quality is improving — is asserted far more often than it is tested, and it is in this second role that it drives release gates, team targets and occasionally performance review.

There is a straightforward reason to be suspicious. Static-analysis warning counts scale with the amount of code written and changed. Post-release defect counts also scale with the amount of code changed. Any study that regresses the second on the first without adjusting for change volume will find a strong relationship regardless of whether the warnings carry information, and much of the literature supporting dashboard practice has this structure.

We test the signals against that confound directly. Our research questions are: RQ1 Which of eleven commonly reported signals are associated with post-release defects before adjustment? RQ2 Which survive adjustment for change volume? RQ3 How much out-of-sample predictive value do the survivors add over change volume alone?

2.Related Work

Defect-prediction research has produced hundreds of models, and repeated methodological critiques have shown that reported performance is highly sensitive to data linking quality, validation scheme and the choice of baseline. Studies that compare against a change-volume baseline consistently report much smaller effects than those that compare against no baseline; our results extend that pattern to the specific signals used in commercial dashboards.

Work on warning actionability has separately established that most reported warnings are never acted upon and that their density varies more between projects than within them. That finding motivates our use of repository random intercepts and repository-held-out validation: without them, between-project variation is easily mistaken for predictive power.

3.Data and Method

Dataset. We mined 180 repositories — 148 open-source and 32 industrial, contributed under agreement by three firms — spanning Java, C++, Python and TypeScript, yielding 1.2 million commits and 8,412 releases. For each release we computed the eleven signals from a uniform analysis configuration rather than each project's own, so that thresholds do not vary with local policy.

Defect linking. Post-release defects were linked to releases through an issue-linking heuristic combining commit references, fix-version fields and temporal windows. We audited a random sample of 1,000 links manually, obtaining precision 0.91 and recall 0.84, and we report sensitivity analyses using the stricter reference-only variant.

Modelling. We fitted mixed-effects negative-binomial models of defect count per release with an offset for exposure duration and random intercepts for repository. Change volume entered as lines changed, files touched and commit count. We report both unadjusted and adjusted coefficients, and evaluate discrimination by repository-held-out cross-validation so that no repository contributes to both training and test — the validation choice that most affects reported performance in this literature.

Signals. The eleven signals comprised aggregate warning count, warning density, cyclomatic-complexity growth in modified functions, duplicated-block count, newly introduced nullability warnings on public interfaces, unresolved concurrency warnings, dead-code count, coupling metrics, comment-density change, test-file ratio change and lint-rule violations.

4.Results

Unadjusted. Ten of the eleven signals showed significant positive association with post-release defect counts, with aggregate warning count among the strongest. Taken at face value this reproduces the conventional dashboard justification.

Adjusted. After adding change volume, only three signals retained significance: cyclomatic-complexity growth in modified functions, newly introduced nullability warnings on public interfaces, and unresolved concurrency warnings. Aggregate warning count and warning density lost significance entirely, with coefficients shrinking toward zero rather than merely losing precision — the pattern expected of a pure proxy.

Out-of-sample. Change volume alone achieved AUC 0.71 under repository-held-out validation. All eleven signals together reached 0.74; the three retained signals alone reached 0.735. The practical reading is that eight of the eleven signals contribute essentially nothing beyond telling the model how much code changed.

Robustness. Results held under the stricter linking variant, under Poisson and zero-inflated alternatives, and when industrial and open-source repositories were modelled separately. Effect sizes for the three surviving signals were somewhat larger in the industrial subset, which had denser defect reporting.

5.Discussion

Change volume dominates. That is not a nihilistic conclusion, because the three surviving signals share an interpretable property: each identifies a specific structural risk introduced into code that is already being modified, rather than counting the accumulated state of the codebase. Signals of accumulated state describe project history; signals of introduced risk describe the change under review, and it is the latter that predict what breaks next.

For engineering practice this suggests rewriting quality gates in delta form. A gate on total warnings penalises large changes and is satisfied by small ones regardless of risk. A gate on complexity growth in touched functions, new public-interface nullability warnings and unresolved concurrency warnings targets what our data show to be informative, and is not automatically triggered by volume.

For research, our results add to the evidence that defect-prediction studies without a change-volume baseline and without repository-held-out validation should be interpreted cautiously. We recommend both as reporting minima and we release our pipeline to make them cheap to apply.

6.Threats to Validity

Issue linking is imperfect at 0.91 precision, and undercounted defects in low-reporting projects will attenuate coefficients; our industrial subset, with better reporting, showed larger effects, consistent with attenuation. We used a uniform analysis configuration, which improves comparability but does not reflect the tuned configurations organisations actually run.

Our repositories skew toward actively maintained projects with release discipline, since both are prerequisites for the measurement. Conclusions may not extend to sporadically maintained code. Finally, association under adjustment is not causation: we do not claim that reducing complexity growth reduces defects, only that it carries information about them.

7.Conclusion

Across 1.2 million commits and 8,412 releases in 180 repositories, ten of eleven common static-analysis signals predicted post-release defects until change volume was controlled, after which only three did: complexity growth in modified functions, newly introduced nullability warnings on public interfaces, and unresolved concurrency warnings. Change volume alone reached 0.71 AUC and all eleven signals added 0.03. Organisations should replace aggregate-count gates with delta-form gates on the three informative signal classes, and researchers should treat change volume as a mandatory baseline.

Data availability. The commit-level dataset, linking audit, analysis configuration and modelling scripts are released for replication, excluding the industrial repositories' source, which is covered by contribution agreements; derived features for those repositories are included.

8.References

1. [1] Russo, G. (2024). Quality proxies and the change-volume confound. Empirical Software Engineering 29(1), 77–98.
2. [2] Whitfield, B. & Lindqvist, K. (2025). Repository-held-out validation for defect models. Proceedings of MSR, 155–167.
3. [3] Lindqvist, K. (2024). Warning actionability across four languages. IEEE Transactions on Software Engineering 50(11), 2790–2808.
4. [4] Whitfield, B. & Russo, G. (2025). Reconciling issue links with release histories. Proceedings of ICPE, 74–86.
5. [5] Haddad, J. (2025). Delta-form quality gates in continuous delivery. JIPCET 2(1), 30–52.
6. [6] Osei, P. & Tan, M. (2024). Measurement instruments and engineering incentives. Journal of Systems and Software 216, 112–130.