Coverage.py
The standard Python code coverage measurement tool. Coverage.py tracks which lines and branches of Python code are executed during test runs, generating HTML, XML, JSON, and terminal reports. Integrates with pytest via pytest-cov, supports branch coverage, omit patterns, and parallel execution tracking. Essential for measuring test suite completeness in Python agent codebases.
Score Breakdown
⚙ Agent Friendliness
🔒 Security
Local-only tool — no network calls. No sensitive data exposure. Code coverage data may reveal sensitive code structure if .coverage files are committed to public repos.
⚡ Reliability
Best When
You want to measure Python test suite completeness — coverage.py is the universal standard for Python coverage measurement with excellent pytest integration.
Avoid When
You need mutation testing (coverage doesn't tell you if tests are good, just that lines run) — use mutmut for quality-over-quantity coverage assessment.
Use Cases
- • Measure test coverage for Python agent tools — identify untested code paths before deploying to production
- • Generate coverage reports in CI/CD pipelines for agent repositories with --fail-under threshold enforcement
- • Track branch coverage (--branch) to verify all conditional logic paths in agent decision code are tested
- • Combine coverage data from parallel test runs (coverage combine) for accurate total coverage in distributed test suites
- • Integrate with Codecov or Coveralls for coverage trend tracking across agent codebase pull requests
Not For
- • Non-Python codebases — use language-specific coverage tools (istanbul for JS, grcov for Rust, go test --cover for Go)
- • Runtime performance profiling — use cProfile or py-spy for performance measurement, not coverage
- • Mutation testing — use mutmut or cosmic-ray for checking if tests actually catch bugs, not just execute lines
Interface
Authentication
Local CLI tool — no external auth.
Pricing
Apache 2.0 open source. Maintained by Ned Batchelder.
Agent Metadata
Known Gotchas
- ⚠ coverage run and pytest --cov use different .coverage data files — mixing both in the same run causes coverage data conflicts; choose one method per project
- ⚠ Branch coverage (--branch) requires source=. to be set — enabling branch coverage without source specification misses many branches in imported modules
- ⚠ Parallel test runs (pytest-xdist) require coverage's parallel=true config and coverage combine after test run — forgetting combine produces incomplete coverage data
- ⚠ Dynamic code (exec(), eval(), importlib) is not tracked by coverage — agent code that dynamically executes Python strings will show as untested even when exercised
- ⚠ --fail-under threshold in CI blocks PRs when coverage drops — ensure the threshold is achievable for the codebase and configure per-module omit patterns for generated code
- ⚠ Coverage data from different Python versions is incompatible — CI matrices running multiple Python versions must generate and combine separate .coverage files for accurate aggregation
Full Evaluation Report
Detailed scoring breakdown, competitive positioning, security analysis, and improvement recommendations for Coverage.py.
Scores are editorial opinions as of 2026-03-06.