pytest-cov
Code coverage reporting for pytest — integrates coverage.py with pytest for automatic coverage measurement during test runs. pytest-cov features: --cov=src flag for source directory coverage, --cov-report=term/html/xml/json for output formats, --cov-fail-under=80 for minimum coverage threshold, --cov-branch for branch coverage, parallel test support, subprocess coverage via COVERAGE_PROCESS_START, omit patterns, coverage.ini configuration, and Codecov/Coveralls upload integration. Standard coverage tool for Python agent CI/CD pipelines.
Score Breakdown
⚙ Agent Friendliness
🔒 Security
Local coverage tool — no security concerns. Coverage XML reports may be uploaded to Codecov — ensure CODECOV_TOKEN is stored as CI secret, not hardcoded. Coverage reports don't contain source code, only line hit counts.
⚡ Reliability
Best When
Every Python agent project running pytest — pytest-cov provides coverage reporting, CI gates, and team visibility into untested agent code with zero configuration beyond --cov flag.
Avoid When
You're not using pytest, need mutation testing, or run performance-sensitive test suites where coverage overhead is unacceptable.
Use Cases
- • Agent CI coverage gate — pytest --cov=src/agent --cov-report=xml --cov-fail-under=80 — fails CI if agent code coverage below 80%; XML report uploaded to Codecov; GitHub PR shows coverage diff
- • Agent test coverage report — pytest --cov=src --cov-report=html — generates htmlcov/ directory; open htmlcov/index.html to see agent code coverage by file and line; red lines are untested agent code paths
- • Agent branch coverage — pytest --cov=src --cov-branch --cov-report=term-missing — branch coverage shows if/else branches both tested; agent decision logic with untested branches flagged; more thorough than line coverage alone
- • Agent coverage threshold enforcement — [tool.pytest.ini_options] addopts = '--cov=src --cov-fail-under=85' in pyproject.toml — every pytest run checks coverage; agent teams maintain 85% threshold without CI flag
- • Codecov integration for agent PRs — pytest --cov=src --cov-report=xml; codecov upload -f coverage.xml — Codecov comments on agent PRs showing coverage change; coverage delta visible in PR review
Not For
- • Mutation testing — coverage shows which lines run, not if tests assert correctly; for agent test quality use mutmut or cosmic-ray for mutation testing
- • Performance profiling — coverage.py adds overhead; for agent performance profiling use cProfile or py-spy
- • Non-pytest test runners — pytest-cov requires pytest; for unittest or nose use coverage.py directly
Interface
Authentication
No auth — local test coverage tool. Codecov upload uses CODECOV_TOKEN environment variable.
Pricing
pytest-cov is MIT licensed. Free for all use. Codecov (upload service) has free tier for public repos.
Agent Metadata
Known Gotchas
- ⚠ --cov flag slows tests 10-30% — pytest --cov adds coverage.py instrumentation overhead; agent test suite taking 60 seconds takes 66-78 seconds with --cov; separate CI jobs: one for fast tests (no coverage), one for coverage reporting; use --no-cov for development test runs
- ⚠ Coverage not measured for subprocess code — agent code spawning subprocesses (subprocess.run) or multiprocessing not covered by default; COVERAGE_PROCESS_START=.coveragerc environment variable and sitecustomize.py setup required; agent tests using subprocess to test CLI tools show 0% coverage on CLI code
- ⚠ --cov source path matters — pytest --cov=src covers src/ directory; running from wrong directory (e.g., inside src/) uses wrong relative path; use absolute paths or package name: --cov=agent instead of --cov=src/agent; CI coverage may show 0% if cov path doesn't match installed package
- ⚠ --cov-fail-under exits 2 not 1 — coverage failure exits with code 2 (not 1); CI systems checking exit code != 0 correctly catch failure; but scripts checking exit code == 1 specifically miss coverage failures; agent CI scripts must check exit code != 0
- ⚠ Branch coverage doubles uncovered count — --cov-branch counts each branch of if/elif/else separately; agent code with many conditionals shows drastically lower coverage with branch enabled vs line-only; establish baseline before adding branch coverage requirement to avoid CI disruption
- ⚠ Parallel test coverage requires pytest-xdist integration — pytest --cov --numprocesses=4 with pytest-xdist requires --cov-append and proper parallel setup; coverage.py may not capture all subprocess coverage in forked workers; use coverage combine after parallel run; agent test suites using -n auto for parallelism need explicit coverage merge step
Alternatives
Full Evaluation Report
Detailed scoring breakdown, competitive positioning, security analysis, and improvement recommendations for pytest-cov.
Scores are editorial opinions as of 2026-03-06.