leptos-shadcn-ui/docs/testing/coverage-reporting.md

Test Coverage Reporting Guide

This guide explains how to use and configure the automated test coverage reporting system for the Leptos ShadCN UI project.

Overview

The coverage reporting system provides comprehensive test coverage analysis with support for multiple tools, output formats, and CI/CD integration. It helps ensure code quality and track coverage trends over time.

Features

• Multi-tool Support: Works with cargo-llvm-cov (recommended) and cargo-tarpaulin
• Multiple Output Formats: HTML, JSON, LCov, and terminal reports
• Coverage Thresholds: Configurable quality gates with fail-fast support
• Trend Analysis: Track coverage changes across builds
• CI/CD Integration: GitHub Actions workflow with PR comments
• Component-Level Coverage: Detailed breakdown by component
• Coverage Badges: Auto-generated SVG badges for README/docs

Quick Start

Installation

First, install the coverage tool:

# Install cargo-llvm-cov (recommended)
cargo install cargo-llvm-cov

# Or install cargo-tarpaulin
cargo install cargo-tarpaulin

Generate Coverage Reports

Generate all coverage report formats:

# Using the shell script
./scripts/generate_coverage_report.sh

# Using the Python script
python3 scripts/coverage_reporter.py

# Using make
make coverage

View HTML Report

# Generate and open HTML report
make coverage-html

# Or using the script
./scripts/generate_coverage_report.sh --format html --open

Usage

Command-Line Options

Shell Script (generate_coverage_report.sh)

./scripts/generate_coverage_report.sh [OPTIONS]

Options:
  --workspace          Generate coverage for entire workspace (default)
  --package <name>     Generate coverage for specific package
  --format <fmt>       Output format: html, json, lcov, terminal, all (default: all)
  --output-dir <dir>   Custom output directory (default: coverage-reports/)
  --open               Open HTML report in browser after generation
  --fail-under <pct>   Fail if coverage below percentage (default: 95)
  --verbose            Enable verbose output
  --ci                 CI mode (optimized for CI/CD pipelines)
  --help               Show help message

Examples:

# Generate all formats
./scripts/generate_coverage_report.sh --format all

# Generate coverage for specific component
./scripts/generate_coverage_report.sh --package leptos-shadcn-button

# Generate HTML report and open in browser
./scripts/generate_coverage_report.sh --format html --open

# CI mode with custom threshold
./scripts/generate_coverage_report.sh --ci --fail-under 90

Python Script (coverage_reporter.py)

python3 scripts/coverage_reporter.py [OPTIONS]

Options:
  --tool <tool>           Coverage tool: llvm-cov, tarpaulin, all (default: llvm-cov)
  --format <fmt>          Output format: html, json, lcov, terminal, all (default: all)
  --output-dir <dir>      Custom output directory
  --fail-under <pct>      Fail if coverage below percentage (default: 95)
  --component <name>      Generate coverage for specific component
  --trend                 Enable trend analysis
  --ci                    CI mode
  --verbose               Enable verbose output

Makefile Targets

# Generate all coverage reports
make coverage

# Generate HTML report and open
make coverage-html

# Generate JSON report
make coverage-json

# Generate LCov report
make coverage-lcov

# Generate terminal report
make coverage-terminal

# Generate coverage for specific component
make coverage-component COMPONENT=button

# Verify coverage meets thresholds
make coverage-verify

# Generate all reports including badges and trends
make coverage-full

Configuration

Coverage Thresholds

Configure coverage thresholds in .coverage-rules.toml:

[overall]
min_line_coverage = 95.0
min_branch_coverage = 90.0
min_function_coverage = 100.0
fail_below_threshold = true

[component_thresholds]
default_min_coverage = 90.0
button_min_coverage = 95.0
input_min_coverage = 95.0

Environment Variables

Set environment variables in .cargo/config.toml or via shell:

# Coverage settings
export COVERAGE_MINIMUM_LINE_COVERAGE=95
export COVERAGE_MINIMUM_BRANCH_COVERAGE=90
export COVERAGE_MINIMUM_FUNCTION_COVERAGE=100

# Output settings
export COVERAGE_OUTPUT_DIR=coverage-reports
export COVERAGE_REPORT_TYPES=html,json,lcov

# Disable incremental compilation for accurate coverage
export CARGO_INCREMENTAL=0

Report Formats

HTML Report

Interactive HTML report with source-level coverage highlighting:

make coverage-html
# Output: coverage-reports/html/index.html

Features:

• Line-by-line coverage highlighting
• Branch coverage visualization
• Searchable component list
• Responsive design

JSON Report

Machine-readable JSON for CI/CD integration:

make coverage-json
# Output: coverage-reports/coverage.json

Structure:

{
  "timestamp": "2024-01-10T12:00:00Z",
  "git_branch": "main",
  "git_commit": "abc123",
  "overall": {
    "line_coverage": 95.5,
    "branch_coverage": 92.0,
    "function_coverage": 100.0
  },
  "components": [...]
}

LCov Report

Standard LCov format for integration with coverage services:

make coverage-lcov
# Output: coverage-reports/coverage.lcov

Terminal Report

Quick terminal summary:

make coverage-terminal

Output:

================================================================================
COVERAGE REPORT
================================================================================
Tool: llvm-cov
Branch: main
Commit: abc123

OVERALL METRICS
--------------------------------------------------------------------------------
overall: Lines: 95.5% (1523/1595), Branches: 92.0% (876/952), Functions: 100.0% (234/234)

COMPONENT METRICS
--------------------------------------------------------------------------------
button: Lines: 96.2% (127/132), ...

CI/CD Integration

GitHub Actions

The .github/workflows/coverage-report.yml workflow automatically:

1. Runs on every PR and push to main
2. Generates coverage reports
3. Posts coverage summary as PR comment
4. Uploads reports as artifacts
5. Fails PR if coverage is below threshold
6. Updates coverage badges

Coverage Comments

PR comments include:

• Overall coverage percentage
• Pass/fail status
• Branch and commit info
• Link to full report

Coverage Badges

Auto-generated badges are stored in docs/badges/coverage.svg and updated automatically on main branch builds.

Display in README:

![Coverage](docs/badges/coverage.svg)

Coverage Thresholds

Default Thresholds

• Line Coverage: 95%
• Branch Coverage: 90%
• Function Coverage: 100%

Component-Specific Thresholds

Some components have higher requirements:

[component_thresholds]
button_min_coverage = 95.0
input_min_coverage = 95.0
form_min_coverage = 95.0

Quality Gates

Coverage gates are enforced in:

• CI/CD pipelines
• Pre-commit hooks
• Manual verification commands

Troubleshooting

Low Coverage Results

If coverage seems low:

1. Check tool settings: Different tools measure coverage differently

   # Use llvm-cov for comprehensive coverage
   ./scripts/generate_coverage_report.sh --tool llvm-cov
   

2. Verify test execution: Ensure tests are actually running

   cargo test --workspace
   

3. Check exclusion patterns: Review .coverage-rules.toml exclusions

Missing Coverage Data

If reports are empty:

1. Clean previous data

   make clean
   rm -rf target/coverage
   

2. Disable incremental compilation

   export CARGO_INCREMENTAL=0
   

3. Rebuild with coverage instrumentation

   cargo clean
   ./scripts/generate_coverage_report.sh
   

CI Failures

If CI fails on coverage:

1. Check threshold settings in .coverage-rules.toml
2. Review coverage report in GitHub Actions artifacts
3. Compare with baseline to identify degradation
4. Update threshold if appropriate (document reason)

Best Practices

1. Run locally first: Check coverage before pushing

   make coverage-verify
   

2. Review HTML reports: Use HTML reports to identify gaps

   make coverage-html
   

3. Monitor trends: Track coverage over time

   make coverage-trend
   

4. Set realistic thresholds: Balance quality with development speed

5. Document exceptions: Use exclusions sparingly and document reasons

6. Fix degradations: Address coverage decreases promptly

Advanced Usage

Custom Coverage Tools

Integrate other coverage tools by extending coverage_reporter.py:

class CustomCoverageTool(CoverageTool):
    def check_installed(self) -> bool:
        # Check if tool is installed
        pass

    def run_coverage(self, packages) -> CoverageReport:
        # Run coverage analysis
        pass

Custom Report Formats

Add custom output formats in ReportGenerator:

def generate_custom_report(self, report: CoverageReport) -> Path:
    # Generate custom format
    pass

Historical Tracking

Enable trend analysis:

python3 scripts/coverage_reporter.py --trend

Historical data is stored in coverage-history.json.

Support

For issues or questions:

• Check existing documentation in docs/testing/
• Review GitHub Actions logs
• Open an issue with coverage report attached

Related Documentation

• Testing Guide
• CI/CD Configuration
• Quality Gates