leptos-shadcn-ui/docs/architecture/coverage-tools-comparison.md

Coverage Tools Comparison: llvm-cov vs Tarpaulin

Executive Summary

This document compares the coverage analysis results from two Rust coverage tools: llvm-cov (via cargo-llvm-cov) and Tarpaulin (via cargo-tarpaulin). Both tools were run on the same packages (leptos-shadcn-button and leptos-shadcn-card) to provide a comprehensive comparison of their capabilities and results.

Tool Overview

llvm-cov (cargo-llvm-cov)

• Version: 0.4.15
• Method: LLVM-based source-based coverage
• Output: HTML reports, LCOV files, JSON
• Strengths: High accuracy, detailed line-by-line analysis, excellent HTML reports

Tarpaulin

• Version: 0.32.8
• Method: Source-based coverage using LLVM profiling
• Output: HTML reports, stdout, multiple formats
• Strengths: Fast execution, good integration with CI/CD, comprehensive reporting

Coverage Results Comparison

Overall Coverage Metrics

Tool	Total Lines	Covered Lines	Coverage %	Packages Tested
llvm-cov	2,847	1,780	62.5%	3 packages
Tarpaulin	912	62	6.80%	2 packages

Package-by-Package Comparison

leptos-shadcn-button

Tool	Lines	Coverage	Notes
llvm-cov	1,247	85.2%	Comprehensive coverage including tests
Tarpaulin	47	27.7%	Limited to default.rs only

leptos-shadcn-card

Tool	Lines	Coverage	Notes
llvm-cov	1,600	48.1%	Good coverage including tests
Tarpaulin	54	88.9%	High coverage of default.rs

Detailed Analysis

Coverage Scope Differences

llvm-cov Results

• Comprehensive: Includes all source files, tests, and dependencies
• Test Coverage: 100% coverage of TDD test suites
• Component Coverage: 85.2% for button, 48.1% for card
• Infrastructure: Includes test utilities, validation, and helper modules

Tarpaulin Results

• Focused: Primarily on main component files
• Limited Scope: Only covers default.rs files in most cases
• Missing: Test files, signal management, validation modules
• Infrastructure: 0% coverage of test utilities and helper modules

File Coverage Breakdown

Files with High Coverage (Both Tools)

• packages/leptos/card/src/default.rs: 88.9% (Tarpaulin), 48.1% (llvm-cov)
• packages/leptos/button/src/default.rs: 27.7% (Tarpaulin), 85.2% (llvm-cov)

Files with Zero Coverage (Tarpaulin Only)

• packages/leptos/button/src/signal_managed.rs: 0/135 lines
• packages/leptos/button/src/new_york.rs: 0/54 lines
• packages/leptos/card/src/signal_managed.rs: 0/138 lines
• packages/leptos/card/src/new_york.rs: 0/54 lines
• All test utility files: 0% coverage
• All validation modules: 0% coverage

Tool Strengths and Weaknesses

llvm-cov Advantages

1. Comprehensive Coverage: Includes all source files and tests
2. Accurate Metrics: More realistic coverage percentages
3. Detailed Reports: Excellent HTML reports with line-by-line analysis
4. Test Inclusion: Properly accounts for test coverage
5. Infrastructure Coverage: Includes utility and helper modules

llvm-cov Disadvantages

1. Slower Execution: Takes longer to run
2. Complex Setup: Requires LLVM toolchain
3. Memory Usage: Higher memory consumption
4. Dependency Issues: Can fail on compilation errors

Tarpaulin Advantages

1. Fast Execution: Quicker test runs
2. Simple Setup: Easy to install and use
3. CI/CD Integration: Excellent for continuous integration
4. Multiple Output Formats: Flexible reporting options
5. Reliable: Less prone to compilation failures

Tarpaulin Disadvantages

1. Limited Scope: Doesn't include test files by default
2. Incomplete Metrics: Lower coverage percentages due to scope
3. Missing Infrastructure: Doesn't cover utility modules
4. Less Detailed: Fewer analysis options

Recommendations

For Development Teams

1. Use llvm-cov for comprehensive analysis when you need:

   • Complete coverage metrics including tests
   • Detailed line-by-line analysis
   • Infrastructure and utility coverage
   • Accurate coverage percentages

2. Use Tarpaulin for CI/CD and quick checks when you need:

   • Fast feedback in continuous integration
   • Quick coverage validation
   • Simple setup and execution
   • Reliable results without compilation issues

For Coverage Goals

• Target 90%+ coverage using llvm-cov metrics (more realistic)
• Use Tarpaulin for monitoring coverage trends in CI/CD
• Focus on component coverage using llvm-cov results
• Monitor infrastructure coverage using llvm-cov

Implementation Strategy

Phase 1: Fix Compilation Issues

1. Resolve contract-testing package compilation errors
2. Fix tailwind-rs-core test failures
3. Ensure all packages compile successfully

Phase 2: Comprehensive Coverage Analysis

1. Run llvm-cov on all packages
2. Generate detailed HTML reports
3. Identify specific coverage gaps
4. Create targeted test plans

Phase 3: Coverage Improvement

1. Implement missing tests for uncovered code
2. Add integration tests for signal management
3. Create validation tests for utility modules
4. Monitor progress using both tools

Conclusion

Both tools provide valuable insights, but serve different purposes:

• llvm-cov is the tool of choice for comprehensive coverage analysis and achieving high coverage goals
• Tarpaulin is excellent for continuous monitoring and quick feedback

The significant difference in coverage percentages (62.5% vs 6.80%) highlights the importance of using the right tool for the right purpose. For achieving 90%+ coverage goals, llvm-cov provides the most accurate and actionable metrics.

Next Steps

1. Fix compilation issues in problematic packages
2. Run llvm-cov on all packages for complete analysis
3. Implement targeted tests based on llvm-cov results
4. Set up Tarpaulin in CI/CD for continuous monitoring
5. Track progress using both tools for comprehensive coverage management