-
-
Notifications
You must be signed in to change notification settings - Fork 0
feat: add automated code coverage protection for pull requests #27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
- Add new coverage-check job that runs on all pull requests - Compare coverage between PR branch and main branch - Track 4 metrics: lines, statements, functions, branches - Block PR merge if any metric decreases (zero-tolerance policy) - Post detailed coverage comparison comment on PRs - Show visual indicators (📈 increase, ➡️ same, 📉 decrease) - Fail CI step if coverage drops to prevent merge This ensures code quality is maintained or improved with every PR. Closes #26
- Update codecov.yml threshold from 1% to 0% (zero-tolerance) - Add if_ci_failed: error to fail CI on coverage drops - Add json-summary reporter to vitest config - Generate coverage-summary.json for CI comparison These changes enforce strict coverage requirements and provide the necessary data for automated coverage comparison. Related to #26
- Add coverage protection section to README.md - Enhance CONTRIBUTING.md with coverage requirements - Add step-by-step guide for checking coverage locally - Update PR requirements to include coverage as blocking - Add troubleshooting tips for coverage issues This provides clear guidance for contributors on how to maintain code coverage and fix coverage-related PR failures. Related to #26
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
🚀 Performance Benchmark Results📦 class-transformer Compatibility📊 Performance Comparison Summary📋 Full class-transformer Benchmark Output✅ class-validator Compatibility📋 Full class-validator Benchmark Output🎯 Core Performance⚡ Simple Mapping Benchmark🔧 Complex Transformations Benchmark💡 Note: These are absolute performance numbers from this PR. Benchmarked with Benchmark.js on Node.js 20 • View History |
✅ ESM Build ValidationStatus: All ESM validation checks passed! Test Matrix Results
Validation Steps
What This ValidatesThe ESM validation suite ensures:
✅ The package is ready for ESM consumption! This validation prevents issues like missing |
🚦 Important Notes for ReviewersThis PR Implements Issue #26This pull request fully implements the automated code coverage protection described in #26. How to Review
🧪 Testing PlanAfter this PR is merged:
This will confirm the coverage protection is working as expected. 📊 Expected Impact
🔗 Related Links
❓ Questions?If you have any questions about the implementation or need clarification on any aspect, please comment below! Ready for review! 🚀 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting
- Add fallback for main branch without coverage-summary.json - Create minimal coverage file with 0% when main doesn't have reporter - Add special PR comment for first-time setup - Show informative message explaining coverage protection will be active after merge - Prevent workflow failure when comparing against old main branch This allows the initial coverage protection PR to pass while still enabling full protection for all future PRs after merge.
🔧 Fix Applied - Coverage Check UpdatedIssue IdentifiedThe initial workflow run failed because the main branch doesn't have the Solution ImplementedI've pushed a fix (commit Changes Made:
Expected Behavior Now✅ This PR will show: ℹ️ Code Coverage Protection - First Time Setup
Status: ✅ PASSED (Initial Setup)
PR Coverage:
| Metric | This PR |
|-------------|---------|
| Lines | XX.XX% |
| Statements | XX.XX% |
| Functions | XX.XX% |
| Branches | XX.XX% |
Note: This is the first PR with coverage protection enabled.
Once merged, all future PRs will be compared and blocked if coverage decreases.✅ Future PRs (after this is merged) will show: ✅/❌ Code Coverage Check
Status: PASSED/FAILED
Coverage Comparison:
| Metric | Main Branch | This PR | Change | Status |
|--------|-------------|---------|--------|--------|
...Next Steps
The fix is now running in the latest workflow! 🏃 |
The template literals in the github-script action had escaped backticks which caused a JavaScript syntax error. In YAML literal block scalars (|), backticks should not be escaped. This fixes the 'Invalid or unexpected token' error in the github-script action.
🔧 Second Fix Applied - JavaScript Syntax Error ResolvedIssueThe github-script action was failing with: Root CauseThe template literals in the JavaScript code had escaped backticks ( Incorrect: script: |
const body = \`Hello ${name}\`; # ❌ Wrong!Correct: script: |
const body = `Hello ${name}`; # ✅ Correct!Fix AppliedCommit Status✅ YAML validated - No syntax errors The workflow should now complete successfully! 🎯 |
ℹ️ Code Coverage Protection - First Time SetupStatus: ✅ PASSED (Initial Setup) PR Coverage
📝 NoteThis is the first PR with coverage protection enabled. The main branch doesn't have the Once this PR is merged, all future PRs will be compared against the main branch coverage and will be blocked if coverage decreases. Coverage protection will be fully active after this PR is merged. |
- Add fallback values (|| echo "0") for coverage extraction - Set default values if variables are empty - Fix bc comparison syntax: use explicit -eq 1 check instead of (( )) - Quote $GITHUB_ENV to prevent word splitting issues - Always set MAIN_BRANCH_NO_COVERAGE (true or false) This fixes: - 'Invalid format' error when coverage is 100% - 'Unable to process file command env' error - Ensures 0% change (no decrease) is acceptable and passes the check
🔧 Third Fix Applied - Coverage Comparison RobustnessIssues Found
Root Causes
Fixes Applied (commit
|
ℹ️ Code Coverage Protection - First Time SetupStatus: ✅ PASSED (Initial Setup) PR Coverage
📝 NoteThis is the first PR with coverage protection enabled. The main branch doesn't have the Once this PR is merged, all future PRs will be compared against the main branch coverage and will be blocked if coverage decreases. Coverage protection will be fully active after this PR is merged. |
Replace grep/cut pipeline with jq for extracting coverage percentages. This is more reliable and handles edge cases better: - Properly extracts only from 'total' object, not individual files - Handles missing values with // 0 fallback operator - No issues with regex matching or field extraction - Cleaner and more maintainable code jq is pre-installed on GitHub Actions ubuntu-latest runners.
ℹ️ Code Coverage Protection - First Time SetupStatus: ✅ PASSED (Initial Setup) PR Coverage
📝 NoteThis is the first PR with coverage protection enabled. The main branch doesn't have the Once this PR is merged, all future PRs will be compared against the main branch coverage and will be blocked if coverage decreases. Coverage protection will be fully active after this PR is merged. |
- Changed coverage-check job condition to run on both pull_request events and push events to non-main branches - Added PR number detection for push events using GitHub API - Updated all github-script actions to use PR_NUMBER environment variable instead of context.issue.number - Applied same fix to esm-validation-summary job for consistency This ensures coverage protection runs on every push to a PR branch, not just on pull_request events.
ℹ️ Code Coverage Protection - First Time SetupStatus: ✅ PASSED (Initial Setup) PR Coverage
📝 NoteThis is the first PR with coverage protection enabled. The main branch doesn't have the Once this PR is merged, all future PRs will be compared against the main branch coverage and will be blocked if coverage decreases. Coverage protection will be fully active after this PR is merged. |
1 similar comment
ℹ️ Code Coverage Protection - First Time SetupStatus: ✅ PASSED (Initial Setup) PR Coverage
📝 NoteThis is the first PR with coverage protection enabled. The main branch doesn't have the Once this PR is merged, all future PRs will be compared against the main branch coverage and will be blocked if coverage decreases. Coverage protection will be fully active after this PR is merged. |
- Generate per-file coverage comparison between main and PR branches - Create GitHub Check Run with annotations for files with decreased coverage - Annotations appear in 'Files changed' tab similar to GitLab MR diff annotations - Add 'Coverage Changes by File' section to PR comment showing top 10 files - Sort files by coverage change (worst first) - Limit to 50 annotations per check run (GitHub API limit) This provides detailed visibility into which specific files have coverage changes, making it easier for reviewers to identify areas that need more tests.
The template literals should use regular backticks and syntax, not escaped versions. This was causing a syntax error in the github-script action.
ℹ️ Code Coverage Protection - First Time SetupStatus: ✅ PASSED (Initial Setup) PR Coverage
📝 NoteThis is the first PR with coverage protection enabled. The main branch doesn't have the Once this PR is merged, all future PRs will be compared against the main branch coverage and will be blocked if coverage decreases. Coverage protection will be fully active after this PR is merged. |
1 similar comment
ℹ️ Code Coverage Protection - First Time SetupStatus: ✅ PASSED (Initial Setup) PR Coverage
📝 NoteThis is the first PR with coverage protection enabled. The main branch doesn't have the Once this PR is merged, all future PRs will be compared against the main branch coverage and will be blocked if coverage decreases. Coverage protection will be fully active after this PR is merged. |
Using heredoc (<<'EOF') instead of double quotes prevents bash from interpreting template literals as shell variables. This fixes the 'Invalid or unexpected token' error in the coverage diff generation.
ℹ️ Code Coverage Protection - First Time SetupStatus: ✅ PASSED (Initial Setup) PR Coverage
📝 NoteThis is the first PR with coverage protection enabled. The main branch doesn't have the Once this PR is merged, all future PRs will be compared against the main branch coverage and will be blocked if coverage decreases. Coverage protection will be fully active after this PR is merged. |
1 similar comment
ℹ️ Code Coverage Protection - First Time SetupStatus: ✅ PASSED (Initial Setup) PR Coverage
📝 NoteThis is the first PR with coverage protection enabled. The main branch doesn't have the Once this PR is merged, all future PRs will be compared against the main branch coverage and will be blocked if coverage decreases. Coverage protection will be fully active after this PR is merged. |
Template literals in github-script should use regular backticks, not escaped ones. This was causing 'Invalid or unexpected token' error.
ℹ️ Code Coverage Protection - First Time SetupStatus: ✅ PASSED (Initial Setup) PR Coverage
📝 NoteThis is the first PR with coverage protection enabled. The main branch doesn't have the Once this PR is merged, all future PRs will be compared against the main branch coverage and will be blocked if coverage decreases. Coverage protection will be fully active after this PR is merged. |
ℹ️ Code Coverage Protection - First Time SetupStatus: ✅ PASSED (Initial Setup) PR Coverage
📝 NoteThis is the first PR with coverage protection enabled. The main branch doesn't have the Once this PR is merged, all future PRs will be compared against the main branch coverage and will be blocked if coverage decreases. Coverage protection will be fully active after this PR is merged. |
- Create GitHub Check Run with coverage diff in the title (e.g., 'Coverage: +2.5% 📈') - Add Commit Status showing coverage change in PR checks section - Display coverage diff prominently without scrolling to bot comment - Show emoji indicators: 📈 (increase), 📉 (decrease), ➡️ (no change) - Include both absolute values and percentage change - Handle first-time setup with 'Coverage: 95.08% (First PR)' format This makes coverage changes immediately visible in the PR header, similar to GitLab's merge request coverage display.
ℹ️ Code Coverage Protection - First Time SetupStatus: ✅ PASSED (Initial Setup) PR Coverage
📝 NoteThis is the first PR with coverage protection enabled. The main branch doesn't have the Once this PR is merged, all future PRs will be compared against the main branch coverage and will be blocked if coverage decreases. Coverage protection will be fully active after this PR is merged. |
1 similar comment
ℹ️ Code Coverage Protection - First Time SetupStatus: ✅ PASSED (Initial Setup) PR Coverage
📝 NoteThis is the first PR with coverage protection enabled. The main branch doesn't have the Once this PR is merged, all future PRs will be compared against the main branch coverage and will be blocked if coverage decreases. Coverage protection will be fully active after this PR is merged. |
Optimizations applied: - Add explicit minimal permissions (contents:read, pull-requests:write, checks:write, statuses:write) - Add concurrency group to cancel in-progress runs on new pushes - Optimize triggers to run only on main and feature/fix branches - Add 'require-successful-checks' job to ensure all checks pass before merge - Improve workflow structure and organization These optimizations reduce CI/CD resource usage and improve developer experience by canceling outdated workflow runs and providing clear merge requirements.
ℹ️ Code Coverage Protection - First Time SetupStatus: ✅ PASSED (Initial Setup) PR Coverage
📝 NoteThis is the first PR with coverage protection enabled. The main branch doesn't have the Once this PR is merged, all future PRs will be compared against the main branch coverage and will be blocked if coverage decreases. Coverage protection will be fully active after this PR is merged. |
1 similar comment
ℹ️ Code Coverage Protection - First Time SetupStatus: ✅ PASSED (Initial Setup) PR Coverage
📝 NoteThis is the first PR with coverage protection enabled. The main branch doesn't have the Once this PR is merged, all future PRs will be compared against the main branch coverage and will be blocked if coverage decreases. Coverage protection will be fully active after this PR is merged. |
Remove coverage protection documentation files that were added during the implementation. The coverage protection functionality is now fully integrated into the CI/CD workflow and these standalone documentation files are no longer needed. Removed files: - docs/COVERAGE_PROTECTION.md - COVERAGE_PROTECTION_SETUP.md - .github/COVERAGE_QUICK_REFERENCE.md - IMPLEMENTATION_SUMMARY.md - GITHUB_WORKFLOW_COMPLETE.md Existing documentation (README.md, CONTRIBUTING.md) remains intact.
ℹ️ Code Coverage Protection - First Time SetupStatus: ✅ PASSED (Initial Setup) PR Coverage
📝 NoteThis is the first PR with coverage protection enabled. The main branch doesn't have the Once this PR is merged, all future PRs will be compared against the main branch coverage and will be blocked if coverage decreases. Coverage protection will be fully active after this PR is merged. |
1 similar comment
ℹ️ Code Coverage Protection - First Time SetupStatus: ✅ PASSED (Initial Setup) PR Coverage
📝 NoteThis is the first PR with coverage protection enabled. The main branch doesn't have the Once this PR is merged, all future PRs will be compared against the main branch coverage and will be blocked if coverage decreases. Coverage protection will be fully active after this PR is merged. |
Changed check name from 'Coverage Diff' to 'Per-File Coverage Changes' to clearly distinguish it from the main coverage status check. Now we have two distinct checks: 1. 'Coverage: +X% 📈' - Overall coverage change (visible in PR header) 2. 'Per-File Coverage Changes' - File-by-file coverage annotations This makes it clear that they serve different purposes and reduces confusion when viewing the PR checks.
ℹ️ Code Coverage Protection - First Time SetupStatus: ✅ PASSED (Initial Setup) PR Coverage
📝 NoteThis is the first PR with coverage protection enabled. The main branch doesn't have the Once this PR is merged, all future PRs will be compared against the main branch coverage and will be blocked if coverage decreases. Coverage protection will be fully active after this PR is merged. |
1 similar comment
ℹ️ Code Coverage Protection - First Time SetupStatus: ✅ PASSED (Initial Setup) PR Coverage
📝 NoteThis is the first PR with coverage protection enabled. The main branch doesn't have the Once this PR is merged, all future PRs will be compared against the main branch coverage and will be blocked if coverage decreases. Coverage protection will be fully active after this PR is merged. |
|
🎉 This PR is included in version 4.2.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
Summary
This PR implements automated code coverage protection to prevent pull requests from being merged if they decrease the overall code coverage percentage.
Closes #26
🎯 What This Does
When a PR is created:
📊 Coverage Metrics Tracked
Policy: All four metrics must be maintained or improved (0% threshold).
🔧 Changes Made
CI/CD Workflow (
.github/workflows/ci.yml)coverage-checkjob that runs on all pull requestsfail_ci_if_error: trueConfiguration Updates
codecov.yml: Updated threshold from 1% to 0% (zero-tolerance policy)codecov.yml: Addedif_ci_failed: errorto fail CI on coverage dropsvitest.config.mts: Addedjson-summaryreporter for coverage comparisonDocumentation
New Files Created:
docs/COVERAGE_PROTECTION.md(300 lines).github/COVERAGE_QUICK_REFERENCE.md(200 lines)COVERAGE_PROTECTION_SETUP.md(300 lines)IMPLEMENTATION_SUMMARY.md(300 lines)Updated Files:
README.md: Added coverage protection section in Contributing areaCONTRIBUTING.md: Enhanced testing section with coverage requirements🎨 Example PR Comment
Developers will see automated comments like this:
✅ Passing Check
❌ Failing Check
🧪 Testing Instructions
For Reviewers
Review the workflow file (
.github/workflows/ci.yml)Review configuration changes
Review documentation
Testing the Coverage Check
To test this PR:
Local Testing
Developers can test coverage locally:
📚 Documentation for Contributors
Once merged, contributors can refer to:
.github/COVERAGE_QUICK_REFERENCE.mddocs/COVERAGE_PROTECTION.mdCONTRIBUTING.md(updated)README.md(updated)✨ Benefits
For the Project
For Maintainers
For Contributors
🔍 Configuration Summary
📋 Checklist
🚀 Next Steps After Merge
📖 Related Documentation
Note: This PR implements the coverage protection system. Once merged, all future PRs will be subject to the coverage requirements. This PR itself should pass all checks as it includes comprehensive documentation and doesn't decrease coverage.
Pull Request opened by Augment Code with guidance from the PR author