-
Notifications
You must be signed in to change notification settings - Fork 201
Open
Description
Bug
The test_batched.py script incorrectly reports test runs as "passed" when the number of failed tests ends in 0 (10, 20, 30, 40, etc.).
Root Cause
The current logic in policyengine_us/tests/test_batched.py (line 264):
if "failed" not in line or "0 failed" in line:
test_passed = TrueWhen pytest outputs "40 failed, 8261 passed...", the substring "0 failed" is found within "40 failed", causing the condition to evaluate to True.
Impact
- PR Implement Arizona TANF (cash assistance) program #7150 shows "Full Suite - States" as passing despite 40 test failures
- Any PR with 10, 20, 30, 40, 50... failures will incorrectly pass CI
Proposed Fix
Parse the actual failure count instead of substring matching:
import re
match = re.search(r'(\d+) failed', line)
if match:
failed_count = int(match.group(1))
test_passed = (failed_count == 0)
else:
# No "X failed" in line means all passed
test_passed = TrueTest Cases
| Pytest Output | Current (buggy) | Fixed |
|---|---|---|
0 failed, 100 passed |
pass | pass |
1 failed, 99 passed |
fail | fail |
10 failed, 90 passed |
pass | fail |
40 failed, 8261 passed |
pass | fail |
100 passed (no failures) |
pass | pass |
Metadata
Metadata
Assignees
Labels
No labels