Skip to content

test_batched.py incorrectly marks tests as passed when failure count ends in 0 #7152

@hua7450

Description

@hua7450

Bug

The test_batched.py script incorrectly reports test runs as "passed" when the number of failed tests ends in 0 (10, 20, 30, 40, etc.).

Root Cause

The current logic in policyengine_us/tests/test_batched.py (line 264):

if "failed" not in line or "0 failed" in line:
    test_passed = True

When pytest outputs "40 failed, 8261 passed...", the substring "0 failed" is found within "40 failed", causing the condition to evaluate to True.

Impact

Proposed Fix

Parse the actual failure count instead of substring matching:

import re

match = re.search(r'(\d+) failed', line)
if match:
    failed_count = int(match.group(1))
    test_passed = (failed_count == 0)
else:
    # No "X failed" in line means all passed
    test_passed = True

Test Cases

Pytest Output Current (buggy) Fixed
0 failed, 100 passed pass pass
1 failed, 99 passed fail fail
10 failed, 90 passed pass fail
40 failed, 8261 passed pass fail
100 passed (no failures) pass pass

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions