Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion conformance/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ Note that some type checkers may not run on some platforms. If a type checker fa

Different type checkers report errors in different ways (with different wording in error messages and different line numbers or character ranges for errors). This variation makes it difficult to fully automate test validation given that tests will want to check for both false positive and false negative type errors. Some level of manual inspection will therefore be needed to determine whether a type checker is fully conformant with all tests in any given test file. This "scoring" process is required only when the output of a test changes — e.g. when a new version of that type checker is released and the tests are rerun. We assume that the output of a type checker will be the same from one run to the next unless/until a new version is released that fixes or introduces a bug. In this case, the output will need to be manually inspected and the conformance results re-scored for those tests whose output has changed.

Conformance results are reported and summarized for each supported type checker. Currently, results are reported for mypy, pyrefly, pyright, and zuban. It is the goal and desire to add additional type checkers over time.
Conformance results are reported and summarized for each supported type checker. Currently, results are reported for mypy, pyrefly, pyright, zuban and ty. It is the goal and desire to add additional type checkers over time.

## Adding a New Test Case

Expand Down
1 change: 1 addition & 0 deletions conformance/requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,4 @@ mypy
pip
zuban
pyrefly
ty
2 changes: 1 addition & 1 deletion conformance/results/mypy/version.toml
Original file line number Diff line number Diff line change
@@ -1 +1 @@
version = "mypy 1.19.0"
version = "mypy 1.19.1"
4 changes: 2 additions & 2 deletions conformance/results/pyrefly/protocols_explicit.toml
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,14 @@ notes = """
Does not reject call to super method with no default implementation.
Does not detect stub methods inherited from protocols as abstract.
"""
conformance_automated = "Fail"
conformance_automated = "Pass"
errors_diff = """
Line 89: Expected 1 errors
"""
output = """
ERROR protocols_explicit.py:27:16-28: Method `draw` inherited from class `PColor` has no implementation and cannot be accessed via `super()` [missing-attribute]
ERROR protocols_explicit.py:56:20-36: `tuple[int, int, str]` is not assignable to attribute `rgb` with type `tuple[int, int, int]` [bad-assignment]
ERROR protocols_explicit.py:60:10-20: Cannot instantiate `Point` because the following members are abstract: `intensity`, `transparency` [bad-instantiation]
ERROR protocols_explicit.py:89:15-17: Cannot instantiate `Concrete1` because the following members are abstract: `cm1` [bad-instantiation]
ERROR protocols_explicit.py:134:15-17: Cannot instantiate `Concrete5` because the following members are abstract: `method1` [bad-instantiation]
ERROR protocols_explicit.py:164:17-19: Cannot instantiate `Concrete7A` because the following members are abstract: `method1` [bad-instantiation]
"""
2 changes: 1 addition & 1 deletion conformance/results/pyrefly/version.toml
Original file line number Diff line number Diff line change
@@ -1 +1 @@
version = "pyrefly 0.45.2"
version = "pyrefly 0.46.0"
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This auto-updated, please let me know if this should be reverted.

185 changes: 163 additions & 22 deletions conformance/results/results.html

Large diffs are not rendered by default.

28 changes: 28 additions & 0 deletions conformance/results/ty/aliases_explicit.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
conformance_automated = "Fail"
errors_diff = """
Line 79: Expected 1 errors
Line 80: Expected 1 errors
Line 81: Expected 1 errors
Line 82: Expected 1 errors
Line 83: Expected 1 errors
Line 84: Expected 1 errors
Line 85: Expected 1 errors
Line 86: Expected 1 errors
Line 87: Expected 1 errors
Line 88: Expected 1 errors
Line 89: Expected 1 errors
Line 90: Expected 1 errors
Line 91: Expected 1 errors
Line 100: Expected 1 errors
Line 57: Unexpected errors ['aliases_explicit.py:57:5 - major: type-assertion-failure: Argument does not have asserted type `(int, str, str, /) -> None` (type-assertion-failure)']
"""
output = """
aliases_explicit.py:57:5 - major: type-assertion-failure: Argument does not have asserted type `(int, str, str, /) -> None` (type-assertion-failure)
aliases_explicit.py:67:9 - major: non-subscriptable: Cannot subscript non-generic type (non-subscriptable)
aliases_explicit.py:68:9 - major: non-subscriptable: Cannot subscript non-generic type (non-subscriptable)
aliases_explicit.py:69:29 - major: invalid-type-arguments: Too many type arguments: expected 1, got 2 (invalid-type-arguments)
aliases_explicit.py:70:29 - major: invalid-type-arguments: Too many type arguments: expected 1, got 2 (invalid-type-arguments)
aliases_explicit.py:71:24 - major: invalid-type-arguments: Type argument for `ParamSpec` must be either a list of types, `ParamSpec`, `Concatenate`, or `...` (invalid-type-arguments)
aliases_explicit.py:101:6 - major: call-non-callable: Object of type `UnionType` is not callable (call-non-callable)
aliases_explicit.py:102:5 - major: non-subscriptable: Cannot subscript non-generic type (non-subscriptable)
"""
29 changes: 29 additions & 0 deletions conformance/results/ty/aliases_implicit.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
conformance_automated = "Fail"
errors_diff = """
Line 106: Expected 1 errors
Line 111: Expected 1 errors
Line 112: Expected 1 errors
Line 113: Expected 1 errors
Line 117: Expected 1 errors
Line 68: Unexpected errors ['aliases_implicit.py:68:5 - major: type-assertion-failure: Argument does not have asserted type `(int, str, str, /) -> None` (type-assertion-failure)']
"""
output = """
aliases_implicit.py:68:5 - major: type-assertion-failure: Argument does not have asserted type `(int, str, str, /) -> None` (type-assertion-failure)
aliases_implicit.py:76:9 - major: non-subscriptable: Cannot subscript non-generic type (non-subscriptable)
aliases_implicit.py:77:9 - major: non-subscriptable: Cannot subscript non-generic type (non-subscriptable)
aliases_implicit.py:78:29 - major: invalid-type-arguments: Too many type arguments: expected 1, got 2 (invalid-type-arguments)
aliases_implicit.py:79:29 - major: invalid-type-arguments: Too many type arguments: expected 1, got 2 (invalid-type-arguments)
aliases_implicit.py:80:24 - major: invalid-type-arguments: Type argument for `ParamSpec` must be either a list of types, `ParamSpec`, `Concatenate`, or `...` (invalid-type-arguments)
aliases_implicit.py:81:25 - major: invalid-type-arguments: Type `str` is not assignable to upper bound `int | float` of type variable `TFloat@GoodTypeAlias12` (invalid-type-arguments)
aliases_implicit.py:107:9 - major: invalid-type-form: Variable of type `list[Unknown | <class 'int'> | <class 'str'>]` is not allowed in a type expression (invalid-type-form)
aliases_implicit.py:108:9 - major: invalid-type-form: Variable of type `tuple[tuple[<class 'int'>, <class 'str'>]]` is not allowed in a type expression (invalid-type-form)
aliases_implicit.py:109:9 - major: invalid-type-form: Variable of type `list[<class 'int'> | Unknown]` is not allowed in a type expression (invalid-type-form)
aliases_implicit.py:110:9 - major: invalid-type-form: Variable of type `dict[Unknown | str, Unknown | str]` is not allowed in a type expression (invalid-type-form)
aliases_implicit.py:114:9 - major: invalid-type-form: Variable of type `Literal[3]` is not allowed in a type expression (invalid-type-form)
aliases_implicit.py:115:10 - major: invalid-type-form: Variable of type `Literal[True]` is not allowed in a type expression (invalid-type-form)
aliases_implicit.py:116:10 - major: invalid-type-form: Variable of type `Literal[1]` is not allowed in a type expression (invalid-type-form)
aliases_implicit.py:118:10 - major: invalid-type-form: Variable of type `Literal["int"]` is not allowed in a type expression (invalid-type-form)
aliases_implicit.py:119:10 - major: invalid-type-form: Variable of type `Literal["int | str"]` is not allowed in a type expression (invalid-type-form)
aliases_implicit.py:133:6 - major: call-non-callable: Object of type `UnionType` is not callable (call-non-callable)
aliases_implicit.py:135:5 - major: non-subscriptable: Cannot subscript non-generic type (non-subscriptable)
"""
19 changes: 19 additions & 0 deletions conformance/results/ty/aliases_newtype.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
conformance_automated = "Fail"
errors_diff = """
Line 50: Expected 1 errors
"""
output = """
aliases_newtype.py:11:8 - major: invalid-argument-type: Argument is incorrect (invalid-argument-type)
aliases_newtype.py:12:14 - major: invalid-assignment: Object of type `Literal[42]` is not assignable to `UserId` (invalid-assignment)
aliases_newtype.py:18:11 - major: invalid-assignment: Object of type `<NewType pseudo-class 'UserId'>` is not assignable to `type` (invalid-assignment)
aliases_newtype.py:23:16 - major: invalid-argument-type: Argument to function `isinstance` is incorrect (invalid-argument-type)
aliases_newtype.py:26:21 - major: invalid-base: Cannot subclass an instance of NewType (invalid-base)
aliases_newtype.py:35:1 - major: invalid-newtype: The name of a `NewType` (`BadName`) must match the name of the variable it is assigned to (`GoodName`) (invalid-newtype)
aliases_newtype.py:41:6 - major: invalid-type-form: `GoodNewType1` is a `NewType` and cannot be specialized (invalid-type-form)
aliases_newtype.py:47:38 - major: invalid-newtype: invalid base for `typing.NewType` (invalid-newtype)
aliases_newtype.py:52:38 - major: invalid-newtype: invalid base for `typing.NewType` (invalid-newtype)
aliases_newtype.py:54:38 - major: invalid-newtype: invalid base for `typing.NewType` (invalid-newtype)
aliases_newtype.py:61:38 - major: invalid-newtype: invalid base for `typing.NewType` (invalid-newtype)
aliases_newtype.py:63:15 - major: invalid-newtype: Wrong number of arguments in `NewType` creation, expected 2, found 3 (invalid-newtype)
aliases_newtype.py:65:38 - major: invalid-newtype: invalid base for `typing.NewType` (invalid-newtype)
"""
16 changes: 16 additions & 0 deletions conformance/results/ty/aliases_recursive.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
conformance_automated = "Fail"
errors_diff = """
Line 19: Expected 1 errors
Line 20: Expected 1 errors
Line 38: Expected 1 errors
Line 39: Expected 1 errors
Line 50: Expected 1 errors
Line 51: Expected 1 errors
Line 52: Expected 1 errors
Line 63: Expected 1 errors
Line 69: Expected 1 errors
Line 72: Expected 1 errors
Line 75: Expected 1 errors
"""
output = """
"""
37 changes: 37 additions & 0 deletions conformance/results/ty/aliases_type_statement.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
conformance_automated = "Fail"
errors_diff = """
Line 56: Expected 1 errors
Line 62: Expected 1 errors
Line 67: Expected 1 errors
Line 84: Expected 1 errors
Lines 51, 52: Expected error (tag 'TA14')
Line 10: Unexpected errors ['aliases_type_statement.py:10:52 - major: invalid-type-arguments: Too many type arguments: expected 2, got 3 (invalid-type-arguments)']
"""
output = """
aliases_type_statement.py:10:52 - major: invalid-type-arguments: Too many type arguments: expected 2, got 3 (invalid-type-arguments)
aliases_type_statement.py:17:1 - major: unresolved-attribute: Object of type `TypeAliasType` has no attribute `bit_count` (unresolved-attribute)
aliases_type_statement.py:19:1 - major: call-non-callable: Object of type `TypeAliasType` is not callable (call-non-callable)
aliases_type_statement.py:23:7 - major: unresolved-attribute: Object of type `TypeAliasType` has no attribute `other_attrib` (unresolved-attribute)
aliases_type_statement.py:26:18 - major: invalid-base: Invalid class base with type `TypeAliasType` (invalid-base)
aliases_type_statement.py:31:22 - major: invalid-argument-type: Argument to function `isinstance` is incorrect (invalid-argument-type)
aliases_type_statement.py:37:22 - major: invalid-type-form: Function calls are not allowed in type expressions (invalid-type-form)
aliases_type_statement.py:38:22 - major: invalid-type-form: List literals are not allowed in this context in a type expression (invalid-type-form)
aliases_type_statement.py:39:22 - major: invalid-type-form: Tuple literals are not allowed in this context in a type expression (invalid-type-form)
aliases_type_statement.py:39:23 - major: invalid-type-form: Tuple literals are not allowed in this context in a type expression (invalid-type-form)
aliases_type_statement.py:40:22 - major: invalid-type-form: List comprehensions are not allowed in type expressions (invalid-type-form)
aliases_type_statement.py:41:22 - major: invalid-type-form: Dict literals are not allowed in type expressions (invalid-type-form)
aliases_type_statement.py:42:22 - major: invalid-type-form: Function calls are not allowed in type expressions (invalid-type-form)
aliases_type_statement.py:43:22 - major: invalid-type-form: Invalid subscript of object of type `list[Unknown | <class 'int'>]` in type expression (invalid-type-form)
aliases_type_statement.py:43:28 - major: invalid-type-form: Int literals are not allowed in this context in a type expression (invalid-type-form)
aliases_type_statement.py:44:22 - major: invalid-type-form: `if` expressions are not allowed in type expressions (invalid-type-form)
aliases_type_statement.py:45:22 - major: invalid-type-form: Variable of type `Literal[1]` is not allowed in a type expression (invalid-type-form)
aliases_type_statement.py:46:23 - major: invalid-type-form: Boolean literals are not allowed in this context in a type expression (invalid-type-form)
aliases_type_statement.py:47:23 - major: invalid-type-form: Int literals are not allowed in this context in a type expression (invalid-type-form)
aliases_type_statement.py:48:23 - major: invalid-type-form: Boolean operations are not allowed in type expressions (invalid-type-form)
aliases_type_statement.py:49:23 - major: fstring-type-annotation: Type expressions cannot use f-strings (fstring-type-annotation)
aliases_type_statement.py:77:27 - major: invalid-type-arguments: Type `str` is not assignable to upper bound `int` of type variable `S@RecursiveTypeAlias2` (invalid-type-arguments)
aliases_type_statement.py:79:32 - major: invalid-type-arguments: Type `int` is not assignable to upper bound `str` of type variable `T@RecursiveTypeAlias2` (invalid-type-arguments)
aliases_type_statement.py:82:1 - major: cyclic-type-alias-definition: Cyclic definition of `RecursiveTypeAlias3` (cyclic-type-alias-definition)
aliases_type_statement.py:88:1 - major: cyclic-type-alias-definition: Cyclic definition of `RecursiveTypeAlias6` (cyclic-type-alias-definition)
aliases_type_statement.py:89:1 - major: cyclic-type-alias-definition: Cyclic definition of `RecursiveTypeAlias7` (cyclic-type-alias-definition)
"""
31 changes: 31 additions & 0 deletions conformance/results/ty/aliases_typealiastype.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
conformance_automated = "Fail"
errors_diff = """
Line 40: Expected 1 errors
Line 43: Expected 1 errors
Line 44: Expected 1 errors
Line 45: Expected 1 errors
Line 46: Expected 1 errors
Line 47: Expected 1 errors
Line 48: Expected 1 errors
Line 23: Unexpected errors ['aliases_typealiastype.py:23:5 - major: invalid-argument-type: Argument to class `TypeAliasType` is incorrect (invalid-argument-type)']
"""
output = """
aliases_typealiastype.py:23:5 - major: invalid-argument-type: Argument to class `TypeAliasType` is incorrect (invalid-argument-type)
aliases_typealiastype.py:32:7 - major: unresolved-attribute: Object of type `TypeAliasType` has no attribute `other_attrib` (unresolved-attribute)
aliases_typealiastype.py:52:40 - major: invalid-type-form: Function calls are not allowed in type expressions (invalid-type-form)
aliases_typealiastype.py:53:40 - major: invalid-type-form: List literals are not allowed in this context in a type expression (invalid-type-form)
aliases_typealiastype.py:54:42 - major: invalid-type-form: Tuple literals are not allowed in this context in a type expression (invalid-type-form)
aliases_typealiastype.py:54:43 - major: invalid-type-form: Tuple literals are not allowed in this context in a type expression (invalid-type-form)
aliases_typealiastype.py:55:42 - major: invalid-type-form: List comprehensions are not allowed in type expressions (invalid-type-form)
aliases_typealiastype.py:56:42 - major: invalid-type-form: Dict literals are not allowed in type expressions (invalid-type-form)
aliases_typealiastype.py:57:42 - major: invalid-type-form: Function calls are not allowed in type expressions (invalid-type-form)
aliases_typealiastype.py:58:42 - major: invalid-type-form: Invalid subscript of object of type `list[Unknown | <class 'int'>]` in type expression (invalid-type-form)
aliases_typealiastype.py:58:48 - major: invalid-type-form: Int literals are not allowed in this context in a type expression (invalid-type-form)
aliases_typealiastype.py:59:42 - major: invalid-type-form: `if` expressions are not allowed in type expressions (invalid-type-form)
aliases_typealiastype.py:60:42 - major: invalid-type-form: Variable of type `Literal[3]` is not allowed in a type expression (invalid-type-form)
aliases_typealiastype.py:61:42 - major: invalid-type-form: Boolean literals are not allowed in this context in a type expression (invalid-type-form)
aliases_typealiastype.py:62:42 - major: invalid-type-form: Int literals are not allowed in this context in a type expression (invalid-type-form)
aliases_typealiastype.py:63:42 - major: invalid-type-form: Boolean operations are not allowed in type expressions (invalid-type-form)
aliases_typealiastype.py:64:42 - major: invalid-type-form: F-strings are not allowed in type expressions (invalid-type-form)
aliases_typealiastype.py:66:47 - major: unresolved-reference: Name `BadAlias21` used when not defined (unresolved-reference)
"""
9 changes: 9 additions & 0 deletions conformance/results/ty/aliases_variance.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
conformance_automated = "Fail"
errors_diff = """
Line 24: Expected 1 errors
Line 28: Expected 1 errors
Line 32: Expected 1 errors
Line 44: Expected 1 errors
"""
output = """
"""
5 changes: 5 additions & 0 deletions conformance/results/ty/annotations_coroutines.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
conformance_automated = "Pass"
errors_diff = """
"""
output = """
"""
35 changes: 35 additions & 0 deletions conformance/results/ty/annotations_forward_refs.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
conformance_automated = "Fail"
errors_diff = """
Line 24: Expected 1 errors
Line 25: Expected 1 errors
Line 41: Expected 1 errors
Line 42: Expected 1 errors
Line 43: Expected 1 errors
Line 44: Expected 1 errors
Line 45: Expected 1 errors
Line 46: Expected 1 errors
Line 48: Expected 1 errors
Line 50: Expected 1 errors
Line 51: Expected 1 errors
Line 52: Expected 1 errors
Line 53: Expected 1 errors
Line 82: Unexpected errors ['annotations_forward_refs.py:82:11 - major: invalid-type-form: Variable of type `Literal[""]` is not allowed in a type expression (invalid-type-form)']
Line 87: Unexpected errors ['annotations_forward_refs.py:87:9 - major: invalid-type-form: Variable of type `def int(self) -> None` is not allowed in a type expression (invalid-type-form)']
Line 95: Unexpected errors ['annotations_forward_refs.py:95:1 - major: type-assertion-failure: Argument does not have asserted type `str` (type-assertion-failure)']
Line 96: Unexpected errors ['annotations_forward_refs.py:96:1 - major: type-assertion-failure: Argument does not have asserted type `int` (type-assertion-failure)']
"""
output = """
annotations_forward_refs.py:22:7 - major: unresolved-reference: Name `ClassA` used when not defined (unresolved-reference)
annotations_forward_refs.py:23:12 - major: unresolved-reference: Name `ClassA` used when not defined (unresolved-reference)
annotations_forward_refs.py:47:10 - major: invalid-type-form: Invalid subscript of object of type `list[Unknown | <class 'int'>]` in type expression (invalid-type-form)
annotations_forward_refs.py:49:10 - major: invalid-type-form: Variable of type `Literal[1]` is not allowed in a type expression (invalid-type-form)
annotations_forward_refs.py:54:11 - major: fstring-type-annotation: Type expressions cannot use f-strings (fstring-type-annotation)
annotations_forward_refs.py:55:11 - major: invalid-type-form: Module `types` is not valid in a type expression (invalid-type-form)
annotations_forward_refs.py:66:26 - major: unresolved-reference: Name `ClassB` used when not defined (unresolved-reference)
annotations_forward_refs.py:80:14 - major: unresolved-reference: Name `ClassF` used when not defined (unresolved-reference)
annotations_forward_refs.py:82:11 - major: invalid-type-form: Variable of type `Literal[""]` is not allowed in a type expression (invalid-type-form)
annotations_forward_refs.py:87:9 - major: invalid-type-form: Variable of type `def int(self) -> None` is not allowed in a type expression (invalid-type-form)
annotations_forward_refs.py:89:8 - major: invalid-type-form: Variable of type `def int(self) -> None` is not allowed in a type expression (invalid-type-form)
annotations_forward_refs.py:95:1 - major: type-assertion-failure: Argument does not have asserted type `str` (type-assertion-failure)
annotations_forward_refs.py:96:1 - major: type-assertion-failure: Argument does not have asserted type `int` (type-assertion-failure)
"""
Loading