Standardize Task Cost Matrix

[PGijsbers]

There is no standard or reasonable input validation for cost matrices right now. This leads to bad matrices such as “yes”, “adam”, or “1”.

We need to define what a valid cost matrix looks like, and add input validation for new tasks with cost matrices.
We might want to convert or delete old tasks with cost matrices (depending on how interpretable they are).

This might be most convenient to process when working on task 2.0, but can be done independently.

[rohansen856]

Hi @PGijsbers i would like to solve the issue with the following approach:

Define Cost Matrix Validation Rules

As i undestand, the Valid Cost Matrix Format is:

• It Must be a JSON array representing a square matrix
• All values must be numeric (can be converted to float)
• Empty array [] is allowed (represents no cost matrix)
• For Example some valid formats are:
  • [] (no cost matrix)
  • [[0]] (1x1 matrix)
  • [[0, 1], [2, 0]] (2x2 matrix)

And in case of Invalid Formats:

• String values: "yes", "adam", "1", "true", "false"
• Non-JSON format: plain text values
• Non-square matrices: [[0, 1], [2]]
• Non-numeric values: [["a", "b"], ["c", "d"]]

To solve this, we need to implement the following components:

1. Schema Validation (src/schemas/tasks.py)

• CostMatrix Pydantic model with strict validation
• validate_cost_matrix_input() function for database values
• TaskInput model with input-specific validation

2. Runtime Validation (src/routers/openml/tasks.py)

• Added validation at task retrieval time
• Invalid cost matrices fallback to empty array []
• Graceful error handling to prevent API breaks

3. Validation Utilities (src/core/task_validation.py)

• validate_task_input() - extensible validation framework
• get_invalid_cost_matrices() - identify problematic entries
• clean_invalid_cost_matrices() - automated cleanup

Also, one solution for processing the already present invalid formats:

A little Database Migration for:

• Identifying all invalid cost matrix entries
• Replacing invalid values with empty arrays

Also implementing something like a --check flag (audit invalid cost matrices) or a --fix flag (automated cleanup of invalid entries) would be good in this case...

So, In short,

For Existing Invalid Data:

1. Run audit using --check or --fix
2. Apply migration to fix those data points after confirming

For New Data:

• Validation runs automatically on task retrieval
• Invalid values fallback to empty arrays
• API remains stable during transition

I would like to work using the above implementation, solving the issue. Thanks!