Probmetrics: Classification metrics and post-hoc calibration

This package (PyTorch-based) currently contains

• classification metrics, especially also metrics for assessing the quality of probabilistic predictions, and
• post-hoc calibration methods, especially
  • a fast and accurate implementation of temperature scaling.
  • an implementation of structured matrix scaling (SMS), a regularized version of matrix scaling that outperforms other logistic-based calibration functions.

It accompanies our papers Rethinking Early Stopping: Refine, Then Calibrate and Structured Matrix Scaling for Multi-Class Calibration. Please cite us if you use this repository for research purposes. The experiments from the papers can be found here:

• Rethinking Early Stopping:
  • vision experiments.
  • tabular experiments.
  • theory.
• Structured Matrix Scaling: all experiments.

Installation

Probmetrics is available via

pip install probmetrics

To obtain all functionality, install probmetrics[extra,dev,dirichletcal].

• extra installs more packages for smooth ECE, Venn-Abers calibration, centered isotonic regression, the temperature scaling implementation in NetCal.
• dev installs more packages for development (esp. documentation)
• dirichletcal installs Dirichlet calibration, which however only works for Python 3.12 upwards.

Using post-hoc calibration methods

You can create a calibrator as follows:

from probmetrics.calibrators import get_calibrator

calib = get_calibrator('logistic')

These are the main supported methods:

• 'logistic' defaults to structured matrix scaling (SMS) for multiclass and affine scaling for binary calibration. We recommend using 'logistic' for best results, especially on multiclass problems. It can be slow for larger numbers of classes. Only runs on CPU. The first call is slower due to numba compilation.
• 'svs': Structured vector scaling (SVS) for multiclass problems, faster than SMS for multiclass while being almost as good in many cases.
• 'quadratic-scaling': Quadratic scaling for binary problems, outperforms 'logistic' (affine scaling) in our benchmarks.
• 'temp-scaling': Our highly efficient implementation of temperature scaling that, unlike some other implementations, does not suffer from optimization issues. Temperature scaling is not as expressive as matrix or vector scaling variants, but it is faster and has the least overfitting risk.
• 'ts-mix': Same as 'temp-scaling' but with Laplace smoothing (slightly preferable for logloss). Can also be achieved using get_calibrator('temp-scaling', calibrate_with_mixture=True)
• 'isotonic' Isotonic regression from scikit-learn. Isotonic variants can be good for binary classification with enough data (around 10K samples or more)
• 'ivap' Inductive Venn-Abers predictor (a version of isotonic regression, slow but a bit better)
• 'cir' Centered isotonic regression (slightly better and slower than isotonic)
• 'dircal' Dirichlet calibration (slow, logistic performs better in our experiments)
• 'dircal-cv' Dirichlet calibration optimized with cross-validation (very slow)

More details on parameters and other methods can be found in the get_calibrator function here.

Usage with numpy

import numpy as np

probas = np.asarray([[0.1, 0.9]])  # shape = (n_samples, n_classes)
labels = np.asarray([1])  # shape = (n_samples,)
calib.fit(probas, labels)
calibrated_probas = calib.predict_proba(probas)

Usage with PyTorch

The PyTorch version can be used directly with GPU tensors, which is leveraged by our temperature scaling implementation but not by most other methods. For temperature scaling, this could accelerate things, but the CPU version can be faster for smaller validation sets (around 1K-10K samples).

from probmetrics.distributions import CategoricalProbs
import torch

probas = torch.as_tensor([[0.1, 0.9]])
labels = torch.as_tensor([1])

# if you have logits, you can use CategoricalLogits instead
calib.fit_torch(CategoricalProbs(probas), labels)
result = calib.predict_proba_torch(CategoricalProbs(probas))
calibrated_probas = result.get_probs()

Using our refinement and calibration metrics

We provide estimators for refinement error (loss after post-hoc calibration) and calibration error (loss improvement through post-hoc calibration). They can be used as follows:

import torch
from probmetrics.metrics import Metrics

# compute multiple metrics at once 
# this is more efficient than computing them individually
metrics = Metrics.from_names(['logloss', 
                              'refinement_logloss_ts-mix_all', 
                              'calib-err_logloss_ts-mix_all'])
y_true = torch.tensor(...)
y_logits = torch.tensor(...)
results = metrics.compute_all_from_labels_logits(y_true, y_logits)
print(results['refinement_logloss_ts-mix_all'].item())

Using more metrics

In general, while some metrics can be flexibly configured using the corresponding classes, many metrics are available through their name. Here are some relevant classification metrics:

from probmetrics.metrics import Metrics

metrics = Metrics.from_names([
    'logloss',
    'brier',  # for binary, this is 2x the brier from sklearn
    'accuracy', 'class-error',
    'auroc-ovr', # one-vs-rest
    'auroc-ovo-sklearn', # one-vs-one (can be slow!)
    # calibration metrics
    'ece-15', 'rmsce-15', 'mce-15', 'smece'
    'refinement_logloss_ts-mix_all', 
    'calib-err_logloss_ts-mix_all',
    'refinement_brier_ts-mix_all', 
    'calib-err_brier_ts-mix_all'
])

The following function returns a list of all metric names:

from probmetrics.metrics import Metrics, MetricType
Metrics.get_available_names(metric_type=MetricType.CLASS)

While there are some classes for regression metrics, they are not implemented.

Contributors

• David Holzmüller
• Eugène Berta

Releases

• v1.0.0: New post-hoc calibrators like 'logistic' including structured matrix scaling (SMS), structured vector scaling (SVS), affine scaling, and quadratic scaling.
• v0.0.2:
  • Removed numpy<2.0 constraint
  • allow 1D vectors in CategoricalLogits / CategoricalProbs
  • add TorchCal temperature scaling
  • minor fixes in AutoGluon temperature scaling that shouldn't affect the performance in practice
• v0.0.1: Initial release