PyTorch Out-of-Distribution Detection¶

Modular, tested, and well-documented implementations of Out-of-Distribution (OOD) detection methods, with a unified interface, pre-trained models, datasets, and benchmarks. Also covers closely related fields such as Open-Set Recognition, Novelty Detection, Confidence Estimation and Anomaly Detection.

35+

detectors

15+

training objectives

45+

datasets

60+

pre-trained models

Quick Start¶

Install the package from PyPI:

pip install pytorch-ood

Every detector maps inputs to an outlier score that is larger for outliers, and out-of-distribution samples carry labels below zero. With those two conventions, evaluating a detector takes a few lines.

Evaluate a detector

Load a WideResNet-40 pre-trained on CIFAR-10 and score your data with the Energy-based detector:

from pytorch_ood.detector import EnergyBased
from pytorch_ood.model import load_model, load_transform
from pytorch_ood.metrics import OODMetrics

data_loader = ...  # your data, OOD samples with label < 0

model = load_model("wrn-40-2/cifar10/energy/s1").cuda()
preprocess = load_transform("wrn-40-2/cifar10/energy/s1")

detector = EnergyBased(model)
metrics = OODMetrics()

for x, y in data_loader:
    metrics.update(detector(preprocess(x).cuda()), y)

print(metrics.compute())

Run a benchmark

Compare detectors on the OpenOOD v1.5 CIFAR-10 benchmark. All datasets are downloaded automatically, and model outputs are computed once and shared by all detectors:

from pytorch_ood.benchmark import CIFAR10_OpenOOD
from pytorch_ood.detector import EnergyBased, MaxSoftmax
from pytorch_ood.model import load_model, load_transform

model = load_model("wrn-40-2/cifar10/crossentropy").to("cuda:0")
trans = load_transform("wrn-40-2/cifar10/crossentropy")

benchmark = CIFAR10_OpenOOD(root="data", transform=trans)
results = benchmark.evaluate(
    [MaxSoftmax(model), EnergyBased(model)], device="cuda:0", cache=True
)

for row in results:  # one row per detector and OOD dataset
    print(row)

Explore the Library¶

Detectors

Post-hoc OOD detectors: probability-, logit-, feature-, and gradient-based methods, plus activation pruning.

Detectors

Training Objectives

Losses that make models better at separating known from unknown inputs, supervised and unsupervised.

Training Objectives

Datasets

Image, text, and audio OOD datasets that download on first use, torchvision-style.

Datasets

Models

Architectures from the literature and a registry of pre-trained, checksum-verified weights.

Models

Benchmarks

Reproduce OpenOOD, OpenMIBOOD, SSB, and ODIN evaluations with one call, with cached representations.

Benchmarks

Utilities

Metrics, feature extraction, hyperparameter search, and transforms that tie the pieces together.

Utilities

Looking for complete scripts? Browse the examples, or use the index and module index to look up a name.

Citing PyTorch-OOD¶

If you use PyTorch-OOD in your research, please cite the paper:

@inproceedings{kirchheim2022pytorch,
  title={Pytorch-ood: A library for out-of-distribution detection based on pytorch},
  author={Kirchheim, Konstantin and Filax, Marco and Ortmeier, Frank},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={4351--4360},
  year={2022}
}