Detectors
This module provides a collection of different Out-of-Distribution Detectors.
API
Each detector implements a common API which contains a predict and a fit method, where fit is optional.
The objects __call__ methods is delegated to the the predict function, so you can use
detector = Detector(model)
detector.fit(data_loader)
scores = detector(x)
Feature-based Interface
Alternatively, you can also use the fit_features and predict_features methods.
In that case, inputs will not be passed through the model. This can help to avoid passing
data through the model multiple times when fitting several detectors. Detectors who do not
support this will raise an exception.
detector = Detector(model=None)
detector.fit_features(train_features, train_labels)
scores = detector.predict_features(test_features)
Some of the detectors support grid-like input, so that they can be used for anomaly segmentation without further adjustment.
- class pytorch_ood.api.Detector[source]
Abstract Base Class for an Out-of-Distribution Detector
- abstract fit(data_loader: DataLoader) Self[source]
Fit the detector to a dataset. Some methods require this.
- Parameters:
data_loader – dataset to fit on. This is usually the training dataset.
- Raises:
ModelNotSetException – if model was not set
- abstract fit_features(x: Tensor, y: Tensor) Self[source]
Fit the detector directly on features. Some methods require this.
- Parameters:
x – training features to use for fitting.
y – corresponding class labels.
Probability-based
Maximum Softmax (MSP)
- class pytorch_ood.detector.MaxSoftmax(model: Module, t: float | None = 1.0)[source]
Implements the Maximum Softmax Probability (MSP) Thresholding baseline for OOD detection.
Optionally, implements temperature scaling, which divides the logits by a constant temperature \(T\) before calculating the softmax. The score is calculated as:
\[- \max_y \sigma_y(f(x) / T)\]where \(\sigma\) is the softmax function and \(\sigma_y\) indicates the \(y^{th}\) value of the resulting probability vector.
- See Paper:
- See Implementation:
- Parameters:
model – neural network to use
t – temperature value \(T\). Default is 1.
Monte Carlo Dropout (MCD)
- class pytorch_ood.detector.MCD(model: Module, samples: int = 30, mode: str = 'var', batch_norm: bool = True)[source]
From the paper Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. Forward-propagates the input through the model \(N\) times with activated dropout and averages the results.
In
meanmode, the outlier score is calculated as\[- \max_y \frac{1}{N} \sum_n^{N} \sigma_y(f_n(x))\]where \(\sigma\) is the softmax function. In
varmode, the scores are calculated as\[\frac{1}{C} \sum_y^C \frac{1}{N} \sum_n^N ( \sigma_y(f_n(x)) - \mu_y )^2\]where \(C\) is the number of classes and \(\mu_y\) is the class mean. This is the mean over the per class variance, which was used in Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understanding.
Warning
This implementations puts the model into evaluation mode (except for variants of the BatchNorm Layers). This could also affect other modules.
- Parameters:
model – the module to use for the forward pass. Should output logits.
samples – number of iterations
mode – can be one of
varormeanbatch_norm – keep batch norm layers in evaluation mode
- n_samples
number \(N\) of samples
- static run(model: Module, x: Tensor, samples: int, batch_norm=True) Tuple[Tensor, Tensor][source]
- Parameters:
model – neural network
x – input
samples – number of rounds
batch_norm – keep batch norm layers in evaluation mode
- Returns:
mean and variance of softmax normalized model outputs
- static run_mean(model: Module, x: Tensor, samples: int, batch_norm=True) Tensor[source]
Assumes that the model outputs logits. More memory efficient implementation.
- Parameters:
model – neural network
x – input
samples – number of rounds
batch_norm – keep batch norm layers in evaluation mode
- Returns:
mean softmax output of the model
Temperature Scaling
- class pytorch_ood.detector.TemperatureScaling(model: Module)[source]
Implements temperature scaling from the paper On Calibration of Modern Neural Networks.
The method uses an additional set of validation samples to determine the optimal temperature value \(T\) to calibrate the softmax output.
The score is calculated as:
\[- \max_y \sigma_y(f(x) / T)\]where \(\sigma\) is the softmax function, \(T\) is the optimal temperature and \(\sigma_y\) indicates the \(y^{th}\) value of the resulting probability vector.
- See Paper:
- Parameters:
model – neural network to use
- fit(loader: DataLoader, device: str = 'cpu') Self[source]
Extracts features and optimizes the temperature.
- Parameters:
loader – data loader
device – device used for extracting logits
KL-Matching
- class pytorch_ood.detector.KLMatching(model: Module)[source]
Implements KL-Matching from the paper Scaling Out-of-Distribution Detection for Real-World Settings.
For each class, an typical posterior distribution \(d_y = \mathbb{E}_{x \sim \mathcal{X}_{val}}[p(y \vert x)]\) is estimated, where \(y\) is the class with the maximum posterior \(y = \arg\max_y p(y \vert x)\), as predicted by the model. Note that the method does not require class labels for the validation set. During evaluation, the KL-Divergence between the observed and the typical posterior \(D_{KL}[p(y \vert x) \Vert d_y]\) is used as outlier score.
This method can also be applied to multi-class settings.
- See Paper:
- Parameters:
model – neural network, is assumed to output logits.
- dists: ParameterDict
Typical posteriors per class
- fit(data_loader: DataLoader, device='cpu') Self[source]
Estimates typical distributions for each class. Ignores OOD samples.
- Parameters:
data_loader – validation data loader
device – device which should be used for calculations
- fit_features(logits: Tensor, labels: Tensor, device='cpu') Self[source]
Estimates typical distributions for each class. Ignores OOD samples.
- Parameters:
logits – logits
labels – class labels
device – device which should be used for calculations
Entropy
- class pytorch_ood.detector.Entropy(model: Module)[source]
Implements Entropy-based OOD detection.
This methods calculates the entropy based on the logits of a classifier. Higher entropy means more uniformly distributed posteriors, indicating larger uncertainty. Entropy is calculated as
\[H(x) = - \sum_i^C \sigma_i(f(x)) \log( \sigma_i(f(x)) )\]where \(\sigma_i\) indicates the \(i^{th}\) softmax value and \(C\) is the number of classes.
- Parameters:
model – the model \(f\)
Logit-based
Maximum Logit
- class pytorch_ood.detector.MaxLogit(model: Module)[source]
Implements the Max Logit Method for OOD Detection as proposed in Scaling Out-of-Distribution Detection for Real-World Settings.
\[- \max_y f_y(x)\]where \(f_y(x)\) indicates the \(y^{th}\) logits value predicted by \(f\).
- See Paper:
- Parameters:
t – temperature value T. Default is 1.
OpenMax
- class pytorch_ood.detector.OpenMax(model: Module, tailsize: int = 25, alpha: int = 10, euclid_weight: float = 1.0)[source]
Implementation of the OpenMax Layer as proposed in the paper Towards Open Set Deep Networks.
The methods determines a center \(\mu_y\) for each class in the logits space of a model, and then creates a statistical model of the distances of correct classified inputs. It uses extreme value theory to detect outliers by fitting a weibull function to the tail of the distance distribution.
We use the activation of the unknown class as outlier score.
Warning
This methods requires
libmrto be installed, which is broken at the moment. You can only use it by installingcythonandnumpy, andlibmrmanually afterward.- See Paper:
- See Implementation:
- Parameters:
model – neural network, assumed to output logits
tailsize – length of the tail to fit the distribution to
alpha – number of class activations to revise
euclid_weight – weight for the Euclidean distance.
- fit(data_loader: DataLoader, device: str | None = 'cpu') Self[source]
Determines parameters of the weibull functions for each class.
- Parameters:
data_loader – Data to use for fitting
device – Device used for calculations
- fit_features(logits: Tensor, y: Tensor) Self[source]
Determines parameters of the weibull functions for each class.
- Parameters:
logits – logits given by the model
y – class labels
- Returns:
Energy Based (EBO)
- class pytorch_ood.detector.EnergyBased(model: Module, t: float | None = 1.0)[source]
Implements the Energy Score of Energy-based Out-of-distribution Detection.
This methods calculates the negative energy for a vector of logits. This value can be used as outlier score.
\[E(x) = -T \log{\sum_i e^{f_i(x)/T}}\]where \(f_i(x)\) indicates the \(i^{th}\) logit value predicted by \(f\).
- See Paper:
- See Implementation:
- Parameters:
t – Temperature value \(T\). Default is 1.
- predict(x: Tensor) Tensor[source]
Calculate negative energy for inputs
- Parameters:
x – input tensor, will be passed through model
- Returns:
Energy score
- static score(logits: Tensor, t: float | None = 1.0) Tensor[source]
- Parameters:
logits – logits of input
t – temperature value
- t: float
Temperature
Weighted Energy Based (WEBO)
- class pytorch_ood.detector.WeightedEBO(model: Module, weights: Linear)[source]
Implements the Weighted Energy Based Score of VOS: Learning what you don’t know by virtual outlier synthesis.
This methods calculates the energy for a vector of logits and the weights (you only get from the training with
pytorch_ood.loss.VOSRegLoss). This value can be used as outlier score.\[E(x) = - \log{\sum_i w_{i} e^{f_i(x)}}\]where \(f_i(x)\) indicates the \(i^{th}\) logit value predicted by \(f\) and \(w\) indicates the weights.
Example Code:
weights = torch.nn.Linear(num_classes, 1)) detector = WeightedEBO(model, weights) scores = detector(images)
- See Paper:
- See Implementation:
- Parameters:
model – neural network to use, is assumed to output features
weights – neural network layer, with num_classes inputs.
ODIN Preprocessing
- class pytorch_ood.detector.ODIN(model: Module, criterion: Callable[[Tensor], Tensor] | None = None, eps: float = 0.05, temperature: float = 1000.0, norm_std: List[float] | None = None)[source]
Implements ODIN from the paper Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks.
ODIN is a preprocessing method for inputs that aims to increase the discriminability of the softmax outputs for IN and OOD data.
The operation requires two forward and one backward pass.
\[\hat{x} = x - \epsilon \ \text{sign}(\nabla_x \mathcal{L}(f(x) / T, \hat{y}))\]where \(\hat{y}\) is the predicted class of the network.
- See Paper:
- See Implementation:
- Parameters:
model – module to backpropagate through
criterion – loss function \(\mathcal{L}\) to use. If None is given, we will use negative log likelihood
eps – step size \(\epsilon\) of the gradient descent step
temperature – temperature \(T\) to use for scaling
norm_std – standard deviations used for normalization
- criterion
criterion \(\mathcal{L}\)
- eps
size \(\epsilon\) of the gradient step in the input space
- predict(x: Tensor) Tensor[source]
Calculates softmax outlier scores on ODIN pre-processed inputs.
- Parameters:
x – input tensor
- Returns:
outlier scores for each sample
- predict_features(x: Tensor) Tensor[source]
Since ODIN requires backpropagating through the model, this method can not be used.
- Raises:
Exception –
- temperature
temperature value \(T\)
- pytorch_ood.detector.odin_preprocessing(model: Module, x: Tensor, y: Tensor | None = None, criterion: Callable[[Tensor], Tensor] | None = None, eps: float = 0.05, temperature: float = 1000, norm_std: List[float] | None = None)[source]
Functional version of ODIN.
- Parameters:
model – module to backpropagate through
x – sample to preprocess
y – the label \(\hat{y}\) which is used to evaluate the loss. If none is given, the models prediction will be used
criterion – loss function \(\mathcal{L}\) to use. If none is given, we will use negative log likelihood
eps – step size \(\epsilon\) of the gradient ascend step
temperature – temperature \(T\) to use for scaling
norm_std – standard deviations used during preprocessing
Feature-based
Mahalanobis Distance (MD)
- class pytorch_ood.detector.Mahalanobis(model: Callable[[Tensor], Tensor], eps: float = 0.002, norm_std: List | None = None)[source]
Implements the Mahalanobis Method from the paper A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks.
This method calculates a class center \(\mu_y\) for each class, and a shared covariance matrix \(\Sigma\) from the data. The outlier scores are then calculated as
\[- \max_k \lbrace (f(x) - \mu_k)^{\top} \Sigma^{-1} (f(x) - \mu_k) \rbrace\]Also uses ODIN preprocessing.
- See Implementation:
- See Paper:
- Parameters:
model – the Neural Network, should output features
eps – magnitude for gradient based input preprocessing
norm_std – Standard deviations for input normalization
- cov: Tensor
Covariance Matrix
- eps: float
epsilon
- fit(data_loader: DataLoader, device: str | None = None) Self[source]
Fit parameters of the multi variate gaussian.
- Parameters:
data_loader – dataset to fit on.
device – device to use
- Returns:
- fit_features(z: Tensor, y: Tensor, device: str | None = None) Self[source]
Fit parameters of the multi variate gaussian.
- Parameters:
z – features
y – class labels
device – device to use
- Returns:
- mu: Tensor
Centers
- property n_classes
Number of classes the model is fitted for
- precision: Tensor
Precision Matrix
Multi-Layer Mahalanobis Distance (MD)
- class pytorch_ood.detector.MultiMahalanobis(model: List[Module], alpha: List[float] | None = None)[source]
Implements the Mahalanobis Method from the paper A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks which supports several layers.
For each of the given \(i\) layers, the method calculates a class center \(\mu_{iy}\) for each class, and a shared covariance matrix \(\Sigma_i\) from the data. The per-layer outlier scores are calculated as
\[M_i(x) = - \max_k \lbrace (f_i(x) - \mu_{ik})^{\top} \Sigma_i^{-1} (f_i(x) - \mu_{ik}) \rbrace\]The final outlier score is the sum of all scores, weighted by \(\alpha\).
Example code is provided here
Note
This does not yet support ODIN preprocessing. Also, the \(\alpha\) values have to be determined manually.
- See Implementation:
- See Paper:
- Parameters:
model – the neural network layers \(f_1(\cdot),...,f_n(\cdot)\), output of one will be used as input to the next.
alpha – weighting of the individual layers. Defaults to uniform weighting.
- alpha
Per-layer weighting factors
- cov: List[Tensor]
Covariance Matrices
- fit(data_loader: DataLoader, device: str | None = None) Self[source]
Fit one gaussian to the features of each layer. Will average over feature maps.
- Parameters:
data_loader – dataset to fit on.
device – device to use
- Returns:
- fit_features(zs: List[Tensor], y: Tensor, device: str | None = None) Self[source]
Fit parameters of the multi variate gaussians.
- Parameters:
zs – list of features for each layer
y – class labels
device – device to use
- Returns:
- mu: List[Tensor]
Centers
- property n_classes
Number of classes the model is fitted for
- precision: List[Tensor]
Precision Matrices
Relative Mahalanobis Distance (RMD)
- class pytorch_ood.detector.RMD(model: Callable[[Tensor], Tensor])[source]
Implements the Relative Mahalanobis Distance (RMD) from the paper A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection.
This method calculates a class center \(\mu_y\) for each class, and a shared covariance matrix \(\Sigma\) from the data.
Additionally, it fits a background gaussian with mean \(\mu_0\) and covariance matrix \(\Sigma_0\) to all of the features and calculates outlier scores as
\[\min_k \lbrace d_k(f(x)) - d_0(f(x)) \rbrace\]where \(d_k\) is the mahalanobis score for class \(k\) and \(d_0\) is the mahalanobis score under the background gaussian.
- See Paper:
- Parameters:
model – the Neural Network, should output features
- fit(loader: DataLoader, device: str = 'cpu') Self[source]
Fit parameters of the multi variate gaussian for the given loader. Ignores OOD Inputs.
Virtual Logit Matching (ViM)
- class pytorch_ood.detector.ViM(model: Callable[[Tensor], Tensor], d: int, w: Tensor, b: Tensor)[source]
Implements Virtual Logit Matching (ViM) from the paper ViM: Out-Of-Distribution with Virtual-logit Matching.
- See Paper:
- See Implementation:
- Parameters:
model – neural network to use, is assumed to output features
d – dimensionality of the principal subspace
w – weights \(W\) of the last layer of the network
b – biases \(b\) of the last layer of the network
- alpha: float
the computed \(\alpha\) value
- fit(data_loader, device='cpu') Self[source]
Extracts features and logits, computes principle subspace and alpha. Ignores OOD samples.
- Parameters:
data_loader – dataset to fit on
device – device to use
- Returns:
- fit_features(features: Tensor, labels: Tensor) Self[source]
Extracts features and logits, computes principle subspace and alpha. Ignores OOD samples.
- Parameters:
features – features
labels – class labels
- Returns:
Nearest Neighbor (kNN)
- class pytorch_ood.detector.KNN(model: Callable[[Tensor], Tensor], **knn_kwargs)[source]
Implements the detector from the paper Out-of-Distribution Detection with Deep Nearest Neighbors.
Fits a nearest neighbor model to the IN samples an uses the distance from the nearest neighbor as outlier score:
\[\min_{z \in \mathcal{D}} \lVert f(x) - f(z) \rVert_2\]where \(\mathcal{D}\) is the dataset used to train the nearest neighbor model.
The original paper found that using contrastive pre-training could increase the performance.
- See PMLR:
- Parameters:
model – neural network to use
knn_kwargs – dict with keyword arguments that will be passed to the scikit learns k-NN
- fit(loader: DataLoader, device: str = 'cpu') Self[source]
Extracts features and fits the kNN-Model
- Parameters:
loader – data loader
device – device used for extracting logits
Simplified Hopfield Energy (SHE)
- class pytorch_ood.detector.SHE(model: Callable[[Tensor], Tensor], head: Callable[[Tensor], Tensor])[source]
Implements Simplified Hopfield Energy from the paper Out-of-Distribution Detection based on In-Distribution Data Patterns Memorization with modern Hopfield Energy
For each class, SHE estimates the mean feature vector \(S_i\) of correctly classified instances. For some new instances with predicted class \(\hat{y}\), SHE then uses the inner product \(f(x)^{\top} S_{\hat{y}}\) as outlier score.
- See Paper:
- Parameters:
model – feature extractor
head – maps feature vectors to logits
- fit(loader: DataLoader, device: str = 'cpu') Self[source]
Extracts features and calculates mean patterns.
- Parameters:
loader – data to fit
device – device to use for computations
Activation Pruning
Activation Shaping (ASH)
- class pytorch_ood.detector.ASH(backbone: Callable[[Tensor], Tensor], head: Callable[[Tensor], Tensor], variant='ash-s', percentile: float = 0.65, detector: Callable[[Tensor], Tensor] | None = None)[source]
Implements ASH from the paper Extremely Simple Activation Shaping for Out-of-Distribution Detection.
ASH prunes the activations in some layer of the network (backbone) by removing a certain percentile of the highest activations. The remaining activations are modified, depending on the particular variant selected, and propagated through the remainder (head) of the network. Then uses the energy based outlier score. This approach has been shown to increase OOD detection rates while maintaining IN accuracy.
ASH-P: only prune, do not modify
ASH-B: binarize remaining activations
ASH-S: rescale remaining activations
The paper applies ASH after the last average pooling layer.
Example Code:
model = WideResNet() detector = ASH( backbone = model.features_before_pool, head = model.forward_from_before_pool, detector=EnergyBased.score ) scores = detector(images)
- See Paper:
- See Website:
- Parameters:
variant – one of
ash-p,ash-b,ash-sbackbone – first part of model to use, should output feature maps
head – second part of model used after applying ash, should output logits
percentile – amount of activations to modify
detector – detector that maps model outputs to outlier scores. Default is Energy based.
ReAct
- class pytorch_ood.detector.ReAct(backbone: Callable[[Tensor], Tensor], head: Callable[[Tensor], Tensor], threshold: float = 1.0, detector: Callable[[Tensor], Tensor] | None = None)[source]
Implements ReAct from the paper ReAct: Out-of-distribution Detection With Rectified Activations.
ReAct clips the activations in some layer of the network (backbone) and forward propagates the result through the remainder of the model (head). In the paper, ReAct is applied to the penultimate layer of the network.
The output of the network is then passed to an outlier detector that maps the output of the model to outlier scores.
Example Code:
model = WideResNet() detector = ReAct( backbone = model.features, head = model.fc, detector = EnergyBased.score ) scores = detector(images)
- See Paper:
- Parameters:
backbone – first part of model to use, should output feature maps
head – second part of model used after applying ash, should output logits
threshold – cutoff for activations
detector – detector that maps outputs to outlier scores. Default is energy based.
DICE
- class pytorch_ood.detector.DICE(model: Callable[[Tensor], Tensor], w: Tensor, b: Tensor, p: float, detector: Callable[[Tensor], Tensor] | None = None)[source]
Implements DICE from the paper DICE: Leveraging Sparsification for Out-of-Distribution Detection.
- See Paper:
- Parameters:
model – feature extractor
w – weights of last layer
b – bias of last layer
p – percentile of weights to drop
- fit(loader: DataLoader, device: str = 'cpu') Self[source]
- Parameters:
loader – data loader to extract features from. OOD inputs will be ignored.
device – device to use for feature extraction