Utilities

class pytorch_ood.utils.TensorBuffer(device='cpu')

Used to buffer tensors

Parameters:

device – device used to store buffers. Default is cpu.

append(key, value: Tensor) → Self

Appends a tensor to the buffer.

Parameters:

• key – tensor identifier

• value – tensor

clear() → Self

Clears the buffer

get(key) → Tensor

Retrieves tensor from the buffer

Parameters:

key – tensor identifier

Returns:

concatenated tensor

is_empty() → bool

Returns true if this buffer does not hold any tensors.

keys() → KeysView

sample(key) → Tensor

Samples a random tensor from the buffer

Parameters:

key – tensor identifier

Returns:

random tensor

save(path) → Self

Save buffer to disk

Returns:

self

pytorch_ood.utils.calc_openness(n_train, n_test, n_target)

In Toward open set recognition the Openness \(\mathcal{O}\) of a problem was defined as:

\[\mathcal{O} = 1 - \sqrt{ \frac{2 \times n_{train}}{n_{test} \times n_{target}} }\]

where \(n\) is the number of classes, respectively.

Parameters:

• n_train – number of classes for training

• n_test – total number of classes used during testing

• n_target – number of classes for classification during testing

Returns:

Openness of the problem

See Paper:

IEEE Explore

pytorch_ood.utils.contains_known(labels) → bool | Tensor

Returns:

True if the labels contains any ID labels

pytorch_ood.utils.contains_known_and_unknown(labels) → bool | Tensor

Returns:

True if the labels contain ID and OOD classes

pytorch_ood.utils.contains_unknown(labels) → bool | Tensor

Returns:

True if the labels contains any OOD labels

pytorch_ood.utils.extract_features(data_loader: DataLoader, model: Callable[[Tensor], Tensor], device: str | None) → Tuple[Tensor, Tensor]

Helper to extract outputs from model. Ignores OOD inputs.

Parameters:

• data_loader – dataset to extract from

• model – neural network to pass inputs to

• device – device used for calculations

Returns:

Tuple with outputs and labels

pytorch_ood.utils.fix_random_seed(seed: int = 12345) → None

Set all random seeds.

Parameters:

seed – seed to set

pytorch_ood.utils.is_known(labels) → bool | Tensor

Returns:

True, if label \(\geq 0\)

pytorch_ood.utils.is_unknown(labels) → bool | Tensor

Returns:

True, if label \(< 0\)

Transformations

class pytorch_ood.utils.ToUnknown

Callable that returns a negative number, used in pipelines to mark specific datasets as OOD or unknown.

class pytorch_ood.utils.ToRGB

Convert Image to RGB, if it is not already.

class pytorch_ood.utils.TargetMapping(known: Set, unknown: Set)

Maps ID (a.k.a. known) classes to labels \(\in [0,n]\), and OOD (a.k.a. unknown) classes to labels in \([-\infty, -1]\). This is required for open set simulations.

Example: If we split up a dataset so that the classes 2,3,4,9 are considered known or ID, these class labels have to be remapped to 0,1,2,3 to be able to train using cross entropy with 1-of-K-vectors. All other classes have to be mapped to values \(<0\) to be marked as OOD.

Metrics

class pytorch_ood.utils.OODMetrics(device: str = 'cpu', mode: str = 'classification', void_label: int | None = None)

Calculates various metrics used in OOD detection experiments.

• AUROC (see ArXiv or ArXiv for more information)

• AUTC (see ArXiv for more information)

• AUPR ID (see ArXiv or ArXiv for more information)

• AUPR OUT (see ArXiv or ArXiv for more information)

• FPR@95TPR (see ArXiv for more information)

The interface is similar to torchmetrics.

metrics = OODMetrics()
outlier_scores = torch.Tensor([0.5, 1.0, -10])
labels = torch.Tensor([1,2,-1])
metrics.update(outlier_scores, labels)
metric_dict = metrics.compute()

In classification mode, the inputs will be flattened, so we treat each value as an individual example. Using this mode for segmentation tasks can require a lot of memory and compute.

In segmentation mode, the inputs will be flattened along the first (batch) dimension so that the shape is \(B \times D\) afterwards. The scores will then be calculated for each sample in the batch (i.e., over \(D\) values each), and the final score will be the mean over all \(B\) samples.

Parameters:

• device – where tensors should be stored

• mode – either classification or segmentation.

• void_label – label that will be ignored during score calculation

compute() → Dict[str, float]

Calculate metrics

Returns:

dictionary with different metrics

Raise:

ValueError if data does not contain ID and OOD points or buffer is empty

reset() → Self

Resets collected metrics

update(scores: Tensor, y: Tensor) → Self

Add batch of results to collection.

Parameters:

• scores – outlier score

• y – target label

pytorch_ood.utils.metrics.aurra(confidence: Tensor, correct: Tensor) → float

See Implementation:

GitHub

Parameters:

• confidence – predicted confidence values

• correct – ground truth

Returns:

score

pytorch_ood.utils.metrics.calibration_error(confidence: Tensor, correct: Tensor, p: str = '2', beta: int = 100) → float

See Implementation:

GitHub

Parameters:

• confidence – predicted confidence

• correct – ground truth

• p – p for norm. Can be one of 1, 2, or infty

• beta – target bin size

Returns:

calculated calibration error