Utilities

class pytorch_ood.utils.TensorBuffer(device='cpu')

Used to buffer tensors

Parameters:

device – device used to store buffers. Default is cpu.

append(key, value: Tensor) → Self

Appends a tensor to the buffer.

Parameters:

• key – tensor identifier

• value – tensor

clear() → Self

Clears the buffer

get(key) → Tensor

Retrieves tensor from the buffer

Parameters:

key – tensor identifier

Returns:

concatenated tensor

is_empty() → bool

Returns true if this buffer does not hold any tensors.

keys() → KeysView

sample(key) → Tensor

Samples a random tensor from the buffer

Parameters:

key – tensor identifier

Returns:

random tensor

save(path) → Self

Save buffer to disk

Returns:

self

pytorch_ood.utils.calc_openness(n_train, n_test, n_target)

In Toward open set recognition the Openness \(\mathcal{O}\) of a problem was defined as:

\[\mathcal{O} = 1 - \sqrt{ \frac{2 \times n_{train}}{n_{test} \times n_{target}} }\]

where \(n\) is the number of classes, respectively.

Parameters:

• n_train – number of classes for training

• n_test – total number of classes used during testing

• n_target – number of classes for classification during testing

Returns:

Openness of the problem

See Paper:

IEEE Explore

pytorch_ood.utils.contains_known(labels) → bool | Tensor

Returns:

true if the labels contains any IN labels

pytorch_ood.utils.contains_known_and_unknown(labels) → bool | Tensor

Returns:

true if the labels contain IN and OOD classes

pytorch_ood.utils.contains_unknown(labels) → bool | Tensor

Returns:

true if the labels contains any OOD labels

pytorch_ood.utils.extract_features(data_loader: DataLoader, model: Callable[[Tensor], Tensor], device: str | None) → Tuple[Tensor, Tensor]

Helper to extract outputs from model. Ignores OOD inputs.

Parameters:

• data_loader – dataset to extract from

• model – neural network to pass inputs to

• device – device used for calculations

Returns:

Tuple with outputs and labels

pytorch_ood.utils.fix_random_seed(seed: int = 12345) → None

Set all random seeds.

Parameters:

seed – seed to set

pytorch_ood.utils.is_known(labels) → bool | Tensor

Returns:

True, if label \(>= 0\)

pytorch_ood.utils.is_unknown(labels) → bool | Tensor

Returns:

True, if label \(< 0\)

Transformations

class pytorch_ood.utils.ToUnknown

Callable that returns a negative number, used in pipelines to mark specific datasets as OOD or unknown.

class pytorch_ood.utils.ToRGB

Convert Image to RGB, if it is not already.

class pytorch_ood.utils.TargetMapping(known: Set, unknown: Set)

Maps known classes to index in \([0,n]\), unknown classes to values in \([-\infty, -1]\). Required for open set simulations.

Example: If we split up a dataset so that the classes 2,3,4,9 are considered known or IN, these class labels have to be remapped to 0,1,2,3 to be able to train using cross entropy with 1-of-K-vectors. All other classes have to be mapped to values \(<0\) to be marked as OOD.

Target mappings have to be known at evaluation time.

class pytorch_ood.utils.InsertCOCO(coco_dir: str, p: float = 0.1, n: int = 1, exclude_classes: List[str] | str | None = None, annotation_per_image: int = 1, ood_mask_value: int = -1, upscale: float = 1.4150357439499515, year: int = 2017, min_img_size: int = 480, download: bool = False)

Transformation that inserts cropped COCO objects into images, marking the corresponding pixels of a segmentation mask as OOD.

The inserted objects can be used as synthetic OOD objects for supervised training of OOD detectors.

This was proposed in the paper Entropy Maximization and Meta Classification for Out-Of-Distribution Detection in Semantic Segmentation.

insert_coco = InsertCOCO(
    coco_dir="data/coco",
    exclude_classes=["train", "bicycle"],
    p=0.1
)

img, mask = insert_coco(img, mask)

See Paper:

ArXiv

Parameters:

• coco_dir – Directory to store the coco dataset

• p – Probability of inserting an OOD object to the image

• n – Number of inserted OOD objects per image

• exclude_classes – List of classes that should not be used for the OOD generation. Can also be one of bddAnomaly or Streethazards.

• annotation_per_image – Number of different annotation that are used for the ood object per coco image. (E.g. if there are 2 elephants on a COCO image, if this parameter is 1, only 1 elephant is inserted)

• ood_mask_value – Value of the OOD segmentation mask pixels

• upscale – Upscale factor for the OOD object

• year – Year of the coco dataset

• min_img_size – Minimum size of the used coco image

• download – Set True to automatically download the COCO dataset

__call__(img: Image, target: Tensor) → Tuple[Image, Tensor]

Check if OOD should be added and add it with the given probability

Parameters:

• img – input image

• target – segmentation mask for image

Returns:

Tuple with image and target tensor with inserted object(s)

Metrics

class pytorch_ood.utils.OODMetrics(device: str = 'cpu', mode: str = 'classification', void_label: int | None = None)

Calculates various metrics used in OOD detection experiments.

• AUROC

• AUPR IN

• AUPR OUT

• FPR@95TPR

The interface is similar to torchmetrics.

metrics = OODMetrics()
outlier_scores = torch.Tensor([0.5, 1.0, -10])
labels = torch.Tensor([1,2,-1])
metrics.update(outlier_scores, labels)
metric_dict = metrics.compute()

In classification mode, the inputs will be flattened, so we treat each value as an individual example. Using this mode for segmentation tasks can require a lot of memory and compute.

In segmentation mode, the inputs will be flattened along the first (batch) dimension so that the shape is \(B \times D\) afterwards. The scores will then be calculated for each sample in the batch (i.e., over \(D\) values each), and the final score will be the mean over all \(B\) samples.

Parameters:

• device – where tensors should be stored

• mode – either classification or segmentation.

• void_label – label that will be ignored during score calculation

compute() → Dict[str, float]

Calculate metrics

Returns:

dictionary with different metrics

Raise:

ValueError if data does not contain IN and OOD points or buffer is empty

reset() → Self

Resets collected metrics

update(scores: Tensor, y: Tensor) → Self

Add batch of results to collection.

Parameters:

• scores – outlier score

• y – target label

pytorch_ood.utils.metrics.aurra(confidence: Tensor, correct: Tensor) → float

See Implementation:

GitHub

Parameters:

• confidence – predicted confidence values

• correct – ground truth

Returns:

score

pytorch_ood.utils.metrics.calibration_error(confidence: Tensor, correct: Tensor, p: str = '2', beta: int = 100) → float

See Implementation:

GitHub

Parameters:

• confidence – predicted confidence

• correct – ground truth

• p – p for norm. Can be one of 1, 2, or infty

• beta – target bin size

Returns:

calculated calibration error