.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/benchmarks/interface/cifar10_odin.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_examples_benchmarks_interface_cifar10_odin.py>`
        to download the full example code.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_benchmarks_interface_cifar10_odin.py:


ODIN - CIFAR10
==================

Reproduces the ODIN benchmark for OOD detection, from the paper
*Enhancing the reliability of out-of-distribution image detection in neural networks*.

.. GENERATED FROM PYTHON SOURCE LINES 11-25

.. code-block:: Python
   :lineno-start: 12


    import pandas as pd  # additional dependency, used here for convenience
    import torch

    from pytorch_ood.benchmark import CIFAR10_ODIN
    from pytorch_ood.detector import ODIN, MaxSoftmax
    from pytorch_ood.model import WideResNet
    from pytorch_ood.utils import fix_random_seed

    fix_random_seed(123)

    device = "cuda:0"
    loader_kwargs = {"batch_size": 64, "num_workers": 12}


.. GENERATED FROM PYTHON SOURCE LINES 26-30

.. code-block:: Python
   :lineno-start: 26

    model = WideResNet(num_classes=10, pretrained="cifar10-pt").eval().to(device)
    trans = WideResNet.transform_for("cifar10-pt")
    norm_std = WideResNet.norm_std_for("cifar10-pt")


.. GENERATED FROM PYTHON SOURCE LINES 31-36

.. code-block:: Python
   :lineno-start: 31

    detectors = {
        "MSP": MaxSoftmax(model),
        "ODIN": ODIN(model, eps=0.002, norm_std=norm_std),
    }


.. GENERATED FROM PYTHON SOURCE LINES 37-51

.. code-block:: Python
   :lineno-start: 37

    results = []
    benchmark = CIFAR10_ODIN(root="data", transform=trans)

    with torch.no_grad():
        for detector_name, detector in detectors.items():
            print(f"> Evaluating {detector_name}")
            res = benchmark.evaluate(detector, loader_kwargs=loader_kwargs, device=device)
            for r in res:
                r.update({"Detector": detector_name})
            results += res

    df = pd.DataFrame(results)
    print((df.set_index(["Dataset", "Detector"]) * 100).to_csv(float_format="%.2f"))


.. GENERATED FROM PYTHON SOURCE LINES 52-81

This produces a table with the following output:

+--------------------+----------+-------+---------+----------+---------+----------+
| Dataset            | Detector | AUROC | AUTC    | AUPR-IN | AUPR-OUT | FPR95TPR |
+====================+==========+=======+=========+=========+==========+==========+
| TinyImageNetCrop   | MSP      | 94.59 | 33.99   | 95.77   | 93.10    | 17.18    |
+--------------------+----------+-------+---------+---------+----------+----------+
| TinyImageNetResize | MSP      | 88.22 | 40.12   | 89.24   | 86.00    | 42.50    |
+--------------------+----------+-------+---------+---------+----------+----------+
| LSUNResize         | MSP      | 91.45 | 37.91   | 92.64   | 89.46    | 29.06    |
+--------------------+----------+-------+---------+---------+----------+----------+
| LSUNCrop           | MSP      | 96.49 | 29.13   | 97.20   | 95.69    | 12.49    |
+--------------------+----------+-------+---------+---------+----------+----------+
| UniformNoise       | MSP      | 86.85 | 43.51   | 98.54   | 30.50    | 38.42    |
+--------------------+----------+-------+---------+---------+----------+----------+
| GaussianNoise      | MSP      | 90.29 | 41.32   | 98.99   | 36.27    | 25.69    |
+--------------------+----------+-------+---------+---------+----------+----------+
| TinyImageNetCrop   | ODIN     | 96.78 | 47.37   | 97.10   | 96.46    | 14.16    |
+--------------------+----------+-------+---------+---------+----------+----------+
| TinyImageNetResize | ODIN     | 91.44 | 48.24   | 91.45   | 91.31    | 38.84    |
+--------------------+----------+-------+---------+---------+----------+----------+
| LSUNResize         | ODIN     | 94.66 | 47.73   | 94.80   | 94.48    | 26.27    |
+--------------------+----------+-------+---------+---------+----------+----------+
| LSUNCrop           | ODIN     | 98.10 | 40.89   | 98.16   | 98.11    | 9.39     |
+--------------------+----------+-------+---------+---------+----------+----------+
| UniformNoise       | ODIN     | 95.11 | 46.46   | 99.46   | 71.50    | 21.46    |
+--------------------+----------+-------+---------+---------+----------+----------+
| GaussianNoise      | ODIN     | 97.68 | 44.19   | 99.76   | 82.50    | 11.02    |
+--------------------+----------+-------+---------+---------+----------+----------+


.. _sphx_glr_download_auto_examples_benchmarks_interface_cifar10_odin.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: cifar10_odin.ipynb <cifar10_odin.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: cifar10_odin.py <cifar10_odin.py>`

    .. container:: sphx-glr-download sphx-glr-download-zip

      :download:`Download zipped: cifar10_odin.zip <cifar10_odin.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_