.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/benchmarks/interface/cifar100_odin.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_examples_benchmarks_interface_cifar100_odin.py>`
        to download the full example code

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_benchmarks_interface_cifar100_odin.py:


ODIN - CIFAR100
==================

Reproduces the ODIN benchmark for OOD detection, from the paper
*Enhancing the reliability of out-of-distribution image detection in neural networks*.

.. GENERATED FROM PYTHON SOURCE LINES 10-23

.. code-block:: Python
   :lineno-start: 10

    import pandas as pd  # additional dependency, used here for convenience
    import torch

    from pytorch_ood.benchmark import CIFAR100_ODIN
    from pytorch_ood.detector import ODIN, MaxSoftmax
    from pytorch_ood.model import WideResNet
    from pytorch_ood.utils import fix_random_seed

    fix_random_seed(123)

    device = "cuda:0"
    loader_kwargs = {"batch_size": 64}


.. GENERATED FROM PYTHON SOURCE LINES 24-28

.. code-block:: Python
   :lineno-start: 24

    model = WideResNet(num_classes=100, pretrained="cifar100-pt").eval().to(device)
    trans = WideResNet.transform_for("cifar100-pt")
    norm_std = WideResNet.norm_std_for("cifar100-pt")


.. GENERATED FROM PYTHON SOURCE LINES 29-34

.. code-block:: Python
   :lineno-start: 29

    detectors = {
        "MSP": MaxSoftmax(model),
        "ODIN": ODIN(model, eps=0.002, norm_std=norm_std),
    }


.. GENERATED FROM PYTHON SOURCE LINES 35-49

.. code-block:: Python
   :lineno-start: 35

    results = []
    benchmark = CIFAR100_ODIN(root="data", transform=trans)

    with torch.no_grad():
        for detector_name, detector in detectors.items():
            print(f"> Evaluating {detector_name}")
            res = benchmark.evaluate(detector, loader_kwargs=loader_kwargs, device=device)
            for r in res:
                r.update({"Detector": detector_name})
            results += res

    df = pd.DataFrame(results)
    print((df.set_index(["Dataset", "Detector"]) * 100).to_csv(float_format="%.2f"))


.. GENERATED FROM PYTHON SOURCE LINES 50-79

This produces a table with the following output:

+--------------------+----------+-------+---------+----------+----------+
| Dataset            | Detector | AUROC | AUPR-IN | AUPR-OUT | FPR95TPR |
+====================+==========+=======+=========+==========+==========+
| TinyImageNetCrop   | MSP      | 86.32 | 84.81   | 88.23    | 43.35    |
+--------------------+----------+-------+---------+----------+----------+
| TinyImageNetResize | MSP      | 74.64 | 70.91   | 77.29    | 65.52    |
+--------------------+----------+-------+---------+----------+----------+
| LSUNResize         | MSP      | 75.38 | 71.16   | 78.50    | 63.36    |
+--------------------+----------+-------+---------+----------+----------+
| LSUNCrop           | MSP      | 85.59 | 84.36   | 87.40    | 47.13    |
+--------------------+----------+-------+---------+----------+----------+
| Uniform            | MSP      | 77.92 | 16.86   | 97.60    | 40.44    |
+--------------------+----------+-------+---------+----------+----------+
| Gaussian           | MSP      | 84.78 | 23.12   | 98.41    | 30.22    |
+--------------------+----------+-------+---------+----------+----------+
| TinyImageNetCrop   | ODIN     | 86.89 | 84.01   | 89.02    | 40.54    |
+--------------------+----------+-------+---------+----------+----------+
| TinyImageNetResize | ODIN     | 80.79 | 78.44   | 82.08    | 60.10    |
+--------------------+----------+-------+---------+----------+----------+
| LSUNResize         | ODIN     | 81.25 | 78.04   | 83.04    | 58.13    |
+--------------------+----------+-------+---------+----------+----------+
| LSUNCrop           | ODIN     | 86.91 | 85.69   | 88.79    | 42.73    |
+--------------------+----------+-------+---------+----------+----------+
| Uniform            | ODIN     | 95.52 | 59.22   | 99.55    | 15.13    |
+--------------------+----------+-------+---------+----------+----------+
| Gaussian           | ODIN     | 98.57 | 85.14   | 99.86    | 5.76     |
+--------------------+----------+-------+---------+----------+----------+


.. _sphx_glr_download_auto_examples_benchmarks_interface_cifar100_odin.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: cifar100_odin.ipynb <cifar100_odin.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: cifar100_odin.py <cifar100_odin.py>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_