.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/benchmarks/interface/cifar10_openood.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_examples_benchmarks_interface_cifar10_openood.py>`
        to download the full example code.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_benchmarks_interface_cifar10_openood.py:


OpenOOD - CIFAR10
==================

Reproduces the OpenOOD benchmark for OOD detection, using the WideResNet
model from the Hendrycks baseline paper.

.. warning :: This is currently incomplete, see :class:`CIFAR10-OpenOOD <pytorch_ood.benchmark.CIFAR10_OpenOOD>`.

.. GENERATED FROM PYTHON SOURCE LINES 12-26

.. code-block:: Python
   :lineno-start: 13


    import pandas as pd  # additional dependency, used here for convenience
    import torch

    from pytorch_ood.benchmark import CIFAR10_OpenOOD
    from pytorch_ood.detector import MaxSoftmax, ReAct, ASH
    from pytorch_ood.model import WideResNet
    from pytorch_ood.utils import fix_random_seed

    fix_random_seed(123)

    device = "cuda:0"
    loader_kwargs = {"batch_size": 64}


.. GENERATED FROM PYTHON SOURCE LINES 27-31

.. code-block:: Python
   :lineno-start: 27

    model = WideResNet(num_classes=10, pretrained="cifar10-pt").eval().to(device)
    trans = WideResNet.transform_for("cifar10-pt")
    norm_std = WideResNet.norm_std_for("cifar10-pt")


.. GENERATED FROM PYTHON SOURCE LINES 32-33

Just add more detectors here if you want to test more

.. GENERATED FROM PYTHON SOURCE LINES 33-37

.. code-block:: Python
   :lineno-start: 33

    detectors = {
        "MSP": MaxSoftmax(model),
    }


.. GENERATED FROM PYTHON SOURCE LINES 38-52

.. code-block:: Python
   :lineno-start: 38

    results = []
    benchmark = CIFAR10_OpenOOD(root="data", transform=trans)

    with torch.no_grad():
        for detector_name, detector in detectors.items():
            print(f"> Evaluating {detector_name}")
            res = benchmark.evaluate(detector, loader_kwargs=loader_kwargs, device=device)
            for r in res:
                r.update({"Detector": detector_name})
            results += res

    df = pd.DataFrame(results)
    print((df.set_index(["Dataset", "Detector"]) * 100).to_csv(float_format="%.2f"))


.. GENERATED FROM PYTHON SOURCE LINES 53-70

This should produce the following table:

+--------------+----------+-------+-------+---------+----------+----------+
| Dataset      | Detector | AUROC | AUTC  | AUPR-IN | AUPR-OUT | FPR95TPR |
+==============+==========+=======+=======+=========+==========+==========+
| CIFAR100     | MSP      | 87.82 | 40.69 | 88.42   | 85.20    | 43.09    |
+--------------+----------+-------+-------+---------+----------+----------+
| TinyImageNet | MSP      | 86.99 | 40.65 | 86.48   | 85.07    | 51.52    |
+--------------+----------+-------+-------+---------+----------+----------+
| MNIST        | MSP      | 92.66 | 37.23 | 94.33   | 90.30    | 22.46    |
+--------------+----------+-------+-------+---------+----------+----------+
| FashionMNIST | MSP      | 94.95 | 33.53 | 96.18   | 93.36    | 15.58    |
+--------------+----------+-------+-------+---------+----------+----------+
| Textures     | MSP      | 88.51 | 39.68 | 92.99   | 78.50    | 40.89    |
+--------------+----------+-------+-------+---------+----------+----------+
| Places365    | MSP      | 88.24 | 39.93 | 71.17   | 95.61    | 44.63    |
+--------------+----------+-------+-------+---------+----------+----------+


.. _sphx_glr_download_auto_examples_benchmarks_interface_cifar10_openood.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: cifar10_openood.ipynb <cifar10_openood.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: cifar10_openood.py <cifar10_openood.py>`

    .. container:: sphx-glr-download sphx-glr-download-zip

      :download:`Download zipped: cifar10_openood.zip <cifar10_openood.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_