.. _tutorial_gradcam_fnf: Tutorial: Doing Grad-CAM on the Food-Non-Food (FNF) model ========================================================= This is a tutorial on how to perform Grad-CAM on the FNF model using the ``XAI.FNFGradCAM()`` class that is provided in this library. **Summary of the recommended workflow:** 1. Instantiate the desired FNF model and load its desired pretrained weights. 2. Create a PyTorch dataset using the ``XAI.utils.datasets.DatasetFromCSV()`` class. 3. Instantiate the ``XAI.FNFGradCAM()`` object. 4. Do the Grad-CAM algorithm on the dataset created in Step 2 using the ``do_gradcam()`` function. 5. If necessary, plot and save the Grad-CAM results using the ``plot_gradcam()`` function. **There will be 2 ways that you can perform thm Grad-CAM on the FNF model, both of which will be covered in this tutorial:** 1. :ref:`tutorial_gradcam_fnf_method1` 2. :ref:`tutorial_gradcam_fnf_method2` To begin, activate the virtual environment that has the ``XAI`` library installed and import the library. .. code:: python3 :number-lines: import XAI ----- .. _tutorial_gradcam_fnf_method1: Method 1: Using the imported ``XAI`` library -------------------------------------------- .. _tutorial_gradcam_fnf_method1.1: Step 1: Load the desired PyTorch model ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This tutorial assumes that this step is already completed and that the desired PyTorch model is loaded into the ``model`` variable. Refer to this tutorial if necessary. .. _tutorial_gradcam_fnf_method1.2: Step 2: Creating the dataset ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This tutorial assumes that this step is already completed and that the dataset of images is created as the ``ds`` variable. Refer to :ref:`tutorial_dataset_from_csv` for a walkthrough on how to create the dataset. The dataset ``ds`` used in this tutorial contains *10 images*. Step 3: Instantiate ``FNFGradCAM()`` class object ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Instantiate the ``XAI.FNFGradCAM()`` class by doing the following. Here, the target layer used in the Grad-CAM computation is ``conv2d_7b``. .. code:: python3 :number-lines: fnf_gc = XAI.FNFGradCam( model=model, target_layer_name="conv2d_7b", ) The ``fnf_gc`` object has the following 3 attributes: >>> fnf_gc.device device(type='cuda', index=0) >>> fnf_gc.target_layer_name 'conv2d_7b' >>> fnf_gc.model Step 4: Do the Grad-CAM using the ``do_gradcam()`` function ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This function does the following 3 observable things: 1. Computes the Grad-CAM attribution map for each image and saves each map as a numpy array (``.npy`` file) in the specified output folder (``output_folder``). 2. Returns a Pandas DataFrame containing the results of performing Grad-CAM and saves this DataFrame as ``gradcam_metadata.csv`` in ``output_folder``. 3. Plots and saves the Grad-CAM numpy arrays as ``.png`` files if ``plot_gradcam=True``. The following folder structure will be created in ``output_folder`` to save the 3 outputs listed above: .. _do_gradcam_folder_structure: .. code:: text output_folder/ ├── gradcam_metadata.csv │ ├── gradcam_numpy_arrays/ │   ├── 164c728e-86d2-11eb-b774-06d7ab6752a4.npy │   ├── 164c728f-86d2-11eb-b774-06d7ab6752a4.npy │   ├── ... │   └── 164c72a5-86d2-11eb-b774-06d7ab6752a4.npy │ └── gradcam_plots/ <--- This folder will be created if `plot_gradcam=True` ├── 164c728e-86d2-11eb-b774-06d7ab6752a4.png ├── 164c728f-86d2-11eb-b774-06d7ab6752a4.png ├── ... └── 164c72a5-86d2-11eb-b774-06d7ab6752a4.png Note: - The dataset passed into the function is ``ds``. - The function performs Grad-CAM on batches of images of size ``batch_size``. - The ReLU function can be included or excluded in the Grad-CAM computation by setting ``relu_attributions`` to either ``True`` or ``False`` respectively. Refer to :ref:`tutorial_interpret_gradcam_fooddx` to understand the purpose of the ReLU function. *It is recommended to set this as ``False``.* - If ``return_metadata=True``, each batch of images will be forward-passed into the model to get the model's predicted probabilities for each image. These probabilities will be included in the returned DataFrame. You can refer to the documentation on the ``FNFGradCAM.do_gradcam()`` function for an explanation of each argument. .. code:: python3 :number-lines: fnf_results_df = fnf_gc.do_gradcam( image_dataset=ds, batch_size=2, relu_attributions=False, return_metadata=True, output_folder="/output_folder", plot_gradcam=False ) .. code:: text Working on batch 1 of 5, batch_size = 2... Working on batch 2 of 5, batch_size = 2... Working on batch 3 of 5, batch_size = 2... Working on batch 4 of 5, batch_size = 2... Working on batch 5 of 5, batch_size = 2... The returned DataFrame (``fnf_results_df``) will look like the following: >>> fnf_results_df .. table:: :widths: auto +--------------------------------------+--------------+-----------+-----------------------------------------------------------------+-----------------------------------------------------------------------------+--------------------+--------------+--------------+ | id | target_layer | label_fnf | path_fns | path_gradcam | path_gradcam_plot | pred_proba_0 | pred_proba_1 | +======================================+==============+===========+=================================================================+=============================================================================+====================+==============+==============+ | 164c729b-86d2-11eb-b774-06d7ab6752a4 | conv2d_7b | 1 | path/to/original/image/164c729b-86d2-11eb-b774-06d7ab6752a4.png | output_folder/gradcam_numpy_arrays/164c729b-86d2-11eb-b774-06d7ab6752a4.npy | nan | 0.000242 | 0.999758 | +--------------------------------------+--------------+-----------+-----------------------------------------------------------------+-----------------------------------------------------------------------------+--------------------+--------------+--------------+ | ... | ... | ... | ... | ... | ... | ... | ... | +--------------------------------------+--------------+-----------+-----------------------------------------------------------------+-----------------------------------------------------------------------------+--------------------+--------------+--------------+ - If ``return_metadata=False``, the ``pred_proba_0`` and ``pred_proba_1`` columns will be ``nan`` values. - If ``plot_gradcam=True``, the ``path_gradcam_plot`` column will be populated with the paths to the plotted ``.png`` images. Step 5: Plot Grad-CAM results using ``plot_gradcam()``, if necessary ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ It is also possible to call the ``plot_gradcam()`` function separately to plot and save the Grad-CAM numpy arrays into ``.png`` images. The ``plot_gradcam()`` function takes in the DataFrame returned from the ``do_gradcam()`` function, retrieves the necessary information from the columns of the DataFrame to plot the images. At this point, it is important to understand that *plotting*, *showing (aka printing)* and *saving* the images are independent of each other. While ``plot_gradcam()`` **always plots the images**, whether or not the images are *shown* inline and/or *saved* as a ``.png`` file is dependent on the ``savefig`` and ``plt_show`` arguments. - If ``savefig=True`` and ``plt_show=True``, the plots will both be shown inline and saved as a ``.png`` file to the folder specified in ``savefig_dir``. - If ``savefig=True`` and ``plt_show=False``, the plots will not be shown inline but will be saved as a ``.png`` file to the folder specified in ``savefig_dir``. Vice versa. - If ``savefig=False`` and ``plt_show=False``, the plots will not be saved and shown inline. It is expected that this case is unhelpful and will not be used. The following folder structure will be created in ``savefig_dir`` where the images will be saved. Here, you can see that if the same folder is used for the ``do_gradcam()`` and ``plot_gradcam()`` functions, we will end up with the folder structure that is shown :ref:`here`. .. code:: text output_folder/ ├── gradcam_metadata.csv <--- If `save_csv=True`, this file will be created and saved. │ └── gradcam_plots/ ├── 164c728e-86d2-11eb-b774-06d7ab6752a4.png ├── 164c728f-86d2-11eb-b774-06d7ab6752a4.png ├── ... └── 164c72a5-86d2-11eb-b774-06d7ab6752a4.png You can refer to the documentation on the ``FNFGradCAM.plot_gradcam()`` fuction for an explanation of each argument. .. code:: python3 :number-lines: fnf_results_df = fnf_gc.plot_gradcam( do_gradcam_df=fnf_results_df, savefig_dir="/output_folder", savefig=True, plt_show=True, save_csv=True ) Below is a sample of the output from the ``plot_gradcam()`` function that you should expect to see. .. code:: text Plotting Grad-CAM results for 164c729b-86d2-11eb-b774-06d7ab6752a4... .. image:: ./img/tutorial_gradcam_fnf/output_14_1.png .. code:: text Plotting Grad-CAM results for 164c72a5-86d2-11eb-b774-06d7ab6752a4... .. image:: ./img/tutorial_gradcam_fnf/output_14_3.png ----- .. _tutorial_gradcam_fnf_method2: Method 2: Using the Command Line Interface (CLI) Tool ----------------------------------------------------- A CLI is the second option that can be used to perform Grad-CAM on the FNF model. Unlike :ref:`Method 1`, the CLI does not require the user to manually execute :ref:`Step 1` and :ref:`Step 2` before running the ``do_gradcam()`` and ``plot_gradcam()`` functions. This is because the CLI executes the above 5 steps all together in a single call. To get an overview of the CLI tool and its arguments, refer to :ref:`tutorial_CLI_overview`. Assuming that we are running the CLI tool from the following parent folder... .. code:: text . ├── data/ │ ├── data.csv │ └── images/ │ ├── 164c728e-86d2-11eb-b774-06d7ab6752a4.png │ ├── 164c728f-86d2-11eb-b774-06d7ab6752a4.png │ ├── ... │ └── 164c72a5-86d2-11eb-b774-06d7ab6752a4.png ├── checkpoints/ │ └── epoch_104.pth └── otuput_folder/ we will have the following paths to the arguments of the CLI tool: - ``path_to_csv = ./data/data.csv`` - ``pretrained_weights_folder = ./checkpoints/epoch_104.pth`` - ``output_folder = ./output_folder`` To maintain consistency with :ref:`Method 1`, we will still be: - Using ``conv2d_7b`` as the target layer - ``batch_size=2`` - ``relu_attributions=False`` - ``return_metadata=True`` - ``plot_gradcam=True`` To perform Grad-CAM on the FNF model, run the following in the command line: .. code:: text $ XAI FNF ./data/data.csv ./checkpoints ./output_folder conv2d_7b --batch_size 2 --return_metadata --plot_gradcam Arguments passed to the parser: ------------------------------- fnf_or_fs : FNF path_to_csv : ./data/data2.csv pretrained_weights_folder : ./checkpoints output_folder : ./output_folder target_layer_name : conv2d_7b batch_size : 2 relu_attributions : False return_metadata : True plot_gradcam : True About the dataset: ------------------ Number of images: 10 Batch size: 2 Running the XAI library: ------------------------ Application : Food-Non-Food (FNF) Number of model(s) loaded : 1 Working on batch 1 of 5, batch_size = 2... Working on batch 2 of 5, batch_size = 2... Working on batch 3 of 5, batch_size = 2... Working on batch 4 of 5, batch_size = 2... Working on batch 5 of 5, batch_size = 2... Plotting Grad-CAM results for 164c729b-86d2-11eb-b774-06d7ab6752a4... Saving 164c729b-86d2-11eb-b774-06d7ab6752a4 plot to ./output_folder/gradcam_plots/164c729b-86d2-11eb-b774-06d7ab6752a4.png Plotting Grad-CAM results for 164c72a5-86d2-11eb-b774-06d7ab6752a4... Saving 164c72a5-86d2-11eb-b774-06d7ab6752a4 plot to ./output_folder/gradcam_plots/164c72a5-86d2-11eb-b774-06d7ab6752a4.png ... Similar to :ref:`Method 1`, the Grad-CAM results will be saved in ``output_folder`` as follows: .. code:: text . ├── data/ │ ├── data.csv │ └── images/ │ ├── 164c728e-86d2-11eb-b774-06d7ab6752a4.png │ ├── 164c728f-86d2-11eb-b774-06d7ab6752a4.png │ ├── ... │ └── 164c72a5-86d2-11eb-b774-06d7ab6752a4.png ├── checkpoints/ │ └── epoch_104.pth └── otuput_folder/ └── gradcam_metadata.csv │ ├── gradcam_numpy_arrays/ │   ├── 164c728e-86d2-11eb-b774-06d7ab6752a4.npy │   ├── 164c728f-86d2-11eb-b774-06d7ab6752a4.npy │   ├── ... │   └── 164c72a5-86d2-11eb-b774-06d7ab6752a4.npy │ └── gradcam_plots/ ├── 164c728e-86d2-11eb-b774-06d7ab6752a4.png ├── 164c728f-86d2-11eb-b774-06d7ab6752a4.png ├── ... └── 164c72a5-86d2-11eb-b774-06d7ab6752a4.png