Basics: Categorical Cross-Entropy Loss

 Categorical Cross-Entropy Loss (also known as Softmax Loss) is a commonly used loss function for classification problems in neural networks, particularly when you have multiple classes. Let's break down what it is and how it works:

1. Purpose of Categorical Cross-Entropy Loss

Categorical Cross-Entropy Loss measures the difference between two probability distributions:

• The true distribution (ground-truth labels).
• The predicted distribution (model's output).

The objective of a classification model is to assign high probabilities to the correct class and low probabilities to incorrect classes. Categorical Cross-Entropy quantifies how well the predicted probabilities match the actual labels.

2. How It Works

Suppose you have:

• A neural network that outputs a probability distribution over $C$ classes.
• A true label represented as a one-hot encoded vector of length $C$. A one-hot vector is a vector where the correct class index has a value of 1, and the rest have 0.

Consider an example of a neural network used for classifying an image into one of three categories (say, Cat, Dog, Bird). After applying the softmax function to the model's output, you get predicted probabilities for each class, such as:

• Cat: 0.1
• Dog: 0.7
• Bird: 0.2

The true label is one-hot encoded. If the image is of a Dog, the true label would be:

• [0, 1, 0]

3. The Formula

Categorical Cross-Entropy Loss $L$ for a single training example can be represented mathematically as:

$$L = -\sum_{i=1}^{C} y_i \cdot \log(\hat{y}_i)$$

Where:

• $C$ is the number of classes.
• $y_i$ is the true label for class $i$, which is either 0 or 1 (from the one-hot encoded vector).
• $\hat{y}_i$ is the predicted probability for class $i$ (output of the softmax function).

In simpler terms, you take the negative log of the predicted probability for the true class.

For the Dog example above:

• $y=[0,1,0]$
• $\widehat{y}=[0.1,0.7,0.2]$

The loss for this example is:

$$L = -(0 \cdot \log(0.1) + 1 \cdot \log(0.7) + 0 \cdot \log(0.2)) = -\log(0.7) \approx 0.357$$

The idea is that if the model assigns a higher probability to the correct class, the loss will be lower. If it assigns a low probability, the loss will be higher.

4. Key Insights

• Minimization Goal: The goal during training is to minimize the Categorical Cross-Entropy Loss, meaning that the model is encouraged to output higher probabilities for the correct classes.
• Log Function: The logarithm in the formula penalizes confident incorrect predictions more heavily. For example, if the model predicts a high probability for a wrong class, the negative log will yield a high value, leading to a higher loss.
• Softmax: Typically, the output layer of a classification neural network applies a softmax activation to convert raw scores (logits) into probabilities that sum to 1. The Categorical Cross-Entropy Loss works well with the output of a softmax.

5. Usage Scenarios

• Multi-Class Classification: It is used when there are more than two classes, and each input belongs to only one class. This is common in image classification tasks, where an image might belong to one of several categories.
• One-Hot Encoding: The true labels are represented using one-hot encoding because only one class is correct for each example.

6. Intuition

• If the model perfectly predicts the correct class (probability of 1 for the true class), the loss will be zero because $\log(1) = 0$
  
• If the model is uncertain and assigns a low probability to the correct class, the loss will be high, encouraging the model to adjust its weights to improve.

7. Comparison to Binary Cross-Entropy

Categorical Cross-Entropy Loss is different from Binary Cross-Entropy Loss, which is used for binary classification tasks (i.e., only two classes). Categorical Cross-Entropy is used for multi-class problems where each instance belongs to one class out of many.

Example in Python (Keras)

In Keras, if you are building a classification model, you can specify the loss function like this:

Python

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

This tells Keras to use Categorical Cross-Entropy as the loss function for training the model.

Summary

• Categorical Cross-Entropy Loss is used to measure the dissimilarity between predicted and actual labels for multi-class classification tasks.
• It penalizes incorrect predictions based on the logarithm of predicted probabilities.
• The goal during training is to minimize this loss, encouraging the model to assign higher probabilities to the correct class.