Sunday, 25 May 2025

Understanding Computational Graphs and Backpropagation: A Beginner's Guide

Deep learning is built on two foundational ideas: computational graphs and backpropagation. While powerful neural networks can learn complex patterns from data, it is the humble computational graph that allows this learning to happen. This article introduces the concepts step-by-step for beginners, exploring what a computational graph is, why it is used, how it's constructed, and how it enables backpropagation through the chain rule.

1. What is a Computational Graph?

A computational graph is a structured representation of a mathematical function broken down into elementary operations. Each node in the graph represents an operation (such as addition, multiplication, or a nonlinear activation function), and each edge represents the flow of data—scalars, vectors, or tensors—between these operations.

For example, consider the function:

\[ z = (x + y) \cdot w \]

Each operation is a building block. By connecting simple operations together, complex functions (such as those in neural networks) can be represented and efficiently computed.

2. Why Do We Use Computational Graphs in Deep Learning?

The computational graph is essential for automating the calculation of derivatives, which is critical for learning in neural networks. Below are the primary reasons for its widespread use:

Backpropagation: Gradients are needed to update weights in neural networks. The computational graph supports automatic differentiation using the chain rule, which allows this to happen systematically.
Modular Design: Complex architectures (e.g., CNNs, LSTMs) can be built as combinations of smaller computational subgraphs.
Efficiency: Intermediate results from the forward pass can be stored and reused in the backward pass, saving computation.
Framework Foundation: Libraries like TensorFlow, PyTorch, and JAX use computational graphs under the hood for both defining models and computing gradients.
Optimization and Deployment: In static graph frameworks, the entire graph can be optimized or exported for deployment across devices.

Thus, the computational graph is more than just a teaching tool—it's the foundation of modern deep learning pipelines.

3. Breaking a Function into a Computational Graph

Breaking down a function into a graph involves expressing it as a sequence of basic operations. Each operation becomes a node, and the intermediate values become edges.

Let’s return to our earlier function:

\[ z = (x + y) \cdot w \]

We break it down into two operations:

\( a = x + y \)
\( z = a \cdot w \)

The table below summarizes this decomposition:

Step	Operation	Node Type	Inputs	Output
1	\( a = x + y \)	Addition	x, y	a
2	\( z = a \cdot w \)	Multiplication	a, w	z

This modular representation is what allows the graph to be traversed forward to compute outputs and backward to compute gradients.

4. What Do the Nodes and Edges Represent?

Each part of the computational graph has a specific role:

Nodes: Represent either input values (e.g., \(x\), \(y\), \(w\)), operations (e.g., +, *, sigmoid), or intermediate results (e.g., \(a\))
Edges: Represent the flow of data—intermediate values computed during the forward pass or gradients during the backward pass

To conceptualize:

Think of nodes as processors that apply a mathematical operation
Think of edges as wires that carry results from one processor to another

In the forward pass, data flows from input to output. In the backward pass, gradients flow in the reverse direction. This two-way traversal is what makes learning possible.

`💡 Bonus: Code Example of a Simple Computational Graph`


# A simple forward and backward pass without any library
x = 2.0
y = 3.0
w = 4.0

# Forward Pass
a = x + y        # Node 1
z = a * w        # Node 2

# Backward Pass
dz_dw = a        # ∂z/∂w
dz_da = w        # ∂z/∂a
da_dx = 1.0      # ∂a/∂x
da_dy = 1.0      # ∂a/∂y

# Chain Rule
dz_dx = dz_da * da_dx
dz_dy = dz_da * da_dy

print(f"Gradient wrt x: {dz_dx}")
print(f"Gradient wrt y: {dz_dy}")
print(f"Gradient wrt w: {dz_dw}")

This snippet demonstrates how gradients are propagated backward using local derivatives and the chain rule.

Conclusion

Computational graphs are the backbone of how modern deep learning systems compute, learn, and optimize. They provide an intuitive and scalable way to break complex functions into atomic operations and enable automatic differentiation. Whether you're hand-coding a neural net or using PyTorch, understanding how your model is represented as a computational graph will give you better control and insight into model behavior, training dynamics, and debugging.

References

Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). "Learning representations by back-propagating errors." Nature, 323(6088), 533–536.
Baydin, A. G., Pearlmutter, B. A., Radul, A. A., & Siskind, J. M. (2017). "Automatic differentiation in machine learning: a survey." Journal of Marchine Learning Research, 18(153), 1-43.
Paszke, A., et al. (2019). "PyTorch: An Imperative Style, High-Performance Deep Learning Library." NeurIPS.
Abadi, M., et al. (2016). "TensorFlow: A system for large-scale machine learning." OSDI.
Goodfellow, I., Bengio, Y., & Courville, A. (2016). "Deep Learning." MIT Press.

Keywords

computational graph, backpropagation, neural network, deep learning, chain rule, gradient descent, automatic differentiation, forward pass, backward pass, tensor operations

Saturday, 24 May 2025

Understanding Monte Carlo Simulation: A Beginner's Guide to Probabilistic Modeling

Monte Carlo Simulation (MCS) is a powerful computational technique used to model the probability of different outcomes in systems that are inherently unpredictable. By using randomness and statistical sampling, it allows researchers and decision-makers to explore the behavior of complex systems and make informed predictions under uncertainty. This beginner-level guide explores the foundational concepts of Monte Carlo Simulation, its applications, key steps, and simple implementation strategies.

What is Monte Carlo Simulation?

Monte Carlo Simulation is named after the famous Monte Carlo Casino in Monaco, symbolizing the randomness and chance inherent in its approach. In essence, MCS involves running simulations many times—often thousands or millions—using random inputs to generate a distribution of possible outcomes. This distribution can then be used to estimate statistical properties such as the mean, variance, and confidence intervals.

Why Use Monte Carlo Simulation?

Many real-world problems contain uncertainty in parameters, market fluctuations, operational risks, and more. Unlike deterministic models, Monte Carlo Simulation allows analysts to incorporate this uncertainty directly into the model. For example:

Finance: Estimating the value at risk (VaR) in portfolio management
Engineering: Assessing reliability of systems with uncertain component behavior
Project Management: Forecasting project completion time under variable task durations
Science: Modeling particle behavior in physics or epidemics in public health

This flexibility and breadth make Monte Carlo Simulation an essential part of any probabilistic analysis toolkit.

Key Steps in Monte Carlo Simulation

Here is a step-by-step breakdown of how a basic Monte Carlo Simulation works:

Step	Description
1. Define the Problem	Identify the process or system you wish to analyze and the uncertain variables involved.
2. Assign Probabilistic Models	Determine the probability distributions for the uncertain variables (e.g., Normal, Uniform, Exponential).
3. Generate Random Inputs	Use random number generators to sample from these distributions.
4. Run Simulations	Compute the output for each set of random inputs, repeating the process many times.
5. Analyze Results	Aggregate the simulation results to form a probability distribution of the output.

Simple Python Example

Let’s estimate the value of \( \pi \) using a Monte Carlo method. The idea is to randomly generate points in a square and check how many fall inside the inscribed circle.

import random
import math

def monte_carlo_pi(num_samples):
    inside_circle = 0
    for _ in range(num_samples):
        x = random.uniform(0, 1)
        y = random.uniform(0, 1)
        if x**2 + y**2 <= 1:
            inside_circle += 1
    return 4 * inside_circle / num_samples

estimated_pi = monte_carlo_pi(100000)
print(f"Estimated π = {estimated_pi}")

In this case, we use the ratio of points inside the circle to the total number of points to estimate the value of \( \pi \). The more points we use, the closer the result gets to the true value.

Interpreting Monte Carlo Results

Monte Carlo output typically consists of a histogram or distribution curve that shows how frequently each outcome occurred. Analysts can then extract insights such as:

Expected Value: The average result from all simulations
Standard Deviation: A measure of variability
Confidence Intervals: Likelihood of outcomes falling within certain ranges

These metrics are invaluable for decision-making under uncertainty, allowing better risk management and contingency planning.

Applications of Monte Carlo Simulation

Monte Carlo Simulation is used extensively across fields:

Finance: Portfolio optimization, derivatives pricing
Manufacturing: Quality control, defect prediction
Supply Chain: Inventory optimization under demand variability
Climate Science: Forecasting weather and climate patterns

Its strength lies in dealing with situations where outcomes depend on multiple random factors and traditional analytical solutions are infeasible.

Advantages and Limitations

Advantages	Limitations
Handles uncertainty effectively	Requires significant computational resources for large simulations
Can model nonlinear, complex systems	Relies on accurate input distributions—garbage in, garbage out
Provides visual insights into variability and risk	Stochastic output can be difficult to interpret without statistical knowledge

Conclusion

Monte Carlo Simulation offers a versatile and intuitive way to understand uncertainty and risk in complex systems. By leveraging randomness and repeated sampling, it empowers analysts to go beyond deterministic models and make data-informed decisions under uncertainty. For beginners, starting with simple simulations—like estimating \( \pi \)—can provide a great introduction before progressing to real-world applications in finance, logistics, or engineering.

References

Metropolis, N., & Ulam, S. (1949). The Monte Carlo Method. Journal of the American Statistical Association, 44(247), 335–341.
Kalos, M.H., & Whitlock, P.A. (2008). Monte Carlo Methods Volume I: Basics. Wiley-VCH.
Fishman, G.S. (1996). Monte Carlo: Concepts, Algorithms, and Applications. Springer.
Rubinstein, R.Y., & Kroese, D.P. (2016). Simulation and the Monte Carlo Method. Wiley.
Glasserman, P. (2003). Monte Carlo Methods in Financial Engineering. Springer.

Keywords

Monte Carlo Simulation, Probabilistic Modeling, Risk Analysis, Random Sampling, Python Simulation, Financial Modeling, Project Risk, Uncertainty Quantification, Statistical Computing, Forecasting

Understanding the Hadamard Product: A Fundamental Operation in Neural Network Gradients

Author: Priyank Goyal
Date: May 24, 2025
Tags: Hadamard Product, Neural Networks, Gradient Computation, NumPy, Python, Machine Learning

Introduction

In the study of deep learning and backpropagation, one encounters a variety of matrix operations. Among these, the Hadamard product—also known as the element-wise product—plays a vital role in computing gradients efficiently. This blog post explains the concept, usage, and implementation of the Hadamard product, particularly in the context of neural networks, where it emerges naturally during the derivative computations of activation functions.

What Is the Hadamard Product?

The Hadamard product is a binary operation that takes two matrices (or vectors) of the same dimension and returns a new matrix (or vector) by multiplying corresponding elements.

The mathematical notation is as follows:

\( A \circ B = \left[ a_{ij} \cdot b_{ij} \right] \)

Here, \( \circ \) denotes the Hadamard product, and \( a_{ij} \), \( b_{ij} \) are the elements of matrices \( A \) and \( B \), respectively.

Important Characteristics

Shape Compatibility: Both matrices or vectors must have the same shape.
Different from Matrix Multiplication: No row-by-column summation is involved; it's a simple element-wise operation.
Common in Neural Networks: Used in backpropagation when combining gradients with activation derivatives.

Hadamard Product in Gradient Computation

In neural networks, the chain rule in backpropagation often involves the derivative of the cost function with respect to biases or activations. Consider the derivative expression below:

\( \frac{\partial s}{\partial b} = u^T \circ f'(z) \)

This expression indicates that to compute the gradient \( \frac{\partial s}{\partial b} \), we take the Hadamard product of the transposed upstream gradient \( u^T \) and the derivative of the activation function \( f'(z) \).

Here, each element of the resulting vector gives the partial derivative of the scalar function \( s \) with respect to each component of the bias vector \( b \).

Python Implementation Using NumPy

Let’s explore how the Hadamard product is implemented in Python using NumPy.

Example 1: Hadamard Product with Vectors

import numpy as np

# Define two vectors
a = np.array([1, 2, 3])
b = np.array([4, 5, 6])

# Hadamard product (element-wise)
hadamard_vector = a * b

print("Vector Hadamard Product:", hadamard_vector)

Output:

Vector Hadamard Product: [ 4 10 18]

Example 2: Hadamard Product with Matrices

# Define two 2D matrices of the same shape
A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])

# Hadamard product (element-wise)
hadamard_matrix = A * B

print("Matrix Hadamard Product:\n", hadamard_matrix)

Output:

Matrix Hadamard Product:
 [[ 5 12]
  [21 32]]

Visualizing the Operation

Here’s a table representation of the Hadamard product for matrices:

Matrix A	Matrix B	A ∘ B
[1, 2]	[5, 6]	[1×5, 2×6] = [5, 12]
[3, 4]	[7, 8]	[3×7, 4×8] = [21, 32]

Why Not Use Matrix Multiplication?

Matrix multiplication combines rows and columns across matrices using dot products. However, in neural networks, especially when computing gradients element-wise (e.g., matching upstream gradients with element-specific activation derivatives), we need a direct one-to-one multiplication. That’s where Hadamard product fits perfectly.

Use Cases in Deep Learning

Backpropagation: Calculating gradients of loss functions with respect to pre-activation values.
Dropout: Masking neurons by multiplying activations with dropout masks element-wise.
Attention Mechanisms: Combining relevance scores and input embeddings.

Mathematical Intuition

If you consider a neural network layer with activation \( a = f(z) \), where \( z = Wx + b \), the derivative of the loss with respect to \( z \) is given by:

\( \frac{\partial L}{\partial z} = \frac{\partial L}{\partial a} \circ f'(z) \)

Here, \( \frac{\partial L}{\partial a} \) is the upstream gradient passed from the next layer, and \( f'(z) \) is the derivative of the activation function. The Hadamard product naturally arises from applying the chain rule element-wise.

Conclusion

The Hadamard product is a simple yet powerful mathematical operation frequently used in neural networks and deep learning. It enables element-wise gradient propagation, allows integration of activation function derivatives, and supports efficient implementation of various deep learning modules such as dropout and attention mechanisms.

Understanding this foundational concept equips researchers and practitioners to build better, more interpretable neural network models. Whether you're coding your first backpropagation loop or debugging gradient mismatches, knowing the Hadamard product is essential.

1. Network Architecture

We consider a compact neural network with the following configuration:

Input Layer: 3 features
Hidden Layer: 2 neurons with ReLU activation
Output Layer: 1 neuron with sigmoid activation

The network performs binary classification. The full data flow is represented by:

\[ z_1 = W_1 x + b_1,\quad h = \text{ReLU}(z_1),\quad z_2 = W_2 h + b_2,\quad \hat{y} = \sigma(z_2) \]

Each component’s shape determines the feasibility and correctness of these computations.

2. Shape Overview of Components

Let’s assign dimensions to each element. The following table summarizes the shapes used in the network:

Component	Symbol	Shape	Meaning
Input	\( x \)	3×1	Input column vector with 3 features
Hidden weights	\( W_1 \)	2×3	2 neurons, each connected to 3 inputs
Hidden bias	\( b_1 \)	2×1	1 bias per hidden neuron
Hidden pre-activation	\( z_1 \)	2×1	Weighted sum before activation
Hidden activation	\( h \)	2×1	After ReLU
Output weights	\( W_2 \)	1×2	Connects 2 hidden outputs to 1 output
Output bias	\( b_2 \)	1×1	Bias added at output node
Output pre-activation	\( z_2 \)	1×1	Scalar score before sigmoid
Output prediction	\( \hat{y} \)	1×1	Sigmoid output: predicted probability

3. Forward Pass: Dimensions at Each Step

Step 1: Hidden Pre-Activation

\[ z_1 = W_1 x + b_1 \] - \( W_1 \in \mathbb{R}^{2 \times 3} \) - \( x \in \mathbb{R}^{3 \times 1} \) - \( W_1 x \in \mathbb{R}^{2 \times 1} \) - \( b_1 \in \mathbb{R}^{2 \times 1} \) - Result: \( z_1 \in \mathbb{R}^{2 \times 1} \)

Step 2: Hidden Activation

\[ h = \text{ReLU}(z_1) \in \mathbb{R}^{2 \times 1} \]

Step 3: Output Pre-Activation

\[ z_2 = W_2 h + b_2 \] - \( W_2 \in \mathbb{R}^{1 \times 2} \) - \( h \in \mathbb{R}^{2 \times 1} \) - \( W_2 h \in \mathbb{R}^{1 \times 1} \) - \( b_2 \in \mathbb{R}^{1 \times 1} \) - Result: \( z_2 \in \mathbb{R}^{1 \times 1} \)

Step 4: Output

\[ \hat{y} = \sigma(z_2) \in \mathbb{R}^{1 \times 1} \] ---

4. Backward Pass: Gradient Dimensions

We now track the shape of each gradient flowing backward from the loss \( \mathcal{L} \).

Step 1: Gradient w.r.t. \( z_2 \)

\[ \frac{\partial \mathcal{L}}{\partial z_2} \in \mathbb{R}^{1 \times 1} \]

Step 2: Gradient w.r.t. Output Layer Parameters

\[ \frac{\partial \mathcal{L}}{\partial W_2} = \frac{\partial \mathcal{L}}{\partial z_2} \cdot h^T \in \mathbb{R}^{1 \times 2} \] \[ \frac{\partial \mathcal{L}}{\partial b_2} = \frac{\partial \mathcal{L}}{\partial z_2} \in \mathbb{R}^{1 \times 1} \]

Step 3: Gradient w.r.t. Hidden Layer Activation

\[ \frac{\partial \mathcal{L}}{\partial h} = W_2^T \cdot \frac{\partial \mathcal{L}}{\partial z_2} \in \mathbb{R}^{2 \times 1} \]

Step 4: Gradient w.r.t. \( z_1 \) Using ReLU Derivative

\[ \frac{\partial \mathcal{L}}{\partial z_1} = \frac{\partial \mathcal{L}}{\partial h} \circ f'(z_1) \in \mathbb{R}^{2 \times 1} \]

Step 5: Gradient w.r.t. Hidden Layer Parameters

\[ \frac{\partial \mathcal{L}}{\partial W_1} = \frac{\partial \mathcal{L}}{\partial z_1} \cdot x^T \in \mathbb{R}^{2 \times 3} \] \[ \frac{\partial \mathcal{L}}{\partial b_1} = \frac{\partial \mathcal{L}}{\partial z_1} \in \mathbb{R}^{2 \times 1} \]

5. Python Code with Shape Comments


# Input
x = np.array([[1.0], [0.5], [-1.5]])       # shape: (3,1)

# Parameters
W1 = np.random.randn(2, 3)                # shape: (2,3)
b1 = np.random.randn(2, 1)                # shape: (2,1)
W2 = np.random.randn(1, 2)                # shape: (1,2)
b2 = np.random.randn(1, 1)                # shape: (1,1)

# Forward pass
z1 = W1 @ x + b1                          # shape: (2,1)
h = np.maximum(0, z1)                     # shape: (2,1)
z2 = W2 @ h + b2                          # shape: (1,1)
y_pred = 1 / (1 + np.exp(-z2))            # shape: (1,1)

# Backward pass
dy = -(1 / y_pred) * (y_pred * (1 - y_pred))  # shape: (1,1)
dW2 = dy @ h.T                            # shape: (1,2)
db2 = dy                                 # shape: (1,1)

dh = W2.T @ dy                           # shape: (2,1)
dz1 = dh * (z1 > 0).astype(float)         # shape: (2,1)
dW1 = dz1 @ x.T                           # shape: (2,3)
db1 = dz1                                 # shape: (2,1)

6. Final Thoughts

When designing neural networks, especially from scratch, paying attention to matrix dimensions ensures model correctness and computational stability. It helps prevent silent shape mismatches and clarifies how data flows through the layers. Whether you’re performing forward passes, computing loss, or updating gradients, checking and tracking matrix dimensions is not optional—it’s essential.

This discipline becomes even more vital in deeper architectures, recurrent models, or when batching inputs. Understanding shapes is your first line of defense—and a powerful debugging ally.

Backpropagation Demystified: A Step-by-Step Guide with Code

By Priyank Goyal

Backpropagation is the engine that drives learning in neural networks. Despite its intimidating name, it is simply an application of the chain rule from calculus, organized to efficiently compute the gradient of a loss function with respect to all the parameters of a network. In this article, we’ll unpack backpropagation by taking you through a small neural network, step by step—from the forward pass, to the computation of gradients, and finally to updating the parameters. We’ll also write code to perform this process manually using NumPy.

1. Overview of the Neural Network

We’ll use a simple feedforward neural network with:

Input layer: 3 neurons (i.e., 3 input features)
Hidden layer: 2 neurons with ReLU activation
Output layer: 1 neuron with sigmoid activation

This network maps input features to a probability value \( \hat{y} \in [0, 1] \), useful for binary classification.

The mathematical flow of data is as follows:

\[ z_1 = W_1 x + b_1,\quad h = \text{ReLU}(z_1),\quad z_2 = W_2 h + b_2,\quad \hat{y} = \sigma(z_2) \]

Where:

\( x \in \mathbb{R}^{3 \times 1} \) is the input column vector
\( W_1 \in \mathbb{R}^{2 \times 3},\ b_1 \in \mathbb{R}^{2 \times 1} \)
\( W_2 \in \mathbb{R}^{1 \times 2},\ b_2 \in \mathbb{R}^{1 \times 1} \)

2. Forward Propagation

Let’s begin with initial values for inputs and weights:

x = np.array([[1.0], [0.5], [-1.5]])

W1 = np.array([[0.2, -0.4, 0.1],
               [0.7,  0.3, -0.5]])
b1 = np.array([[0.1],
               [-0.2]])

W2 = np.array([[0.5, -1.0]])
b2 = np.array([[0.2]])

Compute the intermediate outputs:

\[ z_1 = W_1 x + b_1 = \begin{bmatrix} -0.15 \\ 1.8 \end{bmatrix},\quad h = \text{ReLU}(z_1) = \begin{bmatrix} 0 \\ 1.8 \end{bmatrix} \] \[ z_2 = W_2 h + b_2 = -1.6,\quad \hat{y} = \sigma(z_2) = \frac{1}{1 + e^{1.6}} \approx 0.167 \]

3. Loss Function

Assuming the true label is \( y = 1 \), we use the binary cross-entropy loss:

\[ \mathcal{L} = -\left[y \log(\hat{y}) + (1 - y) \log(1 - \hat{y})\right] \approx -\log(0.167) \approx 1.79 \]

This loss tells us how far off the prediction is from the ground truth.

4. Backpropagation: Computing Gradients

4.1. Calculating Gradient w.r.t. \( z_2 \)

\[ \frac{\partial \mathcal{L}}{\partial \hat{y}} = -\frac{1}{\hat{y}}\]

Calculations

Assume the loss function is the binary cross-entropy loss:

\[ \mathcal{L} = -\left( y \log(\hat{y}) + (1 - y) \log(1 - \hat{y}) \right) \]

We differentiate with respect to \( \hat{y} \):

\[ \frac{\partial \mathcal{L}}{\partial \hat{y}} = -\frac{y}{\hat{y}} + \frac{1 - y}{1 - \hat{y}} \]

Now, consider the special case where the true label \( y = 1 \). The loss function simplifies to:

\[ \mathcal{L} = -\log(\hat{y}) \]

Differentiating this gives:

\[ \frac{\partial \mathcal{L}}{\partial \hat{y}} = -\frac{1}{\hat{y}} \]

Thus, the expression \[ \frac{\partial \mathcal{L}}{\partial \hat{y}} = -\frac{1}{\hat{y}} \] is correct in the case where \( y = 1 \).

\[\quad \frac{\partial \hat{y}}{\partial z_2} = \hat{y}(1 - \hat{y}) \]

Calculation of \[\quad \frac{\partial \hat{y}}{\partial z_2} = \hat{y}(1 - \hat{y}) \]

Step 1: Define the Sigmoid Function

Let the sigmoid function be defined as:

\[ \sigma(z) = \frac{1}{1 + e^{-z}} \]

We aim to compute the derivative:

\[ \frac{d}{dz} \left( \frac{1}{1 + e^{-z}} \right) \]

Step 2: Apply the Chain Rule

Let: \[ f(z) = \frac{1}{g(z)}, \quad \text{where } g(z) = 1 + e^{-z} \]

Then, using the chain rule: \[ \frac{df}{dz} = -\frac{1}{(g(z))^2} \cdot \frac{dg}{dz} \]

Compute the derivative of the inner function: \[ \frac{d}{dz}(1 + e^{-z}) = -e^{-z} \]

Substituting back: \[ \frac{d}{dz} \left( \frac{1}{1 + e^{-z}} \right) = -\frac{1}{(1 + e^{-z})^2} \cdot (-e^{-z}) = \frac{e^{-z}}{(1 + e^{-z})^2} \]

Step 3: Express in Terms of Sigmoid

Recall that: \[ \sigma(z) = \frac{1}{1 + e^{-z}}, \quad 1 - \sigma(z) = \frac{e^{-z}}{1 + e^{-z}} \]

Multiplying both expressions: \[ \sigma(z)(1 - \sigma(z)) = \frac{1}{1 + e^{-z}} \cdot \frac{e^{-z}}{1 + e^{-z}} = \frac{e^{-z}}{(1 + e^{-z})^2} \]

✅ Final Result

\[ \frac{d}{dz} \left( \frac{1}{1 + e^{-z}} \right) = \sigma(z)(1 - \sigma(z)) \]

\[ \delta_2 = \frac{\partial \mathcal{L}}{\partial z_2} = -\frac{1}{0.167} \cdot 0.167(1 - 0.167) \approx -0.83 \]

4.2. Gradients for Output Layer

\[ \frac{\partial \mathcal{L}}{\partial W_2} = \delta_2 \cdot h^T = -0.83 \cdot \begin{bmatrix} 0 & 1.8 \end{bmatrix} = \begin{bmatrix} 0 & -1.49 \end{bmatrix} \] \[ \frac{\partial \mathcal{L}}{\partial b_2} = \delta_2 = -0.83 \]

4.3. Gradient w.r.t. Hidden Layer Output

\[ \frac{\partial \mathcal{L}}{\partial h} = W_2^T \cdot \delta_2 = \begin{bmatrix} 0.5 \\ -1.0 \end{bmatrix} \cdot -0.83 = \begin{bmatrix} -0.415 \\ 0.83 \end{bmatrix} \]

4.4. Gradient w.r.t. \( z_1 \) (ReLU Derivative)

\[ \frac{\partial \mathcal{L}}{\partial z_1} = \frac{\partial \mathcal{L}}{\partial h} \circ f'(z_1) \]

Where \( f'(z_1) = \begin{bmatrix} 0 \\ 1 \end{bmatrix} \), since ReLU’s derivative is 1 for positive inputs and 0 otherwise.

\[ \delta_1 = \begin{bmatrix} -0.415 \\ 0.83 \end{bmatrix} \circ \begin{bmatrix} 0 \\ 1 \end{bmatrix} = \begin{bmatrix} 0 \\ 0.83 \end{bmatrix} \]

4.5. Gradients for Hidden Layer

\[ \frac{\partial \mathcal{L}}{\partial W_1} = \delta_1 \cdot x^T = \begin{bmatrix} 0 & 0 & 0 \\ 0.83 & 0.415 & -1.245 \end{bmatrix} \] \[ \frac{\partial \mathcal{L}}{\partial b_1} = \delta_1 = \begin{bmatrix} 0 \\ 0.83 \end{bmatrix} \]

5. Update Step (Gradient Descent)

With a learning rate \( \eta = 0.1 \), we update:

\[ W_2 := W_2 - \eta \cdot \frac{\partial \mathcal{L}}{\partial W_2},\quad b_2 := b_2 - \eta \cdot \frac{\partial \mathcal{L}}{\partial b_2} \] \[ W_1 := W_1 - \eta \cdot \frac{\partial \mathcal{L}}{\partial W_1},\quad b_1 := b_1 - \eta \cdot \frac{\partial \mathcal{L}}{\partial b_1} \]

This updates all weights and biases in the direction that minimizes the loss.

6. Python Code Implementation


import numpy as np

x = np.array([[1.0], [0.5], [-1.5]])
y_true = 1

W1 = np.array([[0.2, -0.4, 0.1],
               [0.7,  0.3, -0.5]])
b1 = np.array([[0.1],
               [-0.2]])

W2 = np.array([[0.5, -1.0]])
b2 = np.array([[0.2]])

lr = 0.1

# Forward Pass
z1 = W1 @ x + b1
h = np.maximum(0, z1)
z2 = W2 @ h + b2
y_pred = 1 / (1 + np.exp(-z2))
loss = - (y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))

# Backward Pass
dL_dz2 = -(y_true / y_pred) * (y_pred * (1 - y_pred))
dL_dW2 = dL_dz2 @ h.T
dL_db2 = dL_dz2

dL_dh = W2.T @ dL_dz2
dL_dz1 = dL_dh * (z1 > 0)
dL_dW1 = dL_dz1 @ x.T
dL_db1 = dL_dz1

# Parameter Update
W2 -= lr * dL_dW2
b2 -= lr * dL_db2
W1 -= lr * dL_dW1
b1 -= lr * dL_db1

6. Python Code Implementation-Tensor Flow


import tensorflow as tf

# Input and true label
x = tf.constant([[1.0], [0.5], [-1.5]])  # Shape (3, 1)
y_true = tf.constant([[1.0]])            # Shape (1, 1)

# Initialize weights and biases
W1 = tf.Variable([[0.2, -0.4, 0.1],
                  [0.7,  0.3, -0.5]], dtype=tf.float32)
b1 = tf.Variable([[0.1],
                  [-0.2]], dtype=tf.float32)

W2 = tf.Variable([[0.5, -1.0]], dtype=tf.float32)
b2 = tf.Variable([[0.2]], dtype=tf.float32)

lr = 0.1  # Learning rate

# Forward and backward pass using GradientTape
with tf.GradientTape() as tape:
    z1 = tf.matmul(W1, x) + b1
    h = tf.nn.relu(z1)
    z2 = tf.matmul(W2, h) + b2
    y_pred = tf.sigmoid(z2)
    loss = tf.keras.losses.binary_crossentropy(y_true, y_pred)

# Compute gradients
grads = tape.gradient(loss, [W1, b1, W2, b2])
dW1, db1, dW2, db2 = grads

# Print gradients
print("Loss:", loss.numpy())
print("dL/dW1:\n", dW1.numpy())
print("dL/db1:\n", db1.numpy())
print("dL/dW2:\n", dW2.numpy())
print("dL/db2:\n", db2.numpy())

# Update weights and biases
W1.assign_sub(lr * dW1)
b1.assign_sub(lr * db1)
W2.assign_sub(lr * dW2)
b2.assign_sub(lr * db2)

7. Final Updated Parameters

Parameter	Updated Value
`W1`	\[ \begin{bmatrix} 0.2 & -0.4 & 0.1 \\ 0.623 & 0.262 & -0.385 \end{bmatrix} \]
`b1`	\[ \begin{bmatrix} 0.1 \\ -0.277 \end{bmatrix} \]
`W2`	\[ \begin{bmatrix} 0.5 & -0.892 \end{bmatrix} \]
`b2`	\( 0.277 \)
Loss	\( \mathcal{L} \approx 1.463 \)

8. Conclusion

This walk-through shows how backpropagation works at a low level by breaking it into mathematical components and implementing each in Python. While high-level libraries like TensorFlow and PyTorch abstract this process, understanding backpropagation gives you deeper insights into how and why neural networks learn—and is essential for debugging, optimization, and research.

In future posts, we will extend this to multiple training steps, batches, and deeper networks with multiple layers and activation functions. But as always, true mastery begins with simplicity.

Understanding Backpropagation Through a Basic Neural Network Example

By Priyank Goyal

In this article, we explore the foundational workings of a feedforward neural network with a single hidden layer. Our goal is to demystify the process of forward propagation, loss computation, and especially backpropagation—the core algorithm that powers learning in neural networks. This walkthrough answers three essential questions:

What is a basic neural network?
How does backpropagation work in such a network?
How can this process be translated into working Python code?

1. Anatomy of a Basic Neural Network

Let us consider a minimal neural network consisting of three layers:

Input Layer with 3 neurons (features)
Hidden Layer with 2 neurons and ReLU activation
Output Layer with 1 neuron and sigmoid activation

The goal of this network is to perform binary classification. Each input vector \( x \in \mathbb{R}^3 \) is passed through the network, which outputs a probability \( \hat{y} \in [0, 1] \).

The network's computation can be summarized as:

\[ z_1 = W_1 x + b_1,\quad h = \text{ReLU}(z_1),\quad z_2 = W_2 h + b_2,\quad \hat{y} = \sigma(z_2) \]

Where:

\( W_1 \in \mathbb{R}^{2 \times 3} \), \( b_1 \in \mathbb{R}^{2 \times 1} \)
\( W_2 \in \mathbb{R}^{1 \times 2} \), \( b_2 \in \mathbb{R} \)
\( \text{ReLU}(z) = \max(0, z) \)
\( \sigma(z) = \frac{1}{1 + e^{-z}} \) is the sigmoid function

2. Forward Propagation

Given the input vector:

\[ x = \begin{bmatrix} 1.0 \\ 0.5 \\ -1.5 \end{bmatrix} \]

and initial parameters:

\[ W_1 = \begin{bmatrix} 0.2 & -0.4 & 0.1 \\ 0.7 & 0.3 & -0.5 \end{bmatrix}, \quad b_1 = \begin{bmatrix} 0.1 \\ -0.2 \end{bmatrix}, \quad W_2 = \begin{bmatrix} 0.5 & -1.0 \end{bmatrix}, \quad b_2 = 0.2 \]

We compute:

3. Loss Computation

Assuming the true label is \( y = 1 \), the binary cross-entropy loss is:

\[ \mathcal{L} = -\left[y \log(\hat{y}) + (1 - y) \log(1 - \hat{y})\right] \approx 1.79 \]

4. Backpropagation Step-by-Step

We now compute the gradients of the loss with respect to each parameter in reverse order, using the chain rule.

Output layer:
\[ \frac{\partial \mathcal{L}}{\partial \hat{y}} = -\frac{1}{\hat{y}},\quad \frac{\partial \hat{y}}{\partial z_2} = \hat{y}(1 - \hat{y}) \] \[ \delta_2 = \frac{\partial \mathcal{L}}{\partial z_2} \approx -0.83 \] \[ \frac{\partial \mathcal{L}}{\partial W_2} = \delta_2 \cdot h^T,\quad \frac{\partial \mathcal{L}}{\partial b_2} = \delta_2 \]
Hidden layer:
\[ \delta_1 = (W_2^T \delta_2) \circ f'(z_1),\quad \text{where ReLU}'(z) = 1 \text{ if } z > 0, 0 \text{ otherwise} \] \[ \frac{\partial \mathcal{L}}{\partial W_1} = \delta_1 \cdot x^T,\quad \frac{\partial \mathcal{L}}{\partial b_1} = \delta_1 \]

5. Python Implementation

The following code performs a full forward and backward pass on the network and updates weights using gradient descent:


import numpy as np

x = np.array([[1.0], [0.5], [-1.5]])
y_true = 1

W1 = np.array([[0.2, -0.4, 0.1], [0.7, 0.3, -0.5]])
b1 = np.array([[0.1], [-0.2]])
W2 = np.array([[0.5, -1.0]])
b2 = np.array([[0.2]])

lr = 0.1

# Forward
z1 = W1 @ x + b1
h = np.maximum(0, z1)
z2 = W2 @ h + b2
y_pred = 1 / (1 + np.exp(-z2))
loss = - (y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))

# Backward
dL_dz2 = -(y_true / y_pred) * (y_pred * (1 - y_pred))
dL_dW2 = dL_dz2 @ h.T
dL_db2 = dL_dz2

dL_dh = W2.T @ dL_dz2
dL_dz1 = dL_dh * (z1 > 0)
dL_dW1 = dL_dz1 @ x.T
dL_db1 = dL_dz1

# Update
W2 -= lr * dL_dW2
b2 -= lr * dL_db2
W1 -= lr * dL_dW1
b1 -= lr * dL_db1

6. Final Updated Parameters

Parameter	Updated Value
`W1`	\[ \begin{bmatrix} 0.2 & -0.4 & 0.1 \\ 0.623 & 0.262 & -0.385 \end{bmatrix} \]
`b1`	\[ \begin{bmatrix} 0.1 \\ -0.277 \end{bmatrix} \]
`W2`	\[ \begin{bmatrix} 0.5 & -0.892 \end{bmatrix} \]
`b2`	\( 0.277 \)
Loss	\( \mathcal{L} \approx 1.463 \)

7. Conclusion

This example illustrates how a basic neural network performs forward and backward propagation. By following the chain rule through each layer, we calculate how much each parameter contributes to the output error. Backpropagation enables the network to learn from its mistakes by adjusting its weights and biases through gradient descent.

Once you understand this foundation, you're ready to explore deeper networks, regularization, batch training, and advanced optimizers like Adam and RMSprop. But remember, everything starts here—with a dot product, a ReLU, and a sigmoid.

Is the Derivative of a Dot Product Always the Other Vector?

Introduction

In vector calculus, one of the most commonly encountered expressions is the dot product of two vectors. A particularly elegant and useful identity is the derivative of the dot product with respect to one of its vector components: \[ \frac{\partial}{\partial \mathbf{u}}(\mathbf{u}^\top \mathbf{h}) = \mathbf{h}^\top \] This article explores when this identity is valid, why it works, and under what conditions it may fail.

The Setup

Let \( \mathbf{u}, \mathbf{h} \in \mathbb{R}^n \) be two vectors of the same dimension. The dot product between them is: \[ \mathbf{u}^\top \mathbf{h} = \sum_{i=1}^n u_i h_i \] This expression is a scalar function, and we wish to compute its gradient with respect to \( \mathbf{u} \).

Taking the Derivative

Since each term \( u_i h_i \) is linear in \( u_i \), and \( h_i \) is treated as a constant, we get: \[ \frac{\partial}{\partial u_i}(u_i h_i) = h_i \] Therefore, the full gradient vector is: \[ \frac{\partial}{\partial \mathbf{u}}(\mathbf{u}^\top \mathbf{h}) = \begin{bmatrix} h_1 & h_2 & \cdots & h_n \end{bmatrix} = \mathbf{h}^\top \] This result is both intuitive and algebraically clean.

When Is This Identity Valid?

This derivative identity is valid under the following assumptions:

\( \mathbf{u} \) and \( \mathbf{h} \) are vectors of the same length
Only \( \mathbf{u} \) is treated as a variable; \( \mathbf{h} \) is held constant
The function \( \mathbf{u}^\top \mathbf{h} \) is scalar-valued

In such cases, the dot product is symmetric, so the following also holds: \[ \frac{\partial}{\partial \mathbf{u}}(\mathbf{h}^\top \mathbf{u}) = \mathbf{h}^\top \]

When Does It Not Hold?

The identity does not hold if:

\( \mathbf{h} \) is a function of \( \mathbf{u} \); in that case, apply the product rule
The function is not scalar-valued (e.g., outer products)
The transpose is misused in row-vs-column vector contexts

For example, if \( \mathbf{h} = f(\mathbf{u}) \), then: \[ \frac{\partial}{\partial \mathbf{u}}(\mathbf{u}^\top \mathbf{h}) = \mathbf{h}^\top + \left( \frac{\partial \mathbf{h}}{\partial \mathbf{u}} \right)^\top \mathbf{u} \] This includes a second term due to the chain rule.

Conclusion

The identity \( \frac{\partial}{\partial \mathbf{u}}(\mathbf{u}^\top \mathbf{h}) = \mathbf{h}^\top \) is a powerful shortcut in linear algebra and machine learning. It simplifies many derivations, especially in backpropagation and optimization. However, it should be used with care — ensuring that the assumptions hold, particularly that the vector you're not differentiating with respect to is constant. When \( \mathbf{h} \) depends on \( \mathbf{u} \), always apply the product rule.

Jacobian Matrix Tutorial: A Step-by-Step Guide for Students

The Jacobian matrix is a central tool in multivariable calculus, used extensively in optimization, machine learning, robotics, and physics. While symbolic computation tools like SymPy automate its calculation, understanding how to compute a Jacobian by hand is essential for mastering the theory. This article provides a student-friendly, step-by-step guide to computing the Jacobian matrix manually, supported by geometric intuition and worked-out examples.

What Is the Jacobian Matrix?

Given a vector-valued function \[ \mathbf{F}(x_1, x_2, ..., x_n) = \begin{bmatrix} f_1(x_1, ..., x_n) \\ f_2(x_1, ..., x_n) \\ \vdots \\ f_m(x_1, ..., x_n) \end{bmatrix}, \] the Jacobian matrix is the matrix of all first-order partial derivatives: \[ J(\mathbf{x}) = \left[ \frac{\partial f_i}{\partial x_j} \right]_{m \times n}. \] This matrix tells us how sensitive each output \( f_i \) is to each input \( x_j \), serving as a multi-dimensional generalization of the derivative.

Step-by-Step Jacobian Calculation

Step 1: Define the Function

Let us consider a function from \( \mathbb{R}^2 \to \mathbb{R}^2 \): \[ \mathbf{F}(x, y) = \begin{bmatrix} f_1(x, y) \\ f_2(x, y) \end{bmatrix} = \begin{bmatrix} x^2 + y \\ \sin(xy) \end{bmatrix}. \]

Step 2: Compute Partial Derivatives

We compute the partial derivatives for each output function with respect to each input:

\( \frac{\partial f_1}{\partial x} = 2x \)
\( \frac{\partial f_1}{\partial y} = 1 \)
\( \frac{\partial f_2}{\partial x} = y \cos(xy) \) (using chain rule)
\( \frac{\partial f_2}{\partial y} = x \cos(xy) \)

Step 3: Assemble the Jacobian Matrix

\[ J(x, y) = \begin{bmatrix} 2x & 1 \\ y \cos(xy) & x \cos(xy) \end{bmatrix} \]

This matrix shows how small changes in \( x \) and \( y \) affect the outputs \( f_1 \) and \( f_2 \).

Worked Example 2: More Inputs than Outputs

Consider the function \( \mathbf{F}(x, y, z) = \begin{bmatrix} x + yz \\ x^2 + y^2 + z^2 \end{bmatrix} \).

Partial derivatives:

\( \frac{\partial f_1}{\partial x} = 1 \), \( \frac{\partial f_1}{\partial y} = z \), \( \frac{\partial f_1}{\partial z} = y \)
\( \frac{\partial f_2}{\partial x} = 2x \), \( \frac{\partial f_2}{\partial y} = 2y \), \( \frac{\partial f_2}{\partial z} = 2z \)

Jacobian: \[ J(x, y, z) = \begin{bmatrix} 1 & z & y \\ 2x & 2y & 2z \end{bmatrix} \]

Geometric Interpretation

The Jacobian matrix represents the best linear approximation to a function at a point. When the number of inputs equals the number of outputs, the Jacobian determinant tells us how volume (or area) is scaled under the transformation:

\[ \text{Volume}_{\text{new}} \approx |\det(J(\mathbf{x}))| \cdot \text{Volume}_{\text{original}}. \]

For example, in transforming coordinates from Cartesian to polar: \[ x = r \cos \theta,\quad y = r \sin \theta, \] the Jacobian determinant is: \[ \left| \frac{\partial(x, y)}{\partial(r, \theta)} \right| = r. \] This factor appears when integrating over circular regions.

Practice Problems

Compute the Jacobian of: \[ \mathbf{F}(x, y) = \begin{bmatrix} x \cdot y \\ \ln(x^2 + y^2) \end{bmatrix}. \]
For the coordinate transformation \( \mathbf{F}(r, \theta) = (r \cos \theta, r \sin \theta) \), compute the Jacobian and its determinant.

Python Verification (Optional)


import sympy as sp

x, y = sp.symbols('x y')
f1 = x**2 + y
f2 = sp.sin(x * y)

F = sp.Matrix([f1, f2])
vars = sp.Matrix([x, y])
J = F.jacobian(vars)
J

¹ Use sympy.Matrix.jacobian() for symbolic Jacobian computation in Python. Install with pip install sympy.

Summary Table

Concept	Description
Jacobian Matrix	Matrix of first-order partial derivatives
Rows	Number of output functions
Columns	Number of input variables
Determinant	Measures volume change under transformation
Applications	Optimization, robotics, neural networks, integration

Understanding the Jacobian Matrix: A Gateway to Multivariable Calculus, Machine Learning, and Beyond

Introduction

The Jacobian matrix is a foundational concept in multivariable calculus, used widely in machine learning, robotics, physics, and coordinate geometry. It generalizes the notion of derivatives to vector-valued functions, capturing how changes in input variables impact the outputs of a system. In this article, we unpack the Jacobian from first principles, explore its real-world applications, and tackle five fundamental questions to deepen understanding.

What Is the Jacobian?

Suppose we have a vector-valued function \[ \mathbf{F}(\mathbf{x}) = \begin{bmatrix} f_1(x_1, x_2, \ldots, x_n) \\ f_2(x_1, x_2, \ldots, x_n) \\ \vdots \\ f_m(x_1, x_2, \ldots, x_n) \end{bmatrix} \] that maps an \( n \)-dimensional input to an \( m \)-dimensional output. The Jacobian matrix is defined as: \[ J(\mathbf{x}) = \begin{bmatrix} \frac{\partial f_1}{\partial x_1} & \cdots & \frac{\partial f_1}{\partial x_n} \\ \frac{\partial f_2}{\partial x_1} & \cdots & \frac{\partial f_2}{\partial x_n} \\ \vdots & \ddots & \vdots \\ \frac{\partial f_m}{\partial x_1} & \cdots & \frac{\partial f_m}{\partial x_n} \end{bmatrix} \]

Each element \( \frac{\partial f_i}{\partial x_j} \) represents the partial derivative of output function \( f_i \) with respect to input variable \( x_j \). This matrix captures how sensitive each output is to each input, making it an essential tool in sensitivity analysis, optimization, and transformation.

1. What Is the Geometric Interpretation of the Jacobian Matrix and Its Determinant?

The Jacobian matrix is a linear approximation of a nonlinear function near a given point. Geometrically, if we consider a transformation \( \mathbf{F}: \mathbb{R}^n \rightarrow \mathbb{R}^n \), then the Jacobian determinant \( \det(J(\mathbf{x})) \) describes how a small volume near point \( \mathbf{x} \) is scaled by the transformation.

For instance, in two dimensions, suppose a small square in the \( (x, y) \) plane is transformed by \( \mathbf{F} \). The area of the resulting parallelogram is approximately: \[ \text{Area}_{\text{new}} \approx |\det(J(\mathbf{x}))| \times \text{Area}_{\text{original}} \]

A Jacobian determinant of zero means the transformation collapses dimensions (e.g., a surface becomes a line), while a negative value indicates an inversion or reflection.

2. How Is the Jacobian Used in Machine Learning?

In machine learning, the Jacobian plays a crucial role in backpropagation. During the training of neural networks, gradients of loss functions with respect to weights must be computed. If a layer maps inputs \( \mathbf{x} \) to outputs \( \mathbf{y} \), then the Jacobian \( \frac{\partial \mathbf{y}}{\partial \mathbf{x}} \) helps propagate error signals backward.

For example, in an autoencoder or recurrent neural network, the Jacobian is used to analyze vanishing or exploding gradients by examining the spectral norm of the Jacobian matrix: \[ \|J\|_2 \ll 1 \Rightarrow \text{Vanishing gradients}, \quad \|J\|_2 \gg 1 \Rightarrow \text{Exploding gradients} \]

This sensitivity analysis ensures stability during training, especially in deep networks.

3. What Is the Difference Between the Jacobian and the Hessian?

The Jacobian and Hessian are both derivative-based matrices, but they serve different roles:

Jacobian: First-order partial derivatives; applies to vector-valued functions.
Hessian: Second-order partial derivatives; applies to scalar-valued functions.

For a scalar function \( f(x_1, \ldots, x_n) \), the Hessian is: \[ H(f) = \begin{bmatrix} \frac{\partial^2 f}{\partial x_1^2} & \cdots & \frac{\partial^2 f}{\partial x_1 \partial x_n} \\ \vdots & \ddots & \vdots \\ \frac{\partial^2 f}{\partial x_n \partial x_1} & \cdots & \frac{\partial^2 f}{\partial x_n^2} \end{bmatrix} \]

While the Jacobian captures direction and sensitivity, the Hessian captures curvature. In optimization, the Hessian helps assess whether a point is a local minimum, maximum, or saddle point.

4. Real-World Application: Jacobian in Robotics

In robotics, the Jacobian connects joint parameters (like angles) to the position and velocity of an end-effector (like a robot arm’s hand). Suppose the joint angles are \( \theta_1, \theta_2, \ldots, \theta_n \), and the end-effector’s position is \( \mathbf{p} \). Then: \[ \mathbf{J}(\boldsymbol{\theta}) = \frac{\partial \mathbf{p}}{\partial \boldsymbol{\theta}} \]

This matrix maps small changes in joint angles to small changes in position: \[ \Delta \mathbf{p} \approx \mathbf{J}(\boldsymbol{\theta}) \cdot \Delta \boldsymbol{\theta} \]

When the Jacobian is singular (determinant is zero), the robot is in a singular configuration, meaning some directions of motion are inaccessible. This has implications for control and motion planning.

5. How Does the Jacobian Affect Multivariable Integration?

When changing variables in a multiple integral, such as converting from Cartesian to polar coordinates, the Jacobian determinant adjusts for the distortion caused by the transformation.

Example: Convert a double integral to polar coordinates: \[ x = r \cos \theta, \quad y = r \sin \theta \] Then the Jacobian determinant is: \[ \det(J) = \left| \frac{\partial(x, y)}{\partial(r, \theta)} \right| = r \] So, \[ \iint_D f(x, y)\, dx\,dy = \iint_{D'} f(r \cos \theta, r \sin \theta)\, r\, dr\, d\theta \]

This correction ensures that area (or volume) is properly conserved during the variable transformation.

Worked Example

Let’s consider a function:


import sympy as sp

x, y = sp.symbols('x y')
f1 = x**2 + y
f2 = sp.sin(x * y)

F = sp.Matrix([f1, f2])
vars = sp.Matrix([x, y])
J = F.jacobian(vars)
J

The resulting Jacobian matrix is:

\[ J = \begin{bmatrix} 2x & 1 \\ y \cos(xy) & x \cos(xy) \end{bmatrix} \]

This shows how output components \( f_1 \) and \( f_2 \) change with respect to the inputs \( x \) and \( y \).

Conclusion

The Jacobian matrix serves as a bridge between differential calculus and applied sciences. From machine learning to physics, it enables us to understand how systems respond to change. Whether optimizing a loss function or controlling a robotic arm, the Jacobian's power lies in its generality: it describes change across dimensions, systems, and perspectives.

Understanding the Jacobian is more than just mastering derivatives—it's about interpreting how structure, transformation, and sensitivity manifest in real-world models. As mathematical tools go, the Jacobian is both elegant and indispensable.

From Linear Limits to Nonlinear Power: Why Activation Functions Matter in Neural Networks

One of the most pivotal reasons behind the success of deep learning lies not in its depth alone, but in the introduction of non-linear activation functions between layers. This article explores fundamental questions about the role and necessity of non-linearity in neural networks, using ReLU, sigmoid, and tanh as primary examples. We also explore how this affects gradient computation through the lens of the Jacobian matrix.

1. Why Can't We Just Stack Linear Layers?

Suppose we have a series of layers in a neural network where each transformation is linear. That is, each layer performs:

z = W @ x + b

Now, suppose we stack multiple such layers:

y = W3 @ (W2 @ (W1 @ x + b1) + b2) + b3

This entire operation can be collapsed into a single linear transformation:

\[ y = W'x + b' \]

The problem? This is still just a linear function. It doesn’t matter how many layers we stack — we’re still mapping straight lines to straight lines. In mathematical terms, we have not increased the function space the network can represent.

Such a network is incapable of modeling relationships that are nonlinear in nature, such as:

Nonlinear classification boundaries (e.g., XOR problem)
Complex visual features (edges, curves, textures)
Compositional hierarchy in language or images

2. How Activation Functions Introduce Non-Linearity

To break free of this limitation, we insert a non-linear activation function \( f \) after each linear transformation:

\[ h = f(z) = f(Wx + b) \]

These non-linearities enable the model to approximate arbitrary continuous functions — a fact supported by the Universal Approximation Theorem.

2.1 ReLU (Rectified Linear Unit)

\[ f(x) = \max(0, x) \]

Piecewise linear but not globally linear.
Introduces a "kink" at \( x = 0 \), breaking the straight-line assumption.
Efficient and widely used in deep networks due to simplicity and sparsity.

ReLU: sharp transition at 0, linear growth for positive inputs.

2.2 Sigmoid

\[ f(x) = \frac{1}{1 + e^{-x}} \]

S-shaped curve mapping real values to \( (0, 1) \).
Useful in binary classification problems.
Non-zero centered output, which can lead to vanishing gradient issues.

Sigmoid: saturates at both ends, making gradients small when far from 0.

2.3 Tanh (Hyperbolic Tangent)

\[ f(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}} = \tanh(x) \]

S-shaped curve like sigmoid but zero-centered.
Preferred in networks where zero-mean activation is helpful.

Tanh: similar saturation but centered around zero, which helps convergence.

These characteristics influence the training dynamics and stability of neural networks.

3. Gradient Flow and the Jacobian Matrix

When applying activation functions elementwise, the gradient of the output \( \mathbf{h} = f(\mathbf{z}) \) with respect to \( \mathbf{z} \) is captured by the Jacobian matrix:

\[ \frac{\partial h_i}{\partial z_j} = \begin{cases} f'(z_i), & \text{if } i = j \\ 0, & \text{if } i \ne j \end{cases} \]

This results in a diagonal matrix:

\[ \frac{\partial \mathbf{h}}{\partial \mathbf{z}} = \text{diag}(f'(z_1), f'(z_2), \ldots, f'(z_n)) \]

Understanding this Jacobian is essential for backpropagation, as it determines how the loss gradient flows back through the activation function and affects earlier layers.

4. Summary: Why Non-Linearity is Indispensable

In deep learning, depth alone does not bring power. Non-linearity is the true enabler of expressiveness. Without it:

Deep networks are just shallow networks in disguise.
They can't solve problems with curved or irregular boundaries.
No compositional or hierarchical abstraction is possible.

With non-linear activation functions like ReLU, tanh, and sigmoid:

Neural networks can model complex patterns in data.
Gradient flow becomes meaningful due to differentiable non-linearities.
The expressiveness of the function space grows exponentially.

Further Exploration

Visualize the derivative of each activation function to understand gradient strength.
Experiment with deeper networks with and without non-linearities to see the difference.
Explore newer activations like Leaky ReLU, GELU, or Swish.

Non-linearity is not a "hack" — it’s the mathematical heart of why deep learning works.

Footnote:

¹ In Python, the @ symbol represents the matrix multiplication operator. It is equivalent to the dot product or linear transformation used in linear algebra. For example, in NumPy:

import numpy as np

A = np.array([[1, 2],
              [3, 4]])
B = np.array([[5, 6],
              [7, 8]])

C = A @ B  # Matrix multiplication

This is mathematically equivalent to:

\[ C = AB = \begin{bmatrix} 1 & 2 \\ 3 & 4 \\ \end{bmatrix} \cdot \begin{bmatrix} 5 & 6 \\ 7 & 8 \\ \end{bmatrix} = \begin{bmatrix} 19 & 22 \\ 43 & 50 \\ \end{bmatrix} \]

This operator is essential in neural networks, where each layer performs a linear transformation via matrix multiplication followed by a non-linear activation.

Wednesday, 21 May 2025

Understanding the GloVe Equation: A Deep Dive into Word Embedding Mathematics

Demystifying the GloVe Equation: A Deep Dive into Word Embedding Mathematics

Introduction:

Word embeddings are a cornerstone of modern Natural Language Processing (NLP), enabling machines to represent and understand human language in a structured, numeric form. Among the most influential embedding algorithms is GloVe (Global Vectors for Word Representation), which blends the statistical strength of matrix factorization with the efficiency of local context methods like Word2Vec.

At the heart of GloVe lies an elegant equation:

\( w_i^T \cdot \tilde{w}_j + b_i + \tilde{b}_j \approx \log(X_{ij}) \)

In this article, we explore the key concepts, geometric interpretation, real-world analogy, and dynamics of this equation through intuitive explanations, mathematical insights, and examples.

1. Understanding the Equation in Plain English

This equation says that the dot product of the main word vector \( w_i \) and the context word vector \( \tilde{w}_j \), plus their biases \( b_i \) and \( \tilde{b}_j \), should be approximately equal to the logarithm of the number of times word i appears near word j in the corpus (\( X_{ij} \)).

2. Role of Each Term in the Equation

\( w_i \): The embedding of the main word.
\( \tilde{w}_j \): The embedding of the context word.
Dot Product \( w_i^T \cdot \tilde{w}_j \): Measures the alignment or semantic similarity between the words.
\( b_i \) and \( \tilde{b}_j \): Biases that adjust for overall word frequency.
\( \log(X_{ij}) \): The actual signal — how often these two words appear together in the corpus.

Together, the left-hand side forms a prediction that the model adjusts to match the observed co-occurrence on the right-hand side.

3. Real-World Analogy

Consider a smart supermarket assistant trying to learn which items go together. If people often buy "bread" and "butter" together, the assistant sees this in the data as \( X_{ij} = 500 \). It learns to align their vectors such that their similarity, plus some small tweaks (biases), equals \( \log(500) \). This lets it suggest butter to customers who buy bread — learned from co-occurrence, not rules.

4. How the Equation Changes When Terms Change

Change	Effect
Increase similarity between \( w_i \) and \( \tilde{w}_j \)	Dot product increases → predicted co-occurrence increases
Decrease similarity	Dot product drops → model predicts lower association
Increase \( b_i \) or \( \tilde{b}_j \)	Shifts predicted value upward regardless of vector similarity
Increase \( X_{ij} \)	Increases \( \log(X_{ij}) \), forcing model to adjust vectors/biases

5. Key Mathematical Concepts and Insights

Dot Product: Measures similarity through vector alignment.
Matrix Factorization: Embeddings are low-rank approximations of the co-occurrence matrix.
Logarithmic Transformation: Smooths skewed frequency data, making learning easier.
Bias Terms: Adjust for raw frequency without distorting word relationships.
Least Squares Optimization: The model minimizes squared differences between the left and right sides.
Weighting Function: Helps control the influence of frequent and infrequent pairs (used during training).

6. Geometric Interpretation

The left-hand side forms a linear combination of word and context embeddings, which geometrically defines a hyperplane in the embedding space. The dot product is a projection: it tells how aligned two vectors are, while the biases shift this hyperplane. The right-hand side, \( \log(X_{ij}) \), is curved — so the model’s task is to match a logarithmic reality with a linear prediction.

7. Direct or Inverse Proportion?

This equation encodes a direct proportion between predicted similarity (left side) and observed co-occurrence (right side). As \( X_{ij} \) increases, so does \( \log(X_{ij}) \), requiring the dot product and biases to grow. This means the more often words appear together, the more the model aligns them in vector space.

8. Worked-Out Numerical Example

Let’s use concrete vectors to show the math in action:

Word \( i \): "coffee" → \( w_i = [1, 2] \)
Context word \( j \): "cup" → \( \tilde{w}_j = [3, 1] \)
Biases: \( b_i = 0.5 \), \( \tilde{b}_j = 0.2 \)
Co-occurrence count: \( X_{ij} = 20 \)

Dot product: \( 1 \cdot 3 + 2 \cdot 1 = 5 \)
Total left-hand side: \( 5 + 0.5 + 0.2 = 5.7 \)
Right-hand side: \( \log(20) \approx 2.9957 \)

The model overestimates the co-occurrence and will adjust the vectors and biases during training to bring the left-hand side closer to 3.0.

Conclusion

The GloVe equation elegantly bridges linguistic co-occurrence patterns with vector algebra. Through a mix of dot products, bias terms, and logarithmic transformation, it captures the semantics of language in geometric form. Understanding the interplay of its terms provides deep insight not just into GloVe, but into how meaning itself can be learned from raw text.

This equation is more than just math — it's a window into how machines learn to understand language.

Thinking with AI: Explanation of the Mathematical Equation

1. Can you explain this equation in plain English in one line, using the terms in brackets?

2. Can you describe this equation as a story for an 8-year-old, where each term is a character?

3. What is the role of each term in this equation, and how do they interact?

4. Can you give a real-world scenario where this equation comes to life?

5. Can you show how this equation changes when one of the terms changes?

6. What are the key mathematical concepts and insights that this equation uses?

8. Can the equation be seen as a line, curve, area, etc.?

9. Is it a linear or nonlinear equation?

10. Does the equation show a direct or inverse proportion?

11. What does this equation represent, and in what context is it used?

12. Can you break down and explain each term in the equation?

13. What are the assumptions or conditions under which this equation holds true?

14. Can you derive this equation step-by-step (or show how it was derived)?

15. Can you take a small example and show.

How Vector Differences in Word Embeddings Capture Analogical Relationships

One of the most striking properties of distributed word representations is their ability to encode analogical relationships through simple vector arithmetic. This blog explores the mathematical and linguistic foundations of this phenomenon, answers key follow-up questions, and highlights applications and limitations. Our journey begins with a central question:

Produces embeddings where vector differences capture analogical relationships. How does it happen?

Let us consider the now-famous example:

    vec("king") - vec("man") + vec("woman") ≈ vec("queen")

This relation is not hardcoded but emerges during the training of word embedding models such as Word2Vec and GloVe. These models are built on the idea that the meaning of a word is captured by the company it keeps — a hypothesis known as the Distributional Hypothesis.

Word embeddings are trained such that semantically similar words appear close together in a high-dimensional vector space. More importantly, certain linguistic relationships — such as gender, tense, or geography — are consistently reflected as linear offsets between vectors. These offsets arise from statistical regularities in word co-occurrences. For instance, the vector from “man” to “woman” is similar to the vector from “king” to “queen”, because both pairs reflect a consistent gender relationship.

Let’s now explore five deep-dive questions that further unpack this phenomenon.

1. How do cosine similarity and vector arithmetic work together in capturing analogies?

In the vector space, we care less about absolute vector positions and more about their directions. Cosine similarity — defined as:

\[ \cos(\theta) = \frac{\vec{a} \cdot \vec{b}}{\|\vec{a}\| \|\vec{b}\|} \]

— measures the angle between two vectors, not their magnitude. Analogies work by computing the difference vector \( \vec{b} - \vec{a} \) and applying it to another word vector \( \vec{c} \). The result \( \vec{d} = \vec{c} + (\vec{b} - \vec{a}) \) is expected to point in a direction similar to the target word vector. The final step involves finding the word whose vector is closest in cosine similarity to \( \vec{d} \).

2. Why do analogies work better in some cases and fail in others?

Analogical reasoning in embeddings works best when:

The vocabulary has sufficient and balanced training data.
The relationship is linear and systematic across examples.
The embedding dimension is high enough to encode complex patterns.

Failures typically occur due to:

Polysemy: Words like “bank” (river vs. finance) confuse embeddings.
Sparse Data: Rare word pairs are not learned well.
Contextual Ambiguity: Static embeddings average over senses.

For example, analogies like "Tokyo" is to "Japan" as "Cairo" is to "Egypt" are usually successful, but "bat" is to "ball" as "pen" is to "paper" may fail due to multiple interpretations or weak co-occurrence signals.

3. How does GloVe differ from Word2Vec in encoding analogical relationships?

Both GloVe and Word2Vec generate dense word vectors but differ in how they model word relationships:

Word2Vec (Skip-gram): Predicts context words given a center word, optimizing local context windows using negative sampling.
GloVe: Constructs a global co-occurrence matrix \( X_{ij} \) where each entry counts how often word \( i \) appears in the context of word \( j \), then factorizes it by minimizing:

\[ J = \sum_{i,j=1}^{V} f(X_{ij})(w_i^T \tilde{w}_j + b_i + \tilde{b}_j - \log X_{ij})^2 \]

GloVe captures the ratios of co-occurrence probabilities, which are crucial for encoding analogical directions. For example, the ratio of how often “ice” appears near “solid” vs. “gas” versus “steam” near “solid” vs. “gas” reflects semantic distinctions, and GloVe optimizes the embeddings to preserve this information geometrically.

4. Can we visualize how analogy vectors cluster in high-dimensional space?

Yes, visualizing high-dimensional embeddings is a powerful way to understand their structure. Techniques like Principal Component Analysis (PCA), t-SNE, or UMAP reduce dimensionality to 2D or 3D for visualization. When applied to analogy datasets, we often see:

Clusters of similar words (e.g., all country names, or all adjectives).
Parallel vector directions for analogous relationships (e.g., king → queen is parallel to man → woman).

In a 2D t-SNE plot, for example, you might observe:

    • "king" and "queen" form a tight cluster, with vectors pointing in a consistent direction as "man" and "woman"
    • Verb forms like "run" → "ran" and "walk" → "walked" form aligned directional shifts

This visual regularity underpins why vector arithmetic works so well for analogies.

5. How can analogical reasoning be applied in real-world NLP tasks?

Analogical reasoning through vector arithmetic is foundational to several real-world NLP applications:

Knowledge Base Completion: Filling in missing facts by learning relational vectors (e.g., from “Barack Obama” to “Michelle Obama” → spouse relation).
Semantic Search: Retrieving documents or images whose embeddings match a transformed query (e.g., “summer dress” + “wedding” → relevant results).
Dialogue Systems: Generating responses that preserve analogy-like transitions (e.g., “That’s like saying X is to Y”).
Bias Detection and Debiasing: Discovering gender or racial bias in embeddings by inspecting offset directions (e.g., “doctor” - “man” + “woman” ≠ “nurse”).

These applications leverage the embedding space’s geometric structure for relational inference, creativity, and fairness interventions.

Conclusion

The ability of word embeddings to represent analogies through vector differences is not a magical artifact but a statistical consequence of the data and training objectives. Whether using Word2Vec or GloVe, the model learns to preserve meaningful relationships in geometric form. With cosine similarity as the measuring rod and co-occurrence statistics as the source, embeddings unlock new ways to reason, infer, and explore language.

As we continue to evolve toward contextualized embeddings like BERT and GPT, the spirit of analogical reasoning remains — only now encoded in deeper, more dynamic spaces.

Understanding the GloVe Algorithm: Balancing Global Statistics with Local Efficiency

Introduction

In the field of Natural Language Processing (NLP), the development of word embeddings has revolutionized how machines interpret language. Among the most influential models are Word2Vec, Latent Semantic Analysis (LSA), and GloVe (Global Vectors for Word Representation). GloVe distinguishes itself by combining the best of both worlds—leveraging the global statistical information of matrix factorization methods like LSA, and the efficiency and contextual sensitivity of predictive models like Word2Vec. In this article, we unpack how GloVe achieves this integration, why it does not use a neural network, and the rationale behind its unique weighting function.

Matrix Factorization and LSA

Latent Semantic Analysis (LSA) is a matrix factorization method that creates word embeddings by performing Singular Value Decomposition (SVD) on a term-document or co-occurrence matrix. This process captures global co-occurrence patterns in the corpus. However, LSA faces limitations in scalability and does not consider word order or local context.

Local Context Methods and Word2Vec

In contrast, Word2Vec is a predictive model that trains embeddings using a shallow neural network. It captures local context by predicting neighboring words within a fixed-size window. This makes Word2Vec efficient and scalable, but it largely ignores global statistics—learning only from word pairs within a small context window.

GloVe: Marrying Global and Local Insights

The GloVe model addresses the shortcomings of both LSA and Word2Vec by:

Building a word-word co-occurrence matrix to capture global statistics.
Training word embeddings using an explicit loss function that relates co-occurrence counts to vector dot products.

The core idea is to encode the relationship:

\( w_i^\top \tilde{w}_j + b_i + \tilde{b}_j = \log(X_{ij}) \)

where:

\( X_{ij} \) is the co-occurrence count between word \( i \) and word \( j \)
\( w_i \), \( \tilde{w}_j \) are the word and context vectors
\( b_i \), \( \tilde{b}_j \) are their respective biases

This means GloVe aims to learn embeddings such that the dot product of two word vectors approximates the logarithm of their co-occurrence frequency.

Is GloVe a Neural Network?

No. Unlike Word2Vec, GloVe does not use a neural network. Instead, it is a regression-based model. The embeddings are learned by minimizing a weighted least-squares loss function:

\( J = \sum_{i,j=1}^{V} f(X_{ij}) \left( w_i^\top \tilde{w}_j + b_i + \tilde{b}_j - \log(X_{ij}) \right)^2 \)

Training is performed using standard optimization techniques like Stochastic Gradient Descent (SGD) or AdaGrad. No backpropagation or multi-layer architecture is involved—just gradient updates on the loss.

Why Not Use Raw Frequency as Weights?

A natural question arises: Why doesn’t GloVe simply weight the loss function by co-occurrence frequency \( x \)? After all, more frequent word pairs should presumably provide more reliable signals.

However, raw frequency has two major drawbacks:

Overweighting frequent words: Common pairs like ("the", "of") would dominate the loss, leading to embeddings that focus too heavily on semantically uninformative patterns.
Noise in rare pairs: Word pairs with very low counts are often random and noisy. Giving them equal or high importance can destabilize training.

The GloVe Weighting Function: A Balanced Compromise

To address this, GloVe introduces a weighting function:

\( f(x) = \begin{cases} \left( \frac{x}{x_{\text{max}}} \right)^\alpha & \text{if } x < x_{\text{max}} \\ 1 & \text{otherwise} \end{cases} \)

where typically \( \alpha = 0.75 \) and \( x_{\text{max}} = 100 \).

What Does This Do?

For small \( x \), the weight \( f(x) \) grows slowly, reducing the influence of noisy, rare co-occurrences.
For large \( x \), the weight saturates at 1, preventing frequent words from overwhelming the model.

This curve has been empirically validated to provide better results than both constant and linear weighting schemes.

Visual Comparison of Weighting Strategies

The chart below illustrates the difference between linear weighting and the GloVe weighting function:

Notice how the GloVe function quickly grows for small \( x \) but flattens out after \( x = 100 \). In contrast, linear weighting keeps increasing, which could lead to an imbalanced training signal.

Conclusion

GloVe elegantly merges the advantages of count-based global models like LSA with the efficiency and scalability of local context models like Word2Vec. By training on co-occurrence counts using a well-designed objective function and a carefully constructed weighting scheme, it creates word embeddings that are semantically rich, stable, and interpretable.

To summarize:

Global context is captured through the word-word co-occurrence matrix.
Training does not involve a neural network, but rather a regression-based optimization.
The weighting function ensures a balance between rare and frequent word pairs, improving both stability and performance.

Understanding GloVe provides deeper insight into how statistical and predictive methods can be harmonized to represent language more effectively. It’s a cornerstone of modern NLP research and applications, and its design continues to inspire new advances in the field.

Matrix Factorization and Dimensionality Reduction in Word Embeddings: From LSA to GloVe

Abstract: This blog explores the theoretical connection between Latent Semantic Analysis (LSA) and the GloVe algorithm through the lens of matrix factorization and dimensionality reduction. We examine how high-dimensional word relationships are compressed into dense semantic vector spaces and highlight the importance of co-occurrence statistics in shaping word meaning.

1. Introduction

Word embeddings have revolutionized natural language processing by enabling algorithms to understand and manipulate language in vector space. Two well-known approaches for generating embeddings are Latent Semantic Analysis (LSA) and GloVe (Global Vectors for Word Representation). Although developed in different contexts, both can be understood as forms of matrix factorization applied to linguistic data. This article offers a comparative analysis of LSA and GloVe, emphasizing how each reduces high-dimensional co-occurrence information into low-dimensional semantic representations.

2. Understanding Matrix Factorization in NLP

Matrix factorization techniques in NLP aim to uncover latent semantic structures hidden in large text corpora. Typically, we begin with a sparse matrix \( X \) of high dimensionality, where each entry represents the relationship between words and contexts—such as word-document frequencies or word-word co-occurrence counts.

The goal is to approximate this matrix with a product of lower-dimensional matrices. In mathematical terms, this is written as:

\( X \approx U \Sigma V^T \)

Here:

\( X \in \mathbb{R}^{m \times n} \): Original high-dimensional matrix
\( U \in \mathbb{R}^{m \times k} \), \( V \in \mathbb{R}^{n \times k} \): Lower-dimensional factor matrices
\( \Sigma \in \mathbb{R}^{k \times k} \): Diagonal matrix of singular values

This decomposition forms the basis of LSA and inspires models like GloVe, albeit with modifications to the objective and input matrix.

3. LSA: Latent Semantic Analysis

LSA is a classical method based on Singular Value Decomposition (SVD). It starts with a term-document matrix or a term-term matrix. The values are often weighted using techniques like TF-IDF to better capture importance.

The SVD of this matrix yields a set of lower-dimensional word vectors that retain the dominant patterns of word usage across documents. These vectors can be interpreted as coordinates in a "semantic space," where words with similar meanings appear closer together.

Strengths:

Captures linear semantic relationships
Uses global corpus statistics
Unsupervised and interpretable

Limitations:

Computationally expensive for large corpora
Embeddings may be sensitive to rare words
Does not model nonlinear relationships

4. GloVe: Global Vectors for Word Representation

GloVe extends the idea of matrix factorization by focusing on the co-occurrence matrix \( X_{ij} \), where each entry represents the number of times word \( j \) appears in the context of word \( i \).

The key insight of GloVe is to model the logarithm of the co-occurrence count as the inner product of word vectors:

\( w_i^T \tilde{w}_j + b_i + \tilde{b}_j \approx \log(X_{ij}) \)

Where:

\( w_i \), \( \tilde{w}_j \): Word and context vectors
\( b_i \), \( \tilde{b}_j \): Bias terms

The loss function minimized by GloVe is a weighted least squares objective:

\( J = \sum_{i,j=1}^V f(X_{ij}) \left( w_i^T \tilde{w}_j + b_i + \tilde{b}_j - \log(X_{ij}) \right)^2 \)

The weighting function \( f(X_{ij}) \) ensures that very frequent or very rare co-occurrences don't dominate the loss.

Benefits of GloVe:

Explicitly leverages global co-occurrence statistics
Captures linear substructures in word relationships
Generates dense word vectors that excel in analogical reasoning

5. High-Dimensional to Low-Dimensional Embeddings

In raw form, each word is associated with a sparse, high-dimensional vector—e.g., a vector of co-occurrence counts with all other words. For a vocabulary of 100,000 words, this means 100,000-dimensional space per word. This is both computationally intensive and semantically noisy.

Dimensionality reduction aims to find a much smaller vector—say, 100 or 300 dimensions—that still captures the word's most important relationships. This process compresses redundant or irrelevant information while preserving semantic structure.

In the resulting lower-dimensional space:

Semantically similar words are located nearby (high cosine similarity)
Analogical relationships are represented by vector arithmetic

For instance, the famous analogy:

\( \text{king} - \text{man} + \text{woman} \approx \text{queen} \)

is a direct result of the geometric properties of embeddings learned via GloVe.

6. Comparative View: LSA vs. GloVe

Feature	LSA	GloVe
Input Matrix	Term-Document or Term-Term	Word-Word Co-occurrence
Factorization	Singular Value Decomposition (SVD)	Weighted Least Squares on log co-occurrence
Use of Logarithm	Optional pre-processing	Built-in to the model
Objective	Minimize reconstruction error	Approximate log(co-occurrence)
Scalability	Less scalable to large corpora	Highly scalable and efficient

7. Example: Co-occurrence Counts and Embeddings

Consider a small vocabulary where the word “ice” co-occurs with “cold” more frequently than “steam”. GloVe would model this as:

\( X_{\text{ice,cold}} = 50 \Rightarrow \log(50) \approx 3.91 \)
\( X_{\text{ice,steam}} = 1 \Rightarrow \log(1) = 0 \)

The learned vectors are optimized so that:

\( w_{\text{ice}}^T \tilde{w}_{\text{cold}} \approx 3.91 \quad \text{and} \quad w_{\text{ice}}^T \tilde{w}_{\text{steam}} \approx 0 \)

This ensures semantic relationships are preserved geometrically.

8. Conclusion

Matrix factorization methods like LSA and GloVe provide powerful tools for learning meaningful word representations. While LSA relies on linear algebra and SVD, GloVe builds upon global co-occurrence statistics to learn vectors through a weighted regression model. Both methods exemplify the principle of reducing high-dimensional word relationships into low-dimensional vector spaces, a cornerstone of modern NLP.

As NLP continues to evolve, understanding these foundational techniques helps us appreciate the strengths and limitations of newer models like BERT and GPT that build upon these semantic representations in more complex ways.

Understanding Global Statistics in GloVe: A Deep Dive with Examples

GloVe (Global Vectors for Word Representation) is a widely used algorithm in natural language processing for learning word embeddings. What sets GloVe apart from models like Word2Vec is its use of global statistics — co-occurrence information aggregated over the entire corpus. This article explores what “global statistics” mean in GloVe, why they matter, and how they manifest in practical examples.

What Are Global Statistics in GloVe?

In the context of GloVe, global statistics refer to the comprehensive, corpus-wide counts of how often word pairs co-occur. Instead of examining a narrow window of neighboring words, GloVe analyzes the entire corpus to learn relationships between words based on co-occurrence frequencies. These statistics are organized in a co-occurrence matrix, where each entry \( X_{ij} \) indicates how often word j appears in the context of word i.

The heart of the GloVe algorithm lies in this equation:

\( w_i^T \cdot \tilde{w}_j + b_i + \tilde{b}_j \approx \log(X_{ij}) \)

Here, \( w_i \) and \( \tilde{w}_j \) are the word and context word vectors respectively, and \( X_{ij} \) is the co-occurrence count. The model tries to find word vectors such that their dot product (plus biases) approximates the logarithm of how frequently the words co-occur in the corpus.

Example Corpus

To understand how global statistics manifest, let’s consider a small sample corpus:

“I enjoy ice cream. I enjoy cold drinks. Ice is cold. Steam is hot.”

We focus on four target words: ice, steam, cold, and hot. Suppose we define a relatively large context window or treat each sentence as the context unit. We tally the number of times each word appears in the context of the others throughout the corpus. This yields a simplified co-occurrence matrix:

	ice	steam	cold	hot
ice	0	1	3	0
steam	1	0	1	3
cold	3	1	0	1
hot	0	3	1	0

These counts are global — they are accumulated over the entire dataset, not restricted to a single sentence or small context window.

Why Ratios Matter

GloVe emphasizes ratios of co-occurrence probabilities, which are more meaningful than raw counts. Consider the ratio of probabilities that a given context word (e.g., “cold” or “hot”) appears with “ice” versus “steam”:

For “ice”:

\( \frac{P(\text{cold} \mid \text{ice})}{P(\text{hot} \mid \text{ice})} = \frac{X_{\text{ice}, \text{cold}}}{X_{\text{ice}, \text{hot}}} = \frac{3}{\varepsilon} \approx \infty \)

For “steam”:

\( \frac{P(\text{cold} \mid \text{steam})}{P(\text{hot} \mid \text{steam})} = \frac{1}{3} \)

This sharp contrast in ratios reveals the temperature association of the words: “ice” relates more strongly to “cold” than “hot”, whereas “steam” relates more strongly to “hot” than “cold.” GloVe embeds these patterns into the geometry of the learned vector space.

How Global Statistics Are Computed

Let’s consider how we might construct the co-occurrence matrix in practice. Each sentence or context window is scanned, and for every pair of words, we increment their count. For example, “ice is cold” increments counts for:

\( X_{\text{ice}, \text{cold}} \)
\( X_{\text{cold}, \text{ice}} \)

When this process is applied to an entire corpus (possibly millions of sentences), the co-occurrence matrix captures global relationships between all word pairs — regardless of where in the text they appear. This is the key distinction between GloVe and Word2Vec.

GloVe vs Word2Vec: Local vs Global

Word2Vec uses a local window (typically ±5 words) to train its model using either CBOW or Skip-gram. It learns embeddings by predicting a word from its context or vice versa. In contrast, GloVe directly builds a global matrix and factorizes it using optimization techniques.

Aspect	GloVe	Word2Vec
Statistics Used	Global co-occurrence	Local context window
Learning Objective	Factorizes log co-occurrence matrix	Predicts surrounding words or targets
Strength	Captures semantic global relationships	Captures local syntactic patterns well

Semantic Meaning Through Vector Arithmetic

Because GloVe encodes global relationships, it supports rich vector operations such as:

\( \vec{\text{ice}} - \vec{\text{cold}} \approx \vec{\text{steam}} - \vec{\text{hot}} \)

This indicates that the difference between “ice” and “cold” is semantically similar to the difference between “steam” and “hot” — both reflecting the concept of “state of matter and its associated temperature.”

Conclusion

GloVe’s use of global statistics provides a powerful alternative to models based solely on local context. By building a co-occurrence matrix and learning embeddings through matrix factorization, GloVe captures rich, nuanced relationships between words. The example of “ice”, “steam”, “cold”, and “hot” demonstrates how these relationships emerge naturally from the data.

Global statistics matter because they reveal patterns that small, local windows might miss. In an age of deep learning, GloVe’s elegant use of global frequency ratios offers both theoretical clarity and practical power for understanding language.

Sunday, 25 May 2025

Understanding Computational Graphs and Backpropagation: A Beginner's Guide

1. What is a Computational Graph?

2. Why Do We Use Computational Graphs in Deep Learning?

3. Breaking a Function into a Computational Graph

4. What Do the Nodes and Edges Represent?

💡 Bonus: Code Example of a Simple Computational Graph

Conclusion

References

Keywords

Saturday, 24 May 2025

Understanding Monte Carlo Simulation: A Beginner's Guide to Probabilistic Modeling

What is Monte Carlo Simulation?

Why Use Monte Carlo Simulation?

Key Steps in Monte Carlo Simulation

Simple Python Example

Interpreting Monte Carlo Results

Applications of Monte Carlo Simulation

Advantages and Limitations

Conclusion

References

Keywords

Understanding the Hadamard Product: A Fundamental Operation in Neural Network Gradients

Introduction

What Is the Hadamard Product?

Important Characteristics

Hadamard Product in Gradient Computation

Python Implementation Using NumPy

Example 1: Hadamard Product with Vectors

Example 2: Hadamard Product with Matrices

Visualizing the Operation

Why Not Use Matrix Multiplication?

Use Cases in Deep Learning

Mathematical Intuition

Conclusion

Further Reading

Understanding Matrix Dimensions in a Basic Neural Network

1. Network Architecture

2. Shape Overview of Components

3. Forward Pass: Dimensions at Each Step

Step 1: Hidden Pre-Activation

Step 2: Hidden Activation

Step 3: Output Pre-Activation

Step 4: Output

4. Backward Pass: Gradient Dimensions

Step 1: Gradient w.r.t. \( z_2 \)

Step 2: Gradient w.r.t. Output Layer Parameters

Step 3: Gradient w.r.t. Hidden Layer Activation

Step 4: Gradient w.r.t. \( z_1 \) Using ReLU Derivative

Step 5: Gradient w.r.t. Hidden Layer Parameters

5. Python Code with Shape Comments

6. Final Thoughts

Backpropagation Demystified: A Step-by-Step Guide with Code

1. Overview of the Neural Network

2. Forward Propagation

3. Loss Function

4. Backpropagation: Computing Gradients

4.1. Calculating Gradient w.r.t. \( z_2 \)

Calculation of \[\quad \frac{\partial \hat{y}}{\partial z_2} = \hat{y}(1 - \hat{y}) \]

Step 1: Define the Sigmoid Function

Step 2: Apply the Chain Rule

Step 3: Express in Terms of Sigmoid

✅ Final Result

4.2. Gradients for Output Layer

4.3. Gradient w.r.t. Hidden Layer Output

4.4. Gradient w.r.t. \( z_1 \) (ReLU Derivative)

4.5. Gradients for Hidden Layer

5. Update Step (Gradient Descent)

6. Python Code Implementation

6. Python Code Implementation-Tensor Flow

7. Final Updated Parameters

8. Conclusion

Understanding Backpropagation Through a Basic Neural Network Example

1. Anatomy of a Basic Neural Network

2. Forward Propagation

3. Loss Computation

4. Backpropagation Step-by-Step

5. Python Implementation

6. Final Updated Parameters

7. Conclusion

`💡 Bonus: Code Example of a Simple Computational Graph`