The Skip Residual Paper: Deep Residual Learning for Image Recognition by Kaiming He et al

 “The Tale of the Vanishing Gradient and the Rise of Residuals”

The Problem: When Deeper Isn’t Better

Once upon a time in the deep learning world, going deeper was the key to success. From LeNet to AlexNet to VGG, each breakthrough had one thing in common: more layers, better performance. But then something strange happened.

Researchers at Microsoft tried stacking more layers — 30, 40, even 50 — hoping for better accuracy. But instead, the model started performing worse, not just on validation, but also on training! This wasn't overfitting. The deeper networks couldn’t even learn well in the first place.

This became known as the “degradation problem.” It was a mystery.

---

✨ The Insight: What If the Layers Just Had to Learn What Changed?

Then came the aha moment.

What if, instead of learning a completely new mapping $H(x)$, the network learned only the difference from the input? In other words, learn $F(x) = H(x) - x$, so the actual output becomes:

$$H(x) = F(x) + x$$

This is what they called residual learning.

By structuring the network to learn residuals — the part that deviates from an identity transformation — optimization became far easier. Suddenly, deep networks could be trained efficiently and effectively.

---

🔧 The Trick: Identity Shortcuts

They implemented this via identity shortcut connections, which simply add the input $x$ to the output of some stacked layers. No new parameters, no computational burden — just a smarter architecture.

This simple trick made a huge difference. They could now train networks with 152 layers (eight times deeper than VGG!), and not only did they train faster, but also achieved record-breaking accuracy.

---

🏆 The Achievement: ResNet Wins It All

Their residual networks — ResNets — swept the field:

• 1st place in ImageNet 2015 classification

• 1st place in COCO detection and segmentation

• 3.57% top-5 error on ImageNet test set with ResNet ensembles

Their deepest network, ResNet-152, had fewer parameters than VGG but was far more accurate.

They also tested their ideas on CIFAR-10, going all the way to 1,202 layers. Although too deep for the small dataset (leading to some overfitting), it proved the point — optimization was no longer a bottleneck.

---

🧩 The Legacy: A New Paradigm

This paper changed the way deep learning models are designed. The concept of residual learning inspired many future architectures, including DenseNet, Transformers, and even ViTs.

Its impact? Beyond just performance — it solved a fundamental optimization problem that held back deep networks. The ResNet paper marked a transition from engineering around depth to embracing it safely.

A Deep Analysis is as given in the link