Deep Learning Fundamentals

What is Deep Learning?

Deep Learning is a subset of machine learning that uses artificial neural networks with multiple layers (hence "deep") to learn complex patterns from data. These networks are inspired by the structure of the human brain and have revolutionized AI in the past decade.

Deep learning powers breakthrough applications like image recognition, natural language processing, speech recognition, and autonomous vehicles. It excels when you have large amounts of data and computational resources.

Neural Network Basics

The Perceptron

The simplest neural network is a single perceptron - it takes inputs, multiplies them by weights, adds a bias, and passes through an activation function:

# Simple perceptron
output = activation(sum(inputs * weights) + bias)

# Example with numpy
import numpy as np

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

inputs = np.array([0.5, 0.3, 0.2])
weights = np.array([0.4, 0.6, 0.8])
bias = 0.1

output = sigmoid(np.dot(inputs, weights) + bias)

Layers and Architecture

• Input Layer: Receives the raw data
• Hidden Layers: Learn intermediate representations
• Output Layer: Produces final predictions

The "depth" in deep learning refers to having many hidden layers, allowing the network to learn hierarchical features.

Activation Functions

• ReLU: f(x) = max(0, x) - Most popular, helps with vanishing gradients
• Sigmoid: Squashes to 0-1, used for binary classification output
• Softmax: Outputs probability distribution for multi-class
• Tanh: Squashes to -1 to 1, centered at zero

How Neural Networks Learn

Forward Propagation

Data flows through the network layer by layer, each applying weights, biases, and activations until producing an output.

Loss Functions

Measure how wrong the predictions are:

• MSE: For regression tasks
• Cross-Entropy: For classification tasks

Backpropagation

The algorithm that makes learning possible. It calculates gradients of the loss with respect to each weight using the chain rule, then updates weights to minimize loss.

# Gradient descent update rule
weight_new = weight_old - learning_rate * gradient

Optimizers

• SGD: Simple, stochastic gradient descent
• Adam: Adaptive learning rates, most popular choice
• RMSprop: Good for recurrent networks

Types of Neural Networks

Feedforward Networks (MLPs)

Information flows in one direction. Good for tabular data and simple tasks.

Convolutional Neural Networks (CNNs)

Designed for image data. Use convolution layers to detect spatial patterns like edges, textures, and objects.

import torch.nn as nn

class SimpleCNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 32, kernel_size=3)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(32, 64, kernel_size=3)
        self.fc = nn.Linear(64 * 6 * 6, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 64 * 6 * 6)
        return self.fc(x)

Recurrent Neural Networks (RNNs)

Process sequential data by maintaining hidden state. Variants include LSTM and GRU which handle long-term dependencies better.

Transformers

The architecture behind modern NLP breakthroughs like GPT and BERT. Use attention mechanisms to process entire sequences in parallel.

Building with PyTorch

PyTorch is the most popular framework for deep learning research and increasingly for production:

import torch
import torch.nn as nn
import torch.optim as optim

# Define model
model = nn.Sequential(
    nn.Linear(784, 256),
    nn.ReLU(),
    nn.Dropout(0.2),
    nn.Linear(256, 128),
    nn.ReLU(),
    nn.Linear(128, 10)
)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

# Training loop
for epoch in range(10):
    for inputs, labels in train_loader:
        optimizer.zero_grad()
        outputs = model(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
    print(f"Epoch {epoch+1}, Loss: {loss.item():.4f}")

Best Practices

• Start Simple: Begin with small networks and add complexity as needed
• Use Pretrained Models: Transfer learning saves time and data
• Regularization: Dropout, weight decay, and batch normalization prevent overfitting
• Data Augmentation: Artificially expand training data for images
• Learning Rate Scheduling: Decrease learning rate as training progresses
• Monitor Training: Use TensorBoard or Weights & Biases
• GPU Acceleration: Essential for training deep networks efficiently

Related Articles