Lecture 12.#

Backward propagation#

Backward propagation, often referred to as backpropagation, is a fundamental concept in training artificial neural networks, especially deep learning networks. It’s an optimization algorithm used for minimizing the error in the predictions of the network. The basic idea can be summarized as follows: 用来算NN里某一个参数的偏导数的(with chain rule)

  1. Feedforward:

    • Start by passing a training sample through the network (layer by layer) to obtain the prediction.

    • Compute the error (or loss) which is a measure of the difference between the network’s prediction and the actual target.

  2. Backward Pass:

    • Propagate the error backward through the network.

    • Starting from the output layer, compute the gradient of the loss with respect to each neuron’s output.

    • Use these gradients to adjust the weights and biases to minimize the error.

  3. Chain Rule of Calculus:

    • Backpropagation makes extensive use of the chain rule from calculus to compute the gradient of the loss with respect to each parameter (weight or bias).

    • By understanding how much each neuron in the previous layer contributed to the error, you can adjust the weights accordingly.

  4. Weight Update:

    • Once the gradients are calculated, they are used to adjust the weights and biases. This is typically done using an optimization algorithm like gradient descent or one of its variants.

    • The weights are adjusted in the direction that reduces the error.

  5. Iterative Process:

    • This entire process (feedforward, compute error, backpropagate the error, adjust weights) is repeated many times (usually on different subsets of the training data) until the network performs well and the error is minimized.

In essence, backpropagation is a method of teaching a neural network how to correct its mistakes. By understanding which weights in the network contribute more to the error, and in which direction they need to be adjusted, the network can iteratively improve its predictions.

Stochastic Gradient Descent (SGD), Batch Gradient Descent (GD), and Mini-batch SGD#

The slide you provided appears to be discussing the distinctions and trade-offs between Stochastic Gradient Descent (SGD), Batch Gradient Descent (GD), and Mini-batch SGD. Let’s break down the main points:

  1. Stochastic Gradient Descent (SGD) works on single instances:

    • Instead of looking at the entire dataset to compute the gradient, SGD randomly picks one instance from the dataset at each step and computes the gradient based solely on that single instance.

    • High variance in gradients: Because it uses only one instance, the direction of the gradient can vary wildly between iterations.

    • Many, quick updates: It frequently updates the model (for each instance), which can lead to faster convergence, but can also cause the algorithm to oscillate and potentially miss the global minimum.

  2. Gradient Descent (GD) works on whole datasets:

    • It computes the gradient based on the entire dataset.

    • Stable update, but slow: Since it considers all training samples, the gradient direction is more stable (less variance), but this is computationally expensive and hence slower.

    • Computationally expensive: Requires processing the whole dataset to make a single update, which can be prohibitive for large datasets.

  3. Compromise: Mini-batch SGD (often just called “SGD”):

    • Instead of the extremes (one instance or the entire dataset), mini-batch SGD takes a middle ground by using a subset of the dataset (a mini-batch) to compute the gradient.

    • Process batches of size between 1 and m (where m is the number of instances in the dataset): For instance, a common mini-batch size might be ( b = 100 ).

    • Balances computation and stability: It strikes a balance between the computational efficiency of SGD and the stability of batch GD.

    • Can be parallelized over a cluster of GPUs: One of the biggest advantages of mini-batch SGD is that the computation over each mini-batch can be parallelized, which is particularly beneficial with modern GPU architectures. This allows for faster training times, especially when the mini-batch size is optimized for the GPU’s memory.

In essence, the slide is highlighting the trade-offs of these three gradient descent strategies and emphasizing that Mini-batch SGD often provides a good balance between speed and stability, making it a popular choice in deep learning and other large-scale machine learning tasks.

Alright, I’ll provide a summary of the previous explanation in Chinese:

这个幻灯片讨论了随机梯度下降（SGD）、批量梯度下降（GD）和小批量 SGD 之间的区别和权衡：

  1. 随机梯度下降（SGD）针对单个实例工作：

    • SGD 随机从数据集中选择一个实例，并仅基于该单个实例计算梯度。

    • 梯度的方差大： 由于它只使用一个实例，梯度的方向在迭代之间可以变得非常不稳定。

    • 多次、快速的更新： 它频繁地更新模型，可能导致更快的收敛，但也可能导致算法震荡并可能错过全局最小值。

  2. 批量梯度下降（GD）针对整个数据集工作：

    • 它基于整个数据集计算梯度。

    • 更新稳定但慢： 由于它考虑了所有的训练样本，梯度的方向更加稳定，但这在计算上是昂贵的，因此速度较慢。

    • 计算成本高： 需要处理整个数据集来进行单次更新，对于大数据集可能是禁止性的。

  3. 折中方案：小批量 SGD（通常简称为 “SGD”）：

    • 小批量 SGD 通过使用数据集的一个子集（一个小批量）来计算梯度，取了中间的路线。

    • 处理大小在 1 和 m 之间的批次（其中 m 是数据集中的实例数）： 例如，常见的小批量大小可能是 100。

    • 平衡计算和稳定性： 它在 SGD 的计算效率和批量 GD 的稳定性之间取得平衡。

    • 可以在 GPU 集群上并行化： 小批量 SGD 的一个最大的优点是，每个小批量上的计算可以并行化，尤其在现代 GPU 架构中尤为有益。这允许更快的训练时间，尤其是当小批量大小针对 GPU 的内存进行优化时。

简而言之，这个幻灯片强调了这三种梯度下降策略的权衡，并强调小批量 SGD 经常提供速度和稳定性之间的良好平衡，使其成为深度学习和其他大规模机器学习任务中的热门选择。

A simple real-world analogy to explain SGD, GD, and Mini-batch SGD:

Imagine you’re trying to find the deepest part of a large lake.

  1. Stochastic Gradient Descent (SGD):

    • This is like sending a single diver into the lake to find the deepest point. The diver dives at a random spot, measures the depth, comes up, and then dives again at another random spot. This method might find the deepest part quickly, but there’s also a chance the diver keeps missing it because they’re only checking one spot at a time. There’s a lot of variance in where the diver checks.

  2. Batch Gradient Descent (GD):

    • This is like draining the entire lake and then accurately measuring the depth at every single point. It’s very thorough and will definitely find the deepest part, but it’s time-consuming (draining the whole lake) and resource-intensive.

  3. Mini-batch Gradient Descent:

    • This is a compromise between the two methods. Imagine sending a small team of divers to different parts of the lake. Each diver checks a spot, and then they all come up and share their findings. Based on their combined knowledge, they dive again in promising areas. This method is faster than checking the entire lake all at once but is more systematic and reliable than just one diver checking random spots. It’s a balance of speed and accuracy.

In machine learning, the “deepest part of the lake” is the optimal set of parameters (weights) that make the model’s predictions as accurate as possible. The “divers” are equivalent to the data samples used to adjust these parameters.

I hope this analogy helps clarify the concepts for you!

Deep Neural Networks (DNNs)#

This slide seems to be discussing some important points about training Deep Neural Networks (DNNs). Let’s break it down:

  1. Flexibility and the Universal Approximation Theorem:

    • DNNs are known for their flexibility. The Universal Approximation Theorem essentially says that a neural network, even a single-layer one, given the right parameters, can approximate any continuous function to a desired accuracy. This gives neural networks the potential to model complex patterns and relationships in data.

  2. Over-parameterization and Overfitting:

    • The flip side of this flexibility is that DNNs have a large number of parameters (weights and biases). This is referred to as over-parameterization.

    • Over-parameterization can lead to overfitting. Overfitting happens when a model is so tuned to the training data that it starts capturing noise rather than the underlying pattern. As a result, it performs poorly on unseen data or test data.

  3. Initialization of Weights:

    • One common practice in training DNNs is to initialize the weights with small random values centered around zero. Proper initialization is crucial to ensure that training converges efficiently.

  4. Implicit Regularization and Early Stopping:

    • Regularization techniques are used to prevent overfitting. While there are explicit methods like L1 or L2 regularization, early stopping is an example of implicit regularization.

    • Early stopping means that you halt the training process before it has fully minimized the loss on the training data, usually once performance on a validation set starts to degrade. By doing this, you’re preventing the model from overfitting to the training data.

  5. Shrinking DNNs Towards Linear Models with Some Activation Functions:

    • Certain activation functions, when used in DNNs, can make the network behave more like a linear model, especially if the weights are small. This could be due to saturation effects, where parts of the network don’t change much during training, effectively becoming linear.

    • This point seems to hint at an interesting aspect of neural network behavior, suggesting that under certain conditions, DNNs might simplify their representation.

The slide touches upon essential aspects of DNN training, emphasizing both their capabilities and challenges, and shedding light on practical techniques used in the field.

Certainly! Let’s delve deeper into the statement: “With some activation functions, this shrinks the DNN towards a linear model.”

Activation functions introduce non-linearities into neural networks, allowing them to capture complex relationships. However, certain behaviors of activation functions can cause parts of the network to behave in a more linear fashion.

  1. Saturation of Activation Functions:

    • Some activation functions, like the sigmoid or tanh, saturate for values that are too large or too small. In these saturated regions, the gradients are close to zero.

    • During backpropagation, when gradients are close to zero, the weight updates become minimal. This means that these neurons stop learning and adjusting their weights effectively.

    • When neurons are in this saturated state, they essentially become linear or non-responsive, meaning they don’t contribute much to the network’s non-linear capabilities.

  2. Initialization and Regularization:

    • If the weights of a network are initialized too small or if some form of weight regularization pushes the weights towards smaller values, the pre-activation values (before applying the activation function) might stay in the range where the activation function behaves almost linearly. For example, in the middle range of the sigmoid or tanh functions, the behavior is approximately linear.

  3. ReLU Activation Function:

    • The Rectified Linear Unit (ReLU) activation function, which is defined as (f(x) = max(0, x)), is piecewise linear. While it introduces non-linearity (because of the zeroing out of negative values), its behavior is fundamentally linear for positive values. If most of the neurons in a network using ReLU are activated (have positive pre-activation values), the network behaves more like a linear model.

In summary, while the primary role of activation functions is to introduce non-linearities into neural networks, certain circumstances, like saturation or the inherent properties of the activation function, can cause parts of the network to behave in a more linear manner.