Deep Learning with Python
Reading Notes & Math Insights
ruaburabubu • July 2026
📌 Chapter 1: What is Deep Learning?
Abstract
The AI Hierarchy: AI is the overarching field; Machine Learning is a subset of AI; Deep Learning is a specialized subfield of Machine Learning.
Core Definition: Deep learning emphasizes learning successive layers of increasingly meaningful representations of data.
The Three Requirements: To learn representations, a system needs input data, examples of expected output, and a way to measure performance (a loss function) to guide the loop.
Historical Context: Moving away from hand-crafted expert systems (Symbolic AI) toward systems that automatically extract features from raw data.
📌 Chapter 2: The Mathematical Building Blocks of Neural Networks
🚀1. Build your Neural Network with Python
import torch
import tensorflow as tf
from tensorflow import keras
from keras.datasets import mnist
from tensorflow.keras import layers
import numpy as np
import matplotlib.pyplot as plt
import sys
import os
os.environ['TF_ENABLE_ONEDNN_OPTS'] = '1'
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'
print(torch.cuda.get_device_name())
print(tf.__version__)
print(tf.config.list_physical_devices('GPU'))
# 0. Prepare the data set
(train_images, train_labels), (test_images, test_labels) = mnist.load_data()
print(train_images.shape)
print(train_labels.shape)
# Clean and reshape your data
#train_images = train_images.reshape((60000, 28 * 28))
#test_images = test_images.reshape((10000, 28 * 28))
train_images = train_images.astype("float32") / 255.0
test_images = test_images.astype("float32") / 255.0
# 1. Define the fresh model architecture
model = keras.Sequential([
tf.keras.layers.Flatten(input_shape=(28, 28)),
layers.Dense(512, activation='relu'),
layers.Dense(10, activation='softmax')
])
# 2. Compile the model
model.compile(
optimizer="adam",
loss="sparse_categorical_crossentropy",
metrics=["accuracy"]
)
#sparse categorical crossentropy use the Lable such as [3] directly and dont convert it to one-hot
#categorical crossentropy will convert the single number lable [3] to one-hot [0,0,1,0,0,0,0,0,0,]
# 4. Train the model
model.fit(train_images, train_labels, epochs=10, batch_size=128)
🚀2. 代码层的认知，它干了什么 ？
🚀2.1 Prepare the dataset
# 0. Prepare the data set
(train_images, train_labels), (test_images, test_labels) = mnist.load_data()
print(train_images.shape)
print(train_labels.shape)
# Clean and reshape your data
#train_images = train_images.reshape((60000, 28 * 28))
#test_images = test_images.reshape((10000, 28 * 28))
train_images = train_images.astype("float32") / 255.0
test_images = test_images.astype("float32") / 255.0
# 由于我们是用0~255的数字表示灰度值，0就是纯白，255就是纯黑。
# 实际这些像素的每个值都会参与到一个矩阵的计算，所以我们将它们都先处理为浮点数，除以255.0使得这个数据变得很小，
# 在训练计算过程中不至于溢出数字的表示范围而出错或者损失精度
1. MNIST is the dataset distributed as 4 gzipped binary files in a custom format called IDX. It is not those JPG or PNG files.
Four files are :
train-images-idx3-ubyte.gz → 9.9 MB (60,000 images)
train-labels-idx1-ubyte.gz → 29 KB (60,000 labels)
t10k-images-idx3-ubyte.gz → 1.6 MB (10,000 images)
t10k-labels-idx1-ubyte.gz → 5 KB (10,000 labels)
2. IDX Structure
Offset |
Size |
Content |
|---|---|---|
0~3 |
4 bytes |
Magic number: 0x00000803 (means: 3D array of unsigned bytes) 00000000 00000000 000001000 00000011 |
4~7 |
4 bytes |
Number of images: 60000 |
8~11 |
4 bytes |
Number of rows: 28 |
12~15 |
4 bytes |
Number of columns: 28 |
16~ |
4 bytes |
Raw pixel data — one byte per pixel, 0–255 0000100101001010010101011110101 |
IDX File Header tell you there are 60000 items (pictures), each item is 28 row and 28 columns (pixels), now the program knows how to slice the binary data.
例如：其中的一个数字切分出来
[
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 3 18 18 18 126 136 175 26 166 255 247 127 0 0 0 0]
[ 0 0 0 0 0 0 0 0 30 36 94 154 170 253 253 253 253 253 225 172 253 242 195 64 0 0 0 0]
[ 0 0 0 0 0 0 0 49 238 253 253 253 253 253 253 253 253 251 93 82 82 56 39 0 0 0 0 0]
[ 0 0 0 0 0 0 0 18 219 253 253 253 253 253 198 182 247 241 0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 80 156 107 253 253 205 11 0 43 154 0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 14 1 154 253 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 139 253 190 2 0 0 0 0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 11 190 253 70 0 0 0 0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 35 241 225 160 108 1 0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 81 240 253 253 119 25 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 45 186 253 253 150 27 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16 93 252 253 187 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 249 253 249 64 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 46 130 183 253 253 207 2 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 39 148 229 253 253 253 250 182 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 24 114 221 253 253 253 253 201 78 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 23 66 213 253 253 253 253 198 81 2 0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 18 171 219 253 253 253 253 195 80 9 0 0 0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 55 172 226 253 253 253 253 244 133 11 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 136 253 253 253 212 135 132 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
]
3. What does mnist.load_data() do?
When you call (x_train, y_train), (x_test, y_test) = mnist.load_data(), Keras does this:
Checks the local cache — looks in ~/.keras/datasets/ for the .gz files
Downloads if missing — fetches the 4 files from a mirror (originally LeCun’s server, now a Google Storage bucket at storage.googleapis.com/tensorflow/…)
Decompresses — gunzip the .gz archives
Parses the IDX binary format — reads the 16-byte header, then reads the remaining bytes as a flat array
Reshapes into NumPy arrays, returns a tulpe packed — 60000×28×28 for images, 60000 for labels
data = mnist.load_data()
train_set = data[0] # {(60000,28,28),(60000,)}
test_set = data[1] # {(10000,28,28),(10000,)}
x_train = train_set[0] #(60000,28,28)
y_train = train_set[1] #(60000,)
x_test = test_set[0] #(10000,28,28)
y_test = test_set[1] #(10000,)
print(type(x_train)) # <class 'numpy.ndarray'>
print(x_train.shape) # (60000, 28, 28)
print(x_train.dtype) # uint8
print(x_train[0].min()) # 0 (white background)
print(x_train[0].max()) # 255 (black ink)
print(y_train.shape) # (60000,)
print(y_train[0]) # 5 (the digit label for the first image)
所每一张图每一个像素我们用0-255数字标记，那么只需要8位就够了，也就是1个字节。Byte。28X28 = 784 像素 用 784字节 6万张图就是47,040,000 bytes 大概是 44.9MB（47,040,000/ 1024 /1024 得到）
🚀2.2 Define the fresh model architecture
model = keras.Sequential([
keras.layers.Flatten(input_shape=(28, 28)),
layers.Dense(512, activation='relu'),
layers.Dense(10, activation='softmax')
])
Sequential itself measns to handle the data layer by layer, in order.
1. A layer is a data processing station, takes input , does mathematical operations, produces output.
2. Flatten is to make 28 X 28 Input become shape (784,) a straight line vecator. (Don’t worry about the mistakes will prompt if your input is aready (784,0), it won't.)
3. Dense is to say that all input will be handled by every neuron.
4. We selects 2 Dense layers in our model, the first contains 512 neurons with activation relu; the second has 10 neurons with activation softmax.
5. Relu is to make negtive number become 0 , positive number stays the same. Sofmax is to map the number to the value from (0,1) so it can connects to probablity
🚀2.3 Model compile
# 2. Compile the model
model.compile(
optimizer="adam",
loss="sparse_categorical_crossentropy",
metrics=["accuracy"]
)
# 1. We use "adam" as the optimizer
# 2. We use "sparse_categorical_crossentropy" as the model's loss function
# 3. We use "accuracy" to judge, it only shows as a number there
🚀2.4 Model Fit
# 3. Fit and Train
model.fit(train_images, train_labels, epochs=10, batch_size=128)
# epochs is the rounds to train the total data that you input.
# batch_size is to select 128 pictures' data as a group to train at once.
(.venv) PS E:\_deepLearning\deepLearningwithPython> .....略
469/469 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9927 - loss: 0.0266
Epoch 7/10
469/469 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9941 - loss: 0.0215
Epoch 8/10
469/469 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9960 - loss: 0.0154
Epoch 9/10
469/469 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9975 - loss: 0.0113
Epoch 10/10
469/469 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9980 - loss: 0.0090
🚀3. The Mathematical Process Behind the Model
🚀3.1 The first (input) layer core network Transformations
The shape of the MNIST dataset training sample is
(60000, 28, 28).The Flatten process will make it look like
(60000, 784).The first picture info input as an vector \(x\). Its shape now is
(784,), which will be input to the first dense layer.The first layer will generate a weight array \(W^{(1)}\) containing random numbers with a shape of
(784, 512), alongside a bias vector \(b^{(1)}\). Then, the following calculation occurs: $\( \boxed{z^{(1)} = x \cdot W^{(1)} + b^{(1)}} \)\( Where the shapes line up as: \)\( (1, 784) \cdot (784, 512) + (1, 512) \rightarrow (1, 512) \)$Now the Activation Step The nonlinear activation function happens after the matrix dot product and bias addition are completely finished. It does not happen during the calculation of \(z^{(1)}\). $\( \boxed{a^{(1)} = \text{ReLU}(z^{(1)})} \)\( **Or written as a single combined textbook formula:** \)\( \boxed{a^{(1)} = \text{ReLU}(x \cdot W^{(1)} + b^{(1)})} \)\( **Note:** ReLU formula is: \)\( {\text{ReLU}(z^{(1)}_i) = \begin{cases} z_i & \text{if } z_i > 0 \\ 0 & \text{if } z_i \le 0 \end{cases}} \)\( \)\( {\text{ReLU}(z^{(1)}_i) = \max(0, z_i)} \)$
Why this order matters:
If we applied nonlinearity during the dot product, we would destroy the linear geometric relationships that matrix multiplication relies on. By computing \(Z_1\) first, the network stretches and rotates the data linearly, and then the ReLU function bends it nonlinearly so the model can learn complex, curved boundaries.
🚀3.2 The second (output) layer core network Transformations
\(a^{(1)}\) now as the input to the second layer
(1,512).Similarly, the second layer will generate a weight array (\(W_2\)) containing random numbers with a shape of
(512,10), alongside a bias vector (\(b_2\)).(For the MNIST dataset, this layer has 10 neurons corresponding to digits 0-9) Then, the following calculation occurs: $\( \boxed{z^{(2)} = a^{(1)} \cdot W^{(2)} + b^{(2)}} \)\( Where the shapes line up as: \)\( (1, 512) \cdot (512,10) + (1,10) \rightarrow (1, 10) \)$Now the Activation Step (Softmax) Because this is a multi-class classification problem, we use the Softmax activation function instead of ReLU. Softmax squashes the raw scores logits \(z^{(2)}\) into a probability distribution where all outputs are between 0 and 1, and they sum up to exactly 1. The mathematical formula for the \(i\)-th output neuron is:
\(p_i\) is the \(ith\) of the slot where the final probability is stored.
\(p\) is the prediction vector that formed by \(p_i\). An example is \([p_1 \;p_2\; p_3\; ...\; p_{10}]\)
Numerator (\(e^{z_i}\)): Exponentiates the raw score, making sure all values become positive. for \(p_i\) index i here is linked with the \(z_i\)
Denominator (\(\sum e^{z_j}\)): Sums up all exponentiated values to act as a normalization base. Dividing by this sum turns every value into a clean percentage. Example \(\sum e^{z_j}\) = \(e^{z_1}+e^{z_2}+e^{z_3}+...+e^{z_{10}}\)
🚀3.3 The Cross-Entropy Loss
If the model assigns a low probability to the true class, the loss will be HIGH. The Cross-Entropy Loss ((L)) measures the performance of a classification model by penalizing incorrect predictions using a logarithmic scale, causing the loss to approach (0) when the prediction is correct and grow exceptionally large as the prediction diverges.
\(y_i\) is True distribution（if one-hot encoded，1 for correct index, 0 otherwise）
\(p_i\) is The predicted distribution（Output of Softmax, a vector (\(a^{(2)}_i\))
\(log\) here is the nature logarithm
\(y\) is a scalar. But to explain the process well ,the \(y\) we use here is one-hot encoded (only one element is 1, all others are 0, it’s demension is the same as the number of last ouput neurons. for example [0,0,1,0,0,0,0,0,0,]) to multiply the \(log(p_i)\) (for example [0.001,0.1,0.5,0.1,0.1,0.01…]) the summation collapses to just one term: $\( {Loss = -y_i \cdot log(p_i) } \)$
Where \(p_i\) is the model’s predicted probability, where the index is \(i\).
Where \(y_i\) is the true probability where the index is \(i\).
There is only one \(y_i\) is 1 and others are 0 so the summation collapses to one term.
Numerical Example (Handwritten Digit "8")
p = [0.02, 0.01, 0.01, 0.05, 0.01, 0.01, 0.10, 0.02, 0.76, 0.01]
0 1 2 3 4 5 6 7 8 9
y = [ 0, 0, 0, 0, 0, 0, 0, 0, 1, 0]
Loss = 0*log(0.02) + 0*log(0.01) +...-1*log(0.76) + 0log(0.01) ≈ 0.274
Below for Keras W shape (784,512), if it is Pytroch then (512,784) 处理方式不同，是对偶的
Layer |
Input |
Weight |
Bias |
Linear Output (Logits) |
Activation |
Final Output |
|---|---|---|---|---|---|---|
Layer 1 (Hidden) |
\(x\) (784,) |
\(W^{(1)}\) (784, 512) |
\(b^{(1)}\) (512,) |
\(z^{(1)} = xW^{(1)} + b^{(1)}\) (512,) |
ReLU |
\(a^{(1)} = \max(0, z^{(1)})\) |
Layer 2 (Output) |
\(a^{(1)}\) (512,) |
\(W^{(2)}\) (512, 10) |
\(b^{(2)}\) (10,) |
\(z^{(2)} = a^{(1)}W^{(2)} + b^{(2)}\) (10,) |
Softmax |
\(p = \text{Softmax}(z^{(2)})\) |
LOSS |
Cross-Entropy Loss |
\(L = -\sum_{i} y_i \log(p_i)\) |
🚀3.4 Backpropagation Step by Step
We want to know how much change in p or any others in the chain will casuse how much change in Loss.
1. Loss → Output Probability Gradient (Mathematical starting point) Output is \(Loss\), an scalar , its input prediction is \(p = [p_1\quad p_2\quad p_3\quad p_4\quad...]\)
We know the \( {Loss = -\sum_{i} y_i \cdot log(p_i) } = -y_1 \cdot log(p_1)-y_2 \cdot log(p_2)-y_3 \cdot log(p_3)... \)
\(\partial Loss /\partial p_1 = -y_1 \cdot log(p_1)'+0+0+0...=-y_1/p_1\) \(\partial Loss /\partial p_2 = \;0 -y_2 \cdot log(p_2)'+0+0...=-y_2/p_2\) \(\partial Loss /\partial p_3 = \;0 + 0 -y_3 \cdot log(p_3)'+0...=-y_3/p_3\) \(... ...\) \(\partial L /\partial p = [\frac{\partial Loss}{\partial p_1}\quad\frac{\partial Loss}{\partial p_2}\quad\frac{\partial Loss}{\partial p_3}\quad\frac{\partial Loss}{\partial p_4}\quad...] = [-\frac{y_1}{p_1}\; -\frac{y_2}{p_2}\;-\frac{y_3}{p_3}\;-\frac{y_4}{p_4}\;...]\)
This tells us: “If the predicted probability for index i changes slightly, how much does the loss change?”
2. Softmax → Logits Gradient (First optimization step, in reality where it starts, not mathematically)
We know that the \(z\) is the logits prediction, means it may like \([0.01\;9\;2.33\;...]\) its shape is same as the vector \(p\)
这里开始改2026.7.26 18：14
The Softmax function maps a 10‑dimensional input vector \(z\) to a 10‑dimensional output vector \(p\). 由于p是10dimension vector z 是10 dimension vector 求偏导的时候偏导就是一个 10X10的矩阵（其实就是构成了一个雅可比矩阵），（同理其实你可以总结出scalar X 10d vector 得到的是一个10 d vector 的偏导结果集） 意味着偏p偏z 包含了 偏p1偏z 偏p2偏z，其中每个z都要从z1取到z10
The Jacobian matrix J below The Output (p) MUST be as ROW, the input (z) MUST be as column:出行，入列
\({\partial p}/{\partial z} \) here it is the Jacobian Matrix of the Softmax Function. \({\partial p_i}/{\partial z_j} \) 一般表达更具体的内部单个元素的通式
The matrix records how each output probability \(p_i\) changes when each input logit \(z_j\) changes slightly. 针对Softmax这个特定函数： $\(\frac{\partial p}{\partial z} = \partial{(\frac{e^{z}}{\sum{e^{z}}})}/{\partial z} \)\( 具体到分量 \)\( \frac{\partial p_i}{\partial z_j} = \partial({\frac{e^{z_i}}{\sum_{j=1}^{10} e^{z_j}}})/\partial{z_j} \)\( \)\( \frac{\partial p_i}{\partial z_j} = \partial({\frac{e^{z_i}}{e^{z_1}+e^{z_2}+...+e^{z_i}+...+e^{z_{10}}}})/\partial{z_j} \)\( **当你 从偏具体的任意一个pi开始例如p1，对p1将每个微分变量zj都偏一次，由于函数分子一定总是用zi与pi匹配，不会是zj中其他非zi的变量与pi匹配， 并且分母由于是 \)\sum z_j$ 所以分母里一定会包含微分变量zi,使得求偏导的过程中，微分变量Variable Differentiation要么在分子分母都有，要么只在分母中有，这两种情况。**
Remember that \(p\) is 10-d Vector and \(z\) is 10-d Vector, to each, the \(i\) and \(j\) is the index to mark the value inside.
How the Softmax Partial Derivative Works
我们用\(i\)作为下标标记\(p\)，用\(j\)作为下标标记\(z\)，而下面式子里出现的的\(z_i\)是 遍历\(z_j\)里的变量后，我们把它从\(z_j\)里面单独拎了出来写入到了函数分子位置上,因为它同\(p_i\)关联。如果你单独写出 \(\partial p_i/ \partial z_i\) 这里我们认为是针对单独从\(z_j\)里拿出来的那个指定的那个微分变量求偏，我们这里不用\(z_i\)表示所有的向量z的分量，注意区分一下概念。
1. When \(j = i\),it means the same position (same index) 映射 to \(p\) 意味着分子分母都包含了我们要偏的微分变量（就是 \(z_i\) 本身） $\(\partial p_i/\partial{z_j} =\partial p_i/\partial{z_i} = \partial({\frac{e^{z_i}}{\sum_{j=1}^{10} e^{z_j}}})/\partial{z_{i}} \quad \text{Let} \quad N = e^{z_i} , S = {\sum_{j=1}^{10} e^{z_j}} \)$
2. When \(j \neq i\),it means the offset position (different index) 映射 to \(p\) 意味着仅仅分母包含了我们要偏的微分变量 $\(\partial p_i/\partial{z_{j}} = \partial({\frac{e^{z_i}}{\sum_{j=1}^{10} e^{z_j}}})/\partial{z_{j}} \quad \text{Let} \quad N = e^{z_i} , S = {\sum_{j=1}^{10} e^{z_j}} \)\( \)\( \implies \frac{\partial N/\partial z_{j} \cdot S - \partial S/\partial z_j \cdot N } {S^2} \)\( \)\( = \frac{0 \cdot S - e^{z_j} \cdot e^{z_i} } {S^2} = \frac{- e^{z_j} \cdot e^{z_i} } {S \cdot S} = \boxed{-p_j \cdot p_i} \)$
Which means we could use above formulas to calculate the \(J_{ij}\) and we could easily have the result of \(\partial Loss / \partial z_j \;\text{to our model}\), let’s continue.
下面的例子可以自己算看看结论是什么Vector X Matrix, Shape (1,10) \(\cdot\) (10,10) is (1,10) $\( [-\frac{y_1}{p_1}\; -\frac{y_2}{p_2}\;-\frac{y_3}{p_3}\;-\frac{y_4}{p_4}\;...-\frac{y_i}{p_i}....-\frac{y_{10}}{p_{10}}] \cdot \left[ \begin{array}{cccccc} p_1 - p_1^2 & -p_1 p_2 & \dots & -p_1 p_j & \dots & -p_1 p_{10} \\ -p_2 p_1 & p_2 - p_2^2 & \dots & -p_2 p_j & \dots & -p_2 p_{10} \\ \vdots & \vdots & \ddots & \vdots & \ddots & \vdots \\ -p_i p_1 & -p_i p_2 & \dots & -p_i p_j & \dots & -p_i p_{10} \\ \vdots & \vdots & \ddots & \vdots & \ddots & \vdots \\ -p_{10} p_1 & -p_{10} p_2 & \dots & -p_{10} p_j & \dots & p_{10} - p_{10}^2 \end{array} \right]\)$
Dot Product 例如第一列的结果就是第一个元素 of the final result vector \(\delta_{2\cdot i}\) which shape is (1,10) remember： $\(\delta_{2\cdot 1} = -\frac{y_1}{p_1}\cdot (p_1 - p_1^2) -\frac{y_2}{p_2}\cdot (-p_2 \cdot p_1)-\frac{y_3}{p_3}\cdot (-p_3 \cdot p_1)...-\frac{y_i}{p_i}\cdot (-p_i \cdot p_1)... \)$
Others are similar we have this now:
Please note here that \(a_i^{(2)}\) is \(p_i\) itself 第二层经过activation出来的\(a^{(2)}\)我们写成了\(p_i\)
As we know in the 2nd (output) layer we have:
Now we want to know: \(\partial L/\partial a_k^{(1)}\) , \(\partial L/\partial W_{kj}^{(2)}\) , \(\partial L/\partial b_j^{(2)}\)
1. Backward Pass in the 2nd layer reaches \(W^{(2)} , b^{(2)}\) : For Tensorflow ,kj order is input index, output index (here), in PyTorch kj order is output index, input index, 我们展示分量下标\(kj\)，只是为了方便捋清输入输出关系，方便知道每个偏导的元素是怎么算的，但是不是为了只求一个元素的偏导结果，我们更关心 \(\partial z_1/ \partial W , \partial z_2/ \partial W \dots \partial z_j/ \partial W \)
Loss
/ | \
/ | \ ◀── [Sum over 'i'] Loss depends on all outputs
p₁ p₂ p₃
\ | /
\ | / ◀── [Softmax effect] All 'p' look at z_j
│z_j│
│
▼ ◀── [Single connection] Only W_kj affects z_j
W_kj
This matches the shapes from your earlier example where (h) is a vector of size 512 and (\delta _{2}) is a vector of size 10:TensorFlow Layout ((h^T \delta_2)): Here, (h) acts as the left vector (rows) and (\delta _{2}) acts as the right vector (columns). The output shape is (512, 10).PyTorch Layout ((\delta _{2}h^{T})): Here, (\delta _{2}) acts as the left vector (rows) and (h) acts as the right vector (columns). The output shape is inverted to (10, 512).
🧮 Mathematical Rule
If you have a column vector \(u\) of size \((M, 1)\) on the left, and a row vector \(v^T\) of size \((1, N)\) on the right, the matrix multiplication \(u v^T\) yields an \((M \times N)\) matrix:
Every entry at index \((i, j)\) in the final grid is strictly calculated by multiplying the \(i\)-th item from the left vector by the \(j\)-th item from the right vector.
🔄 The True Matrix Transpose Rule If you flip the order of items in a matrix multiplication, you must transpose each individual item and reverse their sequence to keep it mathematically equivalent.The correct identity is for dot product , but for outter shape is stil changed (512,10) to (10,512) they are different matrix: $\((h^{T}\delta _{2})^{T} \implies \delta _{2}^{T}h\)$ This means (\delta ^{T}h) is actually the transpose of the TensorFlow gradient matrix, which evaluates to a shape of (10, 512),which is the PyTorch layout!
2. Backward Pass in the 1st layer need to through \(a^{(1)}\) : $\( \delta_j^{(2)} = \frac{\partial \text{Loss}}{\partial z_j^{(2)}} = \sum_{i} \frac{\partial \text{Loss}}{\partial a_i^{(2)}} \cdot \frac{\partial a_i^{(2)}}{\partial z_j^{(2)}} ？？？？？？ \)$
？？？？？？？？？？？？
3. Logits \(z_2\) → Weights \(W_2\) Gradient $\(\frac{\partial Loss}{\partial W_2} = \frac{\partial Loss}{\partial p} \cdot \frac{\partial p}{\partial z_2} \cdot \frac{\partial z_2}{\partial W_2} =\frac{\partial Loss}{\partial z_2} \cdot \frac{\partial z_2}{\partial W_2} = \delta_2 \cdot \frac{\partial z_2}{\partial W_2}\)$
用一个例子理解数学上对\(W_2\)求偏导的过程（实际机器中不会构造这个过程） \(A\) Shape (3,) as Input (1,3) \(W\) Shape (3,2) as Weight Input 3 output 2 \(Z\) shape (2,) as Output Forward : \(Z = A \cdot W + B\) B 先忽略 \(Z_1 = A_1 \cdot W_{11} + A_2 \cdot W_{21} + A_3 \cdot W_{31} \) \(Z_2 = A_1 \cdot W_{12} + A_2 \cdot W_{22} + A_3 \cdot W_{32} \) Output is \(Z = [Z_1 \quad Z_2]\)
\( \partial Z_1/ \partial W_{11} = A_1 + 0 + 0\) \( \partial Z_1/ \partial W_{21} = 0 + A_2 + 0 \) \( \partial Z_1/ \partial W_{31} = 0 + 0 + A_3 \) \( \partial Z_1/ \partial W_{12} = 0 \) \( \partial Z_1/ \partial W_{22} = 0 \) \( \partial Z_1/ \partial W_{32} = 0 \) Output is
\( \partial Z_2/ \partial W_{11} = 0 \) \( \partial Z_2/ \partial W_{21} = 0 \) \( \partial Z_2/ \partial W_{31} = 0 \) \( \partial Z_2/ \partial W_{12} = A_1 + 0 + 0 \) \( \partial Z_2/ \partial W_{22} = 0 + A_2 + 0 \) \( \partial Z_2/ \partial W_{32} = 0 + 0 + A_3 \) Output is $\( \left[ \begin{array}{cccccc} 0& A_1\\ 0& A_2\\ 0& A_3 \end{array} \right] \)\( 以上两个结果切片可以构成一个(2,3,2) Rank 3 Tensor \)\( \left[ \begin{array}{cc} \left[ \begin{array}{cc} A_1 & 0 \\ A_2 & 0 \\ A_3 & 0 \end{array} \right] & \left[ \begin{array}{cc} 0 & A_1 \\ 0 & A_2 \\ 0 & A_3 \end{array} \right] \end{array} \right] \)$
从这个结果看本质上我们不需要进行求导构造三阶张量的计算，直接写出如下观察到的结果 $\(\frac{\partial Loss}{\partial W_2} = A^T \cdot \frac{\partial Loss}{\partial z_2} = A^T \cdot \delta \)$
Now we can adjust \(W_2\) to adjust the training effect of this layer. We have marched alot to reach here, let’s see what still waits us. If you want to backpropagate to the above layer, till the end of the world, you need to find \(\partial Loss / \partial a_1\) next. Because it comes from above layer (as its output).You will have these waiting for you:
回过头从P传递到W2是这样： $\( \frac{\partial Loss}{ \partial W_2} = \frac{\partial Loss}{ \partial p} \cdot \frac{\partial p}{ \partial z_2} \cdot \frac{\partial z_2}{ \partial W_2} \cdot \)$
这个链条正是我们在输出层执行的计算：
∂Loss / ∂p → 交叉熵对概率的导数 ∂p / ∂z₂ → Softmax 的雅可比矩阵 ∂z₂ / ∂W₂ → 输出层权重导数（即 a₁ᵀ）
如果从P开始传递到W1是这样： $\( \frac{\partial Loss}{ \partial W_1} = \frac{\partial Loss}{ \partial p} \cdot \frac{\partial p}{ \partial z_2} \cdot \frac{\partial z_2}{ \partial a_1} \cdot \frac{\partial a_1}{ \partial z_1} \cdot \frac{\partial z_1}{ \partial W_1} \cdot \)$
这个链条从输出层一路推到输入层：
∂Loss / ∂p → 交叉熵对概率的导数 ∂p / ∂z₂ → Softmax 雅可比矩阵 ∂z₂ / ∂a₁ → W₂ᵀ ∂a₁ / ∂z₁ → ReLU 导数 结果是1和0分段函数 ∂z₁ / ∂W₁ → xᵀ
反向传播完整链式推导（从输出层到输入层）
每一行代表一个独立的链式步骤，从左到右逐层展开。
输出层
步骤 |
链式表达式 |
作用 |
结果 |
|---|---|---|---|
1 |
|
Loss 对输出概率 p 的导数 |
稀疏向量 |
2 |
|
输出概率 p 对 logits z₂ 的导数 |
矩阵（数学上存在，代码中跳过） |
3 |
|
Loss 对 logits z₂ 的导数 |
|
4 |
|
更新输出层权重 W₂ |
|
5 |
|
更新输出层偏置 b₂ |
|
6 |
|
误差从输出层传回隐藏层输出 a₁ |
|
隐藏层（穿过 ReLU）
步骤 |
链式表达式 |
作用 |
结果 |
|---|---|---|---|
7 |
|
误差穿过 ReLU（从 a₁ 到 z₁） |
|
8 |
|
更新隐藏层权重 W₁ |
|
9 |
|
更新隐藏层偏置 b₁ |
|
符号对照表
符号 |
含义 |
形状（以你的模型为例） |
|---|---|---|
|
Softmax 输出概率 |
|
|
one‑hot 真实标签 |
|
|
输出层 logits |
|
|
输出层误差向量 |
|
|
隐藏层输出 |
|
|
输出层权重 |
|
|
输出层权重转置 |
|
|
输出层偏置 |
|
|
隐藏层 logits |
|
|
输入数据 |
|
|
隐藏层权重 |
|
|
隐藏层偏置 |
|
|
逐元素相乘（Hadamard 积） |
— |
检查点：每一层的形状是否匹配？
第 4 步：
a₁ᵀ(1,512)×δ₂(10,)→(512,10)，匹配W₂✅第 6 步：
δ₂(10,)×W₂ᵀ(512,10)→(512,)，匹配a₁✅第 7 步：
∂Loss/∂a₁(512,)⊙ReLU'(z₁)(512,)→(512,)✅第 8 步：
xᵀ(1,784)×∂Loss/∂z₁(512,)→(784,512)，匹配W₁✅