9  Session 9: How a Neural Network Learns, and Why Training Fails

Note

Sources for this page: d2l.ai, Dive into Deep Learning, Multilayer Perceptrons — CC BY-SA 4.0; Wikipedia, Backpropagation, Stochastic gradient descent, Overfitting, Vanishing gradient problem, Regularization (mathematics), Dropout (neural networks), Early stopping — CC BY-SA 4.0. Datasets: the Breast Cancer Wisconsin (Diagnostic) data set (UCI Machine Learning Repository, CC BY 4.0), loaded through scikit-learn; the GB1 fitness landscape of Wu, Dai, Olson, Lloyd-Smith & Sun (2016), eLife 5:e16965 (CC BY 4.0), as distributed by the FLIP benchmark (Dallago et al. 2021, bioRxiv); Session 8’s UniProt subcellular-localization data. Ioffe & Szegedy (2015), “Batch Normalization,” arXiv:1502.03167; Srivastava et al. (2014), “Dropout,” JMLR 15:1929-1958; Kingma & Ba (2015), “Adam: A Method for Stochastic Optimization,” arXiv:1412.6980; Rost & Sander (1993), J. Mol. Biol. 232:584-599 and PNAS 90:7558-7562 (the PHD method). 3Blue1Brown’s neural-network and cross-entropy videos are embedded as further watching. The regression setup (linear baseline vs. 64/64 network, L2 and dropout), the vanishing-gradient demonstration and the tiny hand-computed backpropagation example follow ideas from Magnus Ekman’s Learning Deep Learning (ch. 3, 5 and 6 slide decks in slides/), a copyrighted, not openly licensed textbook, credited here as an internal source and not linked. Every number below was computed in code and read back from the notebook’s output.

TipGeneralisation

I have a machine learning joke, but it doesn’t perform as well on a new audience.

9.1 Learning goals

By the end of this session you should be able to:

  • describe the training cycle (predictions → loss → gradients → update) and write the standard PyTorch training loop;
  • explain backpropagation as the chain rule applied from the loss backwards through the network;
  • choose the loss for classification (cross-entropy) and regression (mean squared error), and explain why confident mistakes are expensive;
  • recognise three common ways training fails (equal starting weights, vanishing gradients, a poor optimiser or learning rate) and name the fix for each;
  • judge when a hidden layer pays off, and why a very large network can lose to a simple baseline;
  • use early stopping, weight decay and dropout against overfitting, and evaluate the result honestly.

Sessions 7, 8 and 9 form one progression. Session 7 asked what a neural network can represent. Session 8 asked how we know whether a model generalises. This session asks how we actually train a network, and why training sometimes succeeds and sometimes fails.

The starting point is concrete. In Session 8 we built a network, but its weights were random, so its predictions were no better than chance. We had labelled examples, so how do we turn the random weights into useful predictions? This session answers that question with four datasets. Breast-cancer biopsies introduce the training loop, and a protein fitness landscape shows why training fails and when a hidden layer pays off. Session 8’s protein localisation data show what happens when a network is too large for its data, and blood-cell images are used in the lab.

9.2 From an untrained network to a trained network

We start with a problem on which training works well. The Wisconsin breast-cancer dataset has 569 fine-needle biopsies, each described by 30 features measured on microscope images of the cell nuclei (radius, texture, perimeter, smoothness, concavity and so on). Each biopsy is labelled malignant (212) or benign (357). We split the data into 398 training, 85 validation and 86 test biopsies, and standardise the features with statistics from the training set only (Session 8). The network has one hidden layer of 16 ReLU units and one output, 513 weights in all.

Training the network means repeating one cycle over and over:

Figure 9.1: The training cycle. The network turns inputs into predictions; the loss measures how wrong they are compared with the labels; backpropagation computes the gradient of the loss with respect to every weight; the optimiser updates the weights. The same five lines of PyTorch (right) do this for every mini-batch, in every epoch.

Before training, the network’s predicted probabilities all sit close to 0.5, so the network is guessing (validation accuracy 0.659, roughly the share of benign biopsies). After 100 passes through the training data, the benign biopsies are pushed towards 0 and the malignant ones towards 1:

Figure 9.2: Left: training and validation loss per epoch. Both fall quickly; after about 50 epochs the validation loss stops improving while the training loss keeps falling, and we keep the weights from the epoch with the lowest validation loss (early stopping). Right: predicted probability of malignancy for the 85 validation biopsies, before and after training.

On the test set the trained network reaches an accuracy of 0.953, an MCC of 0.900 and an AUROC of 0.984. Only 2 benign biopsies are called malignant, and only 2 malignant ones are missed.

Was the hidden layer needed? Not here. We compare the models by 5-fold cross-validation, because a single test split of 86 biopsies is too noisy for this comparison (Session 8). Logistic regression with 31 weights and the network with 513 both reach an accuracy of 0.979. Malignant and benign nuclei differ mainly in size and shape, and a weighted sum of those measurements already separates them. Training therefore works, but the extra capacity has nothing to add. Section 9.6 shows a dataset where it does.

9.3 How backpropagation finds every gradient

Gradient descent (Session 1, Session 7) needs \(\partial L / \partial w\), the slope of the loss, for every weight. For a single layer that was easy. But what about a weight buried several layers inside a network, whose effect on the loss passes through every later layer? The tiny network below shows how backpropagation answers this question.

Figure 9.3: A tiny network: two inputs, one tanh hidden unit \(h\), one sigmoid output \(\hat y\), and the loss \(L\). The forward pass (grey) computes the prediction and the loss. The backward pass (red) applies the chain rule from the loss back towards the inputs: \(\partial L/\partial \hat y\) is computed once and reused for \(w_3\), then multiplied by \(\partial \hat y/\partial h\) and reused for \(w_1\) and \(w_2\).

For \(w_1\), the chain rule multiplies the local slopes along the path from the loss back to the weight:

\[ \frac{\partial L}{\partial w_1} = \frac{\partial L}{\partial \hat y}\;\frac{\partial \hat y}{\partial h}\;\frac{\partial h}{\partial w_1} \]

Backpropagation is an efficient application of the chain rule, from the loss backwards through the network. Each layer’s “error signal” is computed once and reused for all the weights feeding into it, so computing all the gradients costs about as much as one forward pass. Without that reuse, networks with thousands or millions of weights could not be trained.

In practice, you do not have to differentiate networks by hand. The notebook does it once for the tiny network above, working out the five gradients with the chain rule. It then builds the same network in PyTorch and calls loss.backward(), and all five gradients agree to eight decimal places. From then on, PyTorch’s autograd does the work for any network.

Putting these pieces together gives the training loop that you will write yourself in the lab:

for epoch in range(n_epochs):
    for xb, yb in loader:                 # one mini-batch at a time
        optimizer.zero_grad()             # 1. clear the old gradients
        logits = model(xb)                # 2. forward pass
        loss = criterion(logits, yb)      # 3. how wrong is it?
        loss.backward()                   # 4. backpropagation: every gradient
        optimizer.step()                  # 5. update every weight

Real training uses mini-batches. A DataLoader shuffles the training set every epoch and serves it in small batches (here 32 biopsies), and the weights are updated after every batch. Each epoch (one pass through the training data) therefore consists of many cheap, slightly noisy steps instead of one expensive exact step. This is stochastic gradient descent. After every epoch we measure the validation loss and keep the best weights (early stopping).

9.3.1 Further watching

Backpropagation, intuitively (3Blue1Brown, Deep Learning ch. 3): the same chain-rule idea as above, built up visually one layer at a time. (Chapter 1, But what is a neural network?, is embedded on Session 7.)

Optional video: Backpropagation calculus (3Blue1Brown, Deep Learning ch. 4, 10 min), the full chain-rule mechanics for a general multi-layer network.

9.4 Which loss to minimise

The training loop minimises a loss, and the loss decides what “wrong” means. For regression we use the mean squared error between prediction and target. For classification we use cross-entropy, which for one example is the negative log of the probability that the model gave the true class,

\[ \text{CE} = -\log p(\text{true class}), \]

that is, its surprise in the language of Session 1. Averaged over a dataset, cross-entropy is the negative log-likelihood, so minimising it is the same as maximum likelihood (Session 1).

Figure 9.4: Cross-entropy as a function of the probability given to the true class. A confident correct answer costs almost nothing; a confident mistake costs a lot.

Cross-entropy punishes a confident mistake far more than hedging. A probability of 0.9 on the true class costs 0.105, but a probability of 0.01 costs 4.6, over 40 times more. This is what pushes a classifier to separate the classes, as in the right panel of the breast-cancer figure. In PyTorch, nn.BCEWithLogitsLoss (for two classes) and nn.CrossEntropyLoss (for more classes) combine the sigmoid or log-softmax with this loss in one numerically stable step, so the model outputs raw logits (Session 8). The notebook checks both by hand. The lab compares cross-entropy with the mean squared and mean absolute error for the same blood-cell classifier, and cross-entropy trains best.

Cross-entropy is built from the relative entropy (Kullback-Leibler divergence) of Session 4. It is the entropy of the true distribution plus the extra cost of describing it with the model’s distribution \(Q\): \(H(P, Q) = H(P) + D_{\text{KL}}(P \| Q)\). Since \(H(P)\) does not depend on the model, minimising cross-entropy is the same as minimising the KL divergence between the model’s predictions and the true labels.

Optional video: But what is cross-entropy? (3Blue1Brown, Compression is Intelligence part 2, 34 min), cross-entropy from the information-theory side, built up from first principles.

9.5 Why training fails

The training loop always runs, but it does not always learn. This section shows three common failures. Each is produced by changing one thing in a network that predicts the fitness of protein variants, using the GB1 data of Section 9.6 (20,000 training variants, each described by the one-hot encoding of four residues). We measure the validation error as the mean absolute error (MAE) of the predicted \(\log_{10}\) fitness. As reference points, predicting the training mean for every variant gives an MAE of 0.488, and a working 64/64 network reaches 0.250 within 20 epochs.

Figure 9.5: Four experiments. (1) Gradient size per layer in a freshly initialised network with six hidden layers: with sigmoid, the first layer receives about ten million times less gradient than the last. (2) Starting every weight at the same value. (3) Six hidden layers with sigmoid, tanh or ReLU, trained with plain gradient descent (SGD), and sigmoid trained with Adam. (4) Optimiser and learning rate. The dotted line is the error of predicting the mean.

9.5.1 Failure 1: every weight starts the same

  • Observation. Start all weights at 0.1 instead of small random numbers. After 20 epochs the validation MAE is 0.352, little better than the additive (linear) model’s 0.371, against 0.250 from a random start.
  • Why? Every hidden unit computes the same thing and receives the same gradient, so the units stay identical for ever. After training, all 64 units in the first layer are still exactly the same, so 64 units behave like one.
  • Fix. Random initialisation breaks the symmetry, and PyTorch does this by default.

9.5.2 Failure 2: deep sigmoid networks, and vanishing gradients

  • Observation. A network with six hidden layers of sigmoid units, trained with plain SGD, does not learn at all. Its validation MAE stays around 0.48, the error of predicting the mean. The same network with tanh (0.321) or ReLU (0.312) does learn.
  • Why? Backpropagation multiplies one activation derivative per layer, and the derivative of the sigmoid is at most 0.25. Even in the best case, six layers therefore multiply the gradient by \(0.25^6 \approx 0.00024\). Measured in the freshly initialised network, the first layer’s gradient is about ten million times smaller than the last layer’s (for tanh and ReLU, a few hundred to two thousand times). The first layers barely move.
  • Fix. Change what the gradient has to pass through. The options are ReLU-like activations, initialisation scaled to each layer’s size, standardised inputs, normalisation layers such as batch normalisation (Ioffe & Szegedy, 2015), and, in later sessions, residual connections. Adam (below) gets the sigmoid network to 0.291 by enlarging the tiny steps, but that is a workaround, not a cure, because the gradients still vanish.
Figure 9.6: Derivatives of sigmoid, tanh and ReLU. Sigmoid’s is at most 0.25 (at \(z = 0\)) and below 0.01 by \(|z| = 6\). Tanh’s peaks at 1 but also vanishes in both tails. ReLU’s is exactly 0 or exactly 1.

ReLU (Session 8) never shrinks the gradient for positive inputs, which is why it is the default for hidden layers. Its flat side, however, can leave a unit with no gradient at all (“dying ReLU”, the first discussion problem). Vanishing gradients come back in very deep CNNs (Session 10), recurrent networks and Transformers (Session 11), and each of those sessions shows the architectural fix.

9.5.3 Failure 3: the optimiser and the learning rate

  • Observation. The same 64/64 network reaches a validation MAE of 0.250 with Adam (learning rate 0.001), but only 0.305 with plain SGD in the same 20 epochs. With Adam and a learning rate of 0.1, the error jumps around and never gets below 0.298.
  • Why? The learning rate sets the step size of gradient descent (Session 1). A rate that is too small is slow, and one that is too large overshoots the minimum. Plain SGD also uses the same step size for every weight, which Adam does not (next subsection).
  • Fix. Start with Adam and a learning rate around \(10^{-3}\), and always watch the validation curve.

9.5.4 Momentum and Adam: why an optimiser can beat plain SGD

Plain SGD takes each step straight down the gradient of the current mini-batch. This becomes a problem in a long, narrow valley of the loss, which is common because some weights affect the loss far more than others. In such a valley the gradient points mostly across the valley. A learning rate large enough to make progress along the valley makes SGD zigzag from wall to wall, while a safe one only crawls. The three optimisers differ in how they deal with this:

  • SGD follows the current gradient.
  • Momentum follows an average of the recent gradients, like a heavy ball rolling downhill. Directions that agree from step to step add up, and the zigzag across the valley cancels out.
  • Adam (Kingma & Ba, 2015) combines momentum with a separate step size for every weight, scaled to the size of that weight’s recent gradients. Its default learning rate of \(10^{-3}\) works for many networks.
Figure 9.7: Three optimisers, 60 steps each, on a valley 100 times steeper across (\(w_2\)) than along (\(w_1\)). SGD zigzags and then crawls (loss 0.09 after 60 steps); momentum (0.0004) and Adam (0.006) reach the minimum.

Adam is a good default, not a universal winner. In the figure, tuned momentum ends even lower, and which optimiser wins depends on the learning rate and the training time. The lab repeats all three failures on blood-cell images, including this comparison, with one rule: change one thing, see what happens, explain why.

9.6 When a hidden layer pays off: epistasis in a protein

The data. GB1 is a small domain of protein G, a bacterial protein that binds antibodies (IgG). Wu et al. (2016) made all 20⁴ = 160,000 combinations of amino acids at four of its positions (V39, D40, G41 and V54), and measured by deep sequencing how well each variant binds IgG. Of these, 149,361 variants have a value. Fitness is relative to the wild type (VDGV = 1) and spans several orders of magnitude, so we predict \(\log_{10}(\text{fitness} + 0.001)\). Each variant is described by the one-hot encoding of its four residues (Session 7), that is, by 4 × 20 = 80 numbers.

Additive or not? A linear model on this encoding gives every amino acid at every position its own weight and adds them up. It is therefore an additive model, exactly like a profile or PSSM (Session 5), in which each position contributes on its own. Such a model cannot represent epistasis, where the effect of a mutation depends on which residues sit at the other positions. Epistasis is the non-additive, XOR-like pattern of Session 7, now as a measured property of a real protein, and a hidden layer can represent it.

We train the models with early stopping on 20,000 variants, with 5,000 for validation and 20,000 for testing:

Table 9.1: Models trained on the GB1 fitness data.
Model Weights Train MAE Val MAE Test MAE Test \(R^2\)
Linear (additive) 81 0.379 0.371 0.385 0.496
64/64 9,409 0.212 0.250 0.254 0.787
64/64 + weight decay (10⁻³) 9,409 0.215 0.241 0.249 0.795
64/64 + dropout 0.2 9,409 0.220 0.242 0.247 0.798
256/256 86,785 0.193 0.252 0.259 0.779
Figure 9.8: Training (dashed) and validation (solid) MAE per epoch, for the additive model and the 64/64 network.

The network explains about 79% of the variance in fitness, against 50% for the additive model. Weight decay and dropout help a little, and a network ten times larger does not help further.

Figure 9.9: Left and middle: predicted against measured \(\log_{10}\) fitness for the 20,000 test variants (colour: how many variants fall in each hexagon). The additive model compresses the predictions into a narrow band and misses the fit variants; the network follows the diagonal. Right: test \(R^2\) against the number of training variants.

The network needs data to learn the interactions. With 500 training variants the two models are equally good (\(R^2\) 0.43 against 0.41). The interactions exist, but there are too few examples to learn them. From a few thousand variants on, the network pulls away (0.72 at 5,000, 0.82 at 60,000), while the additive model is stuck at about 0.49, however much data it gets. No amount of data lets a sum represent an interaction.

This is Session 7’s lesson, now on real biological data. A hidden layer can represent non-additive relationships, and with enough data, training actually finds them. It is also the limitation of profiles noted in Session 5, since a sum over positions cannot capture interactions, and Session 12 returns to it with pairwise models learned from multiple alignments. The second discussion problem asks where the advantage comes from, by giving the linear model hand-made interaction features for each pair of positions, as Session 7 did for XOR with \(x_1 x_2\).

9.7 Why not make the network enormous?

If larger networks can represent more, why not always use an enormous one? Session 8’s protein-localisation network answers that question. It has 3000 inputs (150 residues, one-hot), a hidden layer of 64 units and 5 outputs, 192,389 weights in all, and it is trained on only 486 proteins of the homology-aware split.

Figure 9.10: Training and validation accuracy of the localisation network per epoch. Training accuracy reaches 1.0 within a few epochs; validation accuracy peaks at 0.486 (epoch 5) and then falls.

Training accuracy reaches 1.000, so the network memorises every training protein. Validation accuracy, in contrast, peaks at 0.486 after five epochs and then falls, to 0.438 by epoch 29. This is overfitting in its plainest form. With far more freedom than data, the network learns the idiosyncrasies of the training sequences instead of the general signal.

Moreover, the simple baseline wins, as the test set (used once) shows:

Table 9.2: The localisation classifier: the untrained network of Session 8, the trained network and logistic regression.
Metric Untrained network (Session 8, val) Trained network (test) Logistic regression (test)
Accuracy 0.181 0.514 0.569
Macro F1 0.124 0.493 —
MCC −0.027 0.394 0.461
Macro AUROC 0.474 0.831 0.849

Training works, since every number is far above chance, but the linear model is better. Before believing that a bigger model is better, always compare it against a simple baseline.

9.8 What can we do about overfitting?

There are several remedies:

  • More data. The most reliable fix, and often the only one that really helps.
  • A smaller model, or a better architecture. Fewer weights for the same input. Session 10’s convolutional network needs 19,237 weights instead of 192,389.
  • Early stopping. Keep the weights from the epoch with the best validation score (below).
  • Weight decay (L2) and L1 regularisation. Add a penalty for large weights to the loss (below).
  • Dropout. Randomly switch off a fraction of the units in every training step, so the network cannot rely on any single unit (below).

Applied to the localisation network, however, dropout and weight decay do not beat early stopping alone (best validation accuracy 0.476 against 0.486). Training accuracy still reaches 1.000 whatever we do, because with 192,389 weights and 486 examples the network can always memorise. Regularisation is therefore not a substitute for enough data. The following subsections explain how the remedies work.

9.8.1 Early stopping: when to stop training

The two curves of a training run answer two different questions:

  • The training loss tells you whether optimisation is working. If it does not fall, something in the previous sections is wrong (initialisation, gradients, learning rate).
  • The validation loss tells you whether further optimisation still helps generalisation. When it stops falling while the training loss keeps going down, the network has started to fit the training set rather than the problem.

For the breast-cancer biopsies the validation loss levels off after about 50 epochs while the training loss keeps falling (the first figure of this page). For protein localisation, training accuracy reaches 1.000 within a few epochs, while validation accuracy peaks at 0.486 in epoch 5 and then falls. Early stopping keeps the weights from the epoch with the best validation score. It costs nothing and is used in every training loop of this course. Because the epoch is chosen on the validation set, however, only the test set, used once, gives an honest estimate (Session 8).

9.8.2 Penalising large weights: L2 and L1

A network that fits noise usually does it with some very large weights, because a big weight makes the output jump when one input changes a little. Regularisation therefore adds a penalty for large weights to the loss, with a strength \(\lambda\) that we choose on the validation set:

\[ \text{loss} = \text{cross-entropy} + \lambda \sum_j w_j^2 \quad\text{(L2)} \qquad\text{or}\qquad \text{cross-entropy} + \lambda \sum_j |w_j| \quad\text{(L1)} \]

  • L2 pulls every weight towards zero in proportion to its size (the gradient of \(\lambda w^2\) is \(2\lambda w\)). Every step therefore shrinks every weight by the same fraction, which is why it is called weight decay and why PyTorch implements it as the optimiser’s weight_decay argument. Weights become small, but almost never exactly zero.
  • L1 pulls with the same force, \(\lambda\), however small a weight already is. Weights that help little are pushed all the way to exactly zero, so the model selects features.

For linear regression the two have their own names, ridge (L2) and lasso (L1). The notebook fits both to the 30 features of the breast-cancer biopsies of Section 9.2. It uses logistic regression, so that every feature has exactly one weight:

Figure 9.11: Logistic regression on the 30 biopsy features. Left and middle: every line is one feature’s weight as the penalty gets stronger (to the right). Right: the number of non-zero weights (solid) and the validation accuracy (dotted).

With \(\lambda\) chosen on the validation set, L2 keeps all 30 weights (test accuracy 0.965), while L1 keeps only 6 (test accuracy 0.942). The six are led by the largest tumour radius and the number of concave points on the cell nuclei. L1 thus gives almost the same accuracy with a model that a pathologist can read. When features are many and examples few, as in gene-expression data with 20,000 genes and a few hundred patients, this is often the main reason to use L1. The third discussion problem checks the same effect by hand.

9.8.3 Dropout

Dropout (Srivastava et al. 2014, JMLR 15:1929-1958) switches off each hidden unit with probability \(p\) (often 0.2-0.5), with a fresh random choice for every mini-batch.

Figure 9.12: Dropout with \(p = 0.5\). In every training step a different random set of hidden units is switched off. At test time every unit is on.

Why should damaging the network help? The answer is that no unit can count on its partners. Without dropout, units can form fragile combinations that fit the quirks of the training set. With dropout, any partner may be missing in the next step, so the information is spread over many units.

Training mode and evaluation mode. Dropout is used only for training, and at evaluation time all units are on. (PyTorch scales the surviving units by \(1/(1-p)\) during training, so nothing else needs to change.) model.train() switches dropout on and model.eval() switches it off, which is why every training loop in this course calls both. If you forget model.eval(), the same input gives a different prediction every time, as the notebook shows.

On GB1, dropout 0.2 raises the training error a little (MAE 0.212 → 0.220) and lowers the test error a little (0.254 → 0.247). The gap narrows, as it should with a regulariser. On the localisation network, dropout does not beat early stopping.

9.9 Is the improvement real?

Session 8’s rules apply to every comparison in this session. Choices (epochs, sizes, regularisation) are made on the validation set, the test set is used once, and every model is compared with a simple baseline. The table puts the three datasets side by side:

Table 9.3: The three datasets of this session side by side.
Dataset Training examples Network weights Examples per weight Network vs. linear
Breast cancer 398 513 0.78 tie (CV accuracy 0.979 vs. 0.979)
GB1 fitness 20,000 9,409 2.1 network (test \(R^2\) 0.787 vs. 0.496)
Protein localisation 486 192,389 0.003 linear (test accuracy 0.514 vs. 0.569)
  • Breast cancer: the network trains well, but a linear model is just as good, because the problem is nearly additive.
  • GB1 fitness: the network clearly wins, because the relationship is non-additive (epistasis) and there are enough data to learn it.
  • Protein localisation: the network memorises and loses to the baseline, because the data are scarce relative to the model.

Examples per weight is a warning sign, not a strict rule. Modern networks often have more weights than training examples and still generalise, thanks to their architecture, regularisation and the way they are trained. When a model is huge relative to its data, however, check it against a baseline before trusting it.

TipNow you can read the whole PHD pipeline

Rost & Sander’s PHD (1993, Session 7) is the recipe of Sessions 5-9 at the scale that mattered historically. It combined profile input from a multiple alignment (Session 5) with networks that have a hidden layer (Session 7) and three outputs (Session 8). The networks were trained by backpropagation (this session) and evaluated by cross-validation on test sets purged of similar sequences (Session 8’s homology partitioning). PHD reached 70.8% three-state accuracy.

Next: fewer weights, by using the structure of the data. A fully connected layer has a separate weight for every connection, which is why the localisation network needs 192,389 of them. A sequence, however, has structure, because the same motif means the same thing wherever it occurs. Can we exploit that and use far fewer weights? That is convolution, the topic of Session 10.

9.9.1 Further reading

9.10 Check your understanding

9.11 The course notebook

notebooks/session09-backprop-training-regularization.ipynb computes every number and figure on this page, section by section. It runs the breast-cancer training and shows the predictions before and after training. It works the backpropagation example by hand and checks it against autograd, and it computes cross-entropy by hand. It then reproduces the three training failures (equal initialisation, deep sigmoid networks, optimiser and learning rate), the GB1 models and the effect of the training-set size, and finally the localisation network’s overfitting, regularisation and comparison with logistic regression.

Open it in Google Colab via the badge at the top of the notebook page to edit and run it yourself.

9.12 The lab

Full lab, on blood-cell images (BloodMNIST, as in the Session 8 lab). The lab is adapted from the previous course’s “Neural Networks” lab and rebuilt in PyTorch, and this time you write the training loop yourself. Work through notebooks/session09-lab.ipynb, replacing every todo(...) call with your own code:

  1. write the four-step mini-batch training loop;
  2. compare cross-entropy with mean squared and mean absolute error as the loss for a classifier;
  3. add hidden layers (tanh, sigmoid, ReLU) and measure the gradient that reaches the first layer;
  4. compare plain SGD, SGD with momentum and Adam, a 128-unit hidden layer with no hidden layer at all, and run a small hyperparameter search chosen on the validation set, using the test set once, at the end.

The best dense network ends up near 78% on the test images. An optional extra at the end of the notebook, not part of the quiz, starts every weight at 1 and shows why random initialisation matters. Session 10’s lab trains a convolutional network on the same cells.

9.12.1 Submitting this lab

This lab is administered through Canvas, not through this book. See the course’s Canvas page for the current quiz, the deadline and how to submit.

9.13 What’s next

Session 10 moves from fully connected networks to convolutional ones, which share weights across positions. It covers image classification and the history of ImageNet, data augmentation and transfer learning (on colorectal-cancer histology images), and the localisation dataset again, now with 10 times fewer weights. It then turns to DNA sequence (DeepSEA) and protein contact prediction.