Neural Networks

Lectures this week
  • Episode 7 (We 18/11, 2:15–4:00 PM): Neural networks I (Giulio)
  • Episode 8 (Fr 20/11, 10:15 AM–12:00 PM): Neural networks II (Giulio)
Deadlines this week

Due at your exercise session this week: Week 3 exercises

  • Bayesian networks (pencil-and-paper) (1p)
  • Bayesian networks: Car (programming) (1.5p)
  • Bayesian networks: Earthquake (programming) (1p)
  • Plastic Film Folding Machine (0.5p)
  • Decision Trees: Play Tennis info-gain (2p)

This week's exercises are due at your exercise session in Week 5.

Exercises
Learning objectives
Theme Objectives (after the course, you ...)
Neural networks
  • can explain the perceptron and implement the perceptron algorithm for a simple binary classification problem
  • can describe multilayer perceptrons and forward propagation
  • can explain why neural networks use activation functions
  • can describe backpropagation as the mechanism that provides gradients for optimization
  • can explain learned representations and word embeddings at a conceptual level

From perceptrons to multilayer networks

Neural networks are models whose parameters are learned from examples. They are built by composing simple transformations. This makes it possible to learn useful intermediate representations instead of requiring every feature to be designed by hand.

The perceptron

A perceptron receives an input vector x, multiplies each input by a learned weight, adds a bias, and applies a threshold. In compact form, it predicts from wTx + b. During training, the weights are adjusted when the prediction is wrong. A perceptron can learn a linear decision boundary, but it cannot represent every classification problem.

x₁x₂x₃weighted sumw·x + bactivationf(z)ŷw₁w₂w₃

For example, with inputs (1, 2), weights (0.7, -0.2), and bias 0.1, the weighted sum is 0.7 × 1 - 0.2 × 2 + 0.1 = 0.4. A step activation produces class 1 because the result is positive. The weights determine the position and orientation of the separating line; the bias moves it without changing its orientation.

A linear decision boundaryupdate weightsErrors drive the boundary

The limit of a single boundary

The perceptron learning rule converges when the classes are linearly separable. It cannot solve XOR: no single straight line can place the two blue points on one side and the two orange points on the other. This is a representational limitation, not a failure to train long enough.

XOR requires more than one boundary

Multilayer perceptrons

A multilayer perceptron combines neurons into an input layer, one or more hidden layers, and an output layer. Each layer transforms the representation produced by the previous layer. Width gives a layer more units with which to represent patterns, while depth allows the model to build more complex transformations from simpler ones.

A forward pass applies these transformations from the input to the prediction. For a layer, the main calculation is a linear transformation followed by an activation function:

h = activation(Wx + b)

inputhidden representation 1hidden representation 2output

Why activation functions matter

Without nonlinear activation functions, stacking layers would still produce only a linear model. ReLU is widely used because it is simple and works well in many settings. Sigmoid maps values between zero and one, while tanh maps them between minus one and one. The activation function influences how easily the network trains and what kinds of decision boundaries it can learn.

stepsigmoidtanhReLU

Learning representations

Training a neural network

Training begins with a loss function that measures the difference between predictions and target values. Backpropagation applies the chain rule to calculate how each parameter contributed to that loss. An optimizer then uses the resulting gradients to update the parameters. For this course, the important idea is the division of labour: the forward pass produces a prediction, the loss evaluates it, backpropagation supplies gradients, and the optimizer updates the model.

forward passpredictionlosserror signalbackpropagationgradientsoptimizernew parameters

Performance on the training data alone is not enough. We use separate validation and test data to evaluate whether a network generalizes to new examples. Excessive model capacity or training can lead to overfitting. Regularization, suitable model size, and early stopping can reduce this problem.

Embeddings

An embedding represents a discrete item, such as a word, as a vector of learned numbers. Items used in similar contexts tend to acquire similar vectors. Embeddings therefore replace arbitrary identifiers with representations in which distance and direction can encode useful relationships. The same idea is used for text tokens, images, users, products, and many other kinds of data.

catdogkittencartruckbusapplepearbananaIllustrative two-dimensional projection of learned vectors

Summary

Neural networks learn layered representations by combining linear transformations with nonlinear activation functions. Training adjusts their parameters using gradients. Embeddings extend representation learning to discrete items such as words and prepare the ground for attention and Transformers in Week 5.

Tensorflow Playground: Basic Exploration of Neural Networks (0.5p)

Open the TensorFlow Playground:
Go to TensorFlow Playground in your web browser.

Task 1: Build a Simple Neural Network

  • Start with the default dataset (two circles) and network settings.
  • Use one hidden layer with one neuron to classify the points.
  • Observe the decision boundary and how the network performs.

Task 2: Add More Neurons

  • Increase the number of neurons in the hidden layer to three neurons.
  • Observe the changes in the decision boundary and how well the network classifies the data.
  • How does the performance change with more neurons?

Task 3: Add More Layers

  • Add a second hidden layer with two neurons.
  • Compare the performance to the previous network with just one hidden layer. How does adding more layers affect the learning?

Results:
Keep a screenshot of each configuration (one neuron, three neurons, two layers). Briefly describe (2-3 sentences) how each change affected the decision boundary and model performance.

Tensorflow Playground: Experimenting with Complex Data and Advanced Features (1p)

After the basic playground exercise, switch to a harder dataset such as Spirals. The tasks below help you see how architecture choices, activation functions, learning rates, and regularization affect what the network can learn.

Task 1: Investigate Different Datasets

  • Change the dataset to "Spirals" using the dropdown menu.
  • Experiment with different network architectures to successfully classify the data:
  • Start with one hidden layer and four neurons.
  • Gradually increase the number of neurons and layers until the network correctly classifies the spirals.
  • What is the minimum number of layers and neurons needed to classify the spirals accurately?

Task 2: Adjusting Activation Functions and Learning Rate

  • Change the activation function from "ReLU" to "tanh" and observe how it affects the network's ability to learn.
  • Now, adjust the learning rate slider and observe how different learning rates (e.g., 0.01, 0.1, 3) impact the training speed and model accuracy.

Task 3: Experiment with Regularization

  • Add L2 regularization to your network and observe how it impacts the decision boundary and prevents overfitting.
  • Try increasing the regularization strength to see its effects on training.

Results:
Keep screenshots of your experiments with the "Spirals" dataset and different network configurations. Briefly describe (2-3 sentences) how changing the activation function, learning rate, and regularization influenced the network's performance.

Training Your First Neural Network (programming) (1.5p)

Time to stop drawing perceptrons on paper and actually train one. We'll use scikit-learn's MLPClassifier, which hides the training loop (forward pass, loss, backward pass, weight update) behind a single .fit() call — exactly the same building blocks you've read about, just already wired up for you.

Run this on the MNIST digit data (load MNIST via sklearn.datasets.fetch_openml("mnist_784"), or reuse mnist-x.data/mnist-y.data from Perceptron.zip): 

from sklearn.neural_network import MLPClassifier

clf = MLPClassifier(
    hidden_layer_sizes=(40,),   # one hidden layer, 40 neurons
    max_iter=8,
    solver="sgd",
    learning_rate_init=0.2,
    random_state=1,
)
clf.fit(X_train, y_train)
print("Test accuracy:", clf.score(X_test, y_test))

  1. Run the code above as-is and note the test accuracy.
  2. Change hidden_layer_sizes=(40,) to (10,), then to (100, 50) (two hidden layers). Rerun each and record the accuracy.
  3. Change max_iter=8 to 50 (keeping hidden_layer_sizes=(40,)) and rerun.
  4. In 3–4 sentences: which change helped most, which helped least or hurt, and does that match your intuition about network capacity vs. training time?
What's actually happening inside .fit()?

Every one of the ~8 iterations does exactly what you've seen in lecture: a forward pass through the network, a loss computation, and a weight update via gradient descent. MLPClassifier doesn't do anything conceptually different from the perceptron you studied earlier — it's just stacked into multiple layers and trained automatically.

Same Network, Three Activation Functions (programming) (2p)

Start from the Keras MNIST script in TensorFlow's beginner quickstart, which uses the model below. The model architecture, training loop, and dataset all stay exactly the same — the only thing that changes is one word: 

model = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=(28, 28)),
    tf.keras.layers.Dense(128, activation='relu'),   # ← change just this word
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(10)
])

  1. Train and evaluate the model with activation='relu' (the default), 'tanh', and 'sigmoid' — three separate runs, everything else identical.
  2. Record the final test accuracy and how many epochs it took to visibly "settle" (stop improving much epoch to epoch) for each.
  3. In 4–5 sentences: which activation trained fastest/most accurately in your runs? Recall from lecture why sigmoid in particular can suffer from vanishing gradients in deeper networks — does your small 1-hidden-layer result seem consistent with that, or too small a network to see the effect?

Loading Pretrained Word Vectors in Three Lines (programming) (1p)

You don't need to train a word-embedding model yourself to explore one — huge pretrained ones are a download away. This entire exercise is three lines of setup: 

import gensim.downloader as api

model = api.load("glove-wiki-gigaword-50")   # ~66MB, downloads once
model.most_similar("glass")

  1. Run the code above. You'll get a ranked list of the words whose vectors are closest to "glass" — record the top 5.
  2. Try at least 4 more words of your own choosing, including one ambiguous word (multiple meanings — e.g. "bank," "bat," "crane") and one very specific, uncommon word.
  3. For the ambiguous word: do the returned neighbors reflect one meaning, multiple meanings, or something confused? Why do you think that happened, given that the model was trained on raw text with no sense of "which meaning is intended" ever labeled?
  4. model.similarity("cat", "dog") returns a single number between −1 and 1 — the cosine similarity between the two words' vectors. Compute it for 3 word pairs of your choosing (some you expect to be similar, some not) and report the numbers alongside your expectations.

Table of Contents