Deep Learning — PyTorch & NumPy MLP Fundamentals

Lab 1 of the Deep Learning course, UPF (3rd year, 2024). A binary classifier for a small 2-D two-class dataset, built three times over: first a multi-layer perceptron written from scratch in NumPy with a hand-derived backward pass; then a momentum optimiser (OptimMom) swapped in for plain gradient descent and compared across momentum coefficients; then the whole thing re-implemented in PyTorch with autograd and swept over hidden-layer width and learning rate. The point of the exercise is that all three arrive at the same architecture — 2 inputs → 1 sigmoid hidden layer → 1 sigmoid output, binary cross-entropy loss — so the differences that show up are attributable to the optimiser and the hyper-parameters, not the model.

Taken from the group's report (docs/P1-Report.pdf):

• Cross-entropy, not MSE. The task is binary classification with a sigmoid output, so binary cross-entropy is the appropriate loss. The NumPy version differentiates it by hand; the PyTorch version uses nn.BCELoss.
• Normalise first. Both feature dimensions are shifted to zero mean and unit standard deviation before anything else, so the two axes contribute comparably to the gradient.
• Momentum. A fraction beta of the previous update vector is folded into the current one, which damps oscillations and keeps "speed" along consistent descent directions. The report's expectation, borne out by the measurements below, is that a smaller beta adapts faster to changes in the gradient here, and that beta = 1 degenerates entirely — the optimiser rides the first (randomly initialised) gradient forever and never optimises anything. The report's own conclusion is that momentum "does not add great extra value because the dataset is very small and the optimization problem is very easy".
• Epoch count. 50,000 epochs at lr = 0.01 was chosen because the loss has gone flat by then while test accuracy still tracks train accuracy, which the report reads as evidence the model is not overfitting.
• Sweeps over width and learning rate. More hidden nodes let the network approximate a richer function and converge faster, at a compute cost; the learning rate turns out to matter far more strongly than the width on this problem.

src/
  __init__.py       shared hyper-parameters (LR = 0.01, EPOCHS = 50000)
  data.py           CSV loading, per-split normalisation, scatter plots
  mlp_numpy.py      MLP — from-scratch forward/backward, NumPy only
  optimizers.py     OptimMom — gradient descent with momentum
  training.py       train / train_woptimizer loops, accuracy helper
  mlp_torch.py      MLP_PyTorch, tensor conversion, train_PyTorch, evaluate
  plotting.py       loss-curve helpers
experiments/
  ex1_numpy_mlp.py           Exercise 1 driver
  ex2_momentum_optimizer.py  Exercise 2 driver — beta sweep + GD baseline
  ex3_pytorch_sweeps.py      Exercise 3 driver — hidden-node and lr sweeps
notebooks/
  P1_Questions_Answers.ipynb the original Colab notebook, with the exercise
                             statements, stored outputs and all plots
docs/
  P1-Report.pdf              the submitted report and its analysis
data/
  README.md                  what belongs in the (gitignored) data directory

pip install -r requirements.txt

# Put train.csv and test.csv in data/ — see data/README.md for the format.
# Or point elsewhere with --data-dir / $DL_P1_DATA_DIR.

python -m experiments.ex1_numpy_mlp                     # Ex 1
python -m experiments.ex1_numpy_mlp --no-plots          # ...without the animation
python -m experiments.ex2_momentum_optimizer            # Ex 2
python -m experiments.ex3_pytorch_sweeps                # Ex 3, both sweeps
python -m experiments.ex3_pytorch_sweeps --sweep lr     # Ex 3, learning rate only

Run from the repository root. Every driver accepts --data-dir and --epochs; Ex 1 and Ex 2 also accept --lr. Defaults reproduce the notebook's settings (50,000 epochs, lr = 0.01), which take a while — pass a smaller --epochs for a quick check.

The dataset is not in the repo. The original notebook mounted Google Drive and read from a shared-drive path; that path has been replaced by a parameter. See data/README.md.

All numbers below are read from the stored outputs of notebooks/P1_Questions_Answers.ipynb. Nothing here was re-measured — see Limitations.

The report quotes 96% / 92% for this configuration; the notebook's stored output says 0.97 / 0.92. The 0.97 figure is what the code actually printed. The report also records an earlier configuration — lr = 0.001, 5,000 epochs — at 90% train / 84% test, but that run is not in the notebook's stored outputs.

Final training loss at epoch 49,999:

Momentum does not help on this problem: the loss increases monotonically with beta, and beta = 0 (equivalent to plain gradient descent) is best. At beta = 1 the loss never moves from its initial value of 0.593662 across all 50,000 epochs — the optimiser is fully committed to the first gradient and ignores every subsequent one, exactly as the report predicts. Note the sweep uses the default MLP() (3 hidden nodes), not the 10-node network of Ex 1, which is why the loss floor is higher than in the table above.

No accuracy was computed for Exercise 2 — the notebook only records losses.

Best: 20 hidden nodes — lowest loss and highest accuracy on both splits. The trend is not monotonic: 50 nodes does worse than 20 on every metric, so the report's "more hidden nodes converge faster" reasoning does not hold at the top end of this sweep.

Best test accuracy: lr = 0.1, at 0.9200. Learning rate dominates hidden width on this problem — the spread here (0.76 → 0.92) is wider than across the whole hidden-node sweep. At lr = 1.0 and 3.0 the network reaches 100% training accuracy and a near-zero loss but test accuracy drops back to 0.88, the classic overfitting signature. The report's closing note is that a high learning rate looking best here is not generally advisable, since it risks instability and convergence at a poor loss value.

Note that lr = 0.01 with 10 hidden nodes appears in both sweeps and gives different results (0.8400 vs 0.8000 test accuracy). This is genuine: the notebook seeds torch.manual_seed(22) once, before both sweeps, so the second sweep initialises from a different RNG state. The drivers preserve this.

• The dataset is not included and could not be inspected. data/README.md documents the format as inferred from how the notebook parses the CSVs; the test split's row count could not be determined from the outputs.

• A momentum-state bug in OptimMom.step was fixed. The submitted loop ended with self.v = v, keeping only the last layer's velocity instead of the list of per-layer velocities. On the next call zip then paired each weight matrix against a slice of that single array, so every layer but one lost its momentum between steps. It now accumulates new_v and assigns the full list.

  Consequence: the Exercise 2 beta sweep in the Results table above was measured with the original behaviour. Re-running it now will give different — and more meaningful — numbers, because momentum will actually persist across steps for every layer. The qualitative conclusion (that momentum does not help on this small, easy problem, and that beta = 1 degenerates) is unaffected.

• Report vs notebook discrepancy on the Exercise 1 accuracy (96% vs 97%), noted inline above.

• Per-split normalisation. Test data is normalised using test statistics, not training statistics. This leaks distributional information and is not standard practice; it is what the notebook does and has been kept.

• Course-provided scaffolding. The exercise statements, the MLP skeleton and the OptimSGD class that OptimMom was modelled on came from the course materials. The OptimMom implementation, the training loops, the sweeps, the analysis and the report are the students' own work.

• Refactoring. The notebook was split into modules and the two Exercise 3 sweep cells were turned into functions behind a --sweep flag. Logic was copied, not rewritten; original comments (including their typos) are preserved.

Group work for UPF Deep Learning, Lab 1:

• José María Pérez Clar
• Martí Oms Graells
• Telmo Linacisoro