Generative Models and Sequence-to-Sequence RNNs

Three generative models — a convolutional autoencoder, a variational autoencoder and a GAN — trained on 20,000 anime face images, and encoder–decoder LSTMs that learn two-digit arithmetic across text and image modalities.

The GAN produces the sharpest images and the least varied ones. Six independently sampled latent vectors decode to six near-identical dark-haired, teal-eyed faces, while its discriminator loss falls to 0.28 as generator loss climbs past 3.0 — the signature of mode collapse under a winning discriminator. The VAE is the opposite: every sample is blurry, and every sample is different. On the RNN side, character accuracy hides sequence failure. At a 10% training split the text-to-text model gets 59% of individual characters right and 8% of complete answers.

Full write-up: report.pdf

The two grids are the comparison. Both show six samples from six random latent vectors. The VAE varies hair colour, eye colour and framing while smoothing away detail; the GAN renders strands of hair and defined eyes on what is essentially the same face six times.

The loss curves diverge monotonically instead of settling into the equilibrium a healthy GAN reaches. Once the discriminator wins, the generator's gradient points toward whichever narrow region of output space still fools it, and diversity is the thing that gets sacrificed.

Text-to-text, by training fraction:

Character accuracy degrades gracefully; sequence accuracy falls off a cliff. Getting a three-character answer right requires every position correct, so per-character errors compound — and at 10% the model has learned digit statistics rather than arithmetic.

Adding LSTM depth to the encoder, all at the same splits as the baselines:

Depth helps most where the task is hardest: +25 points for image-to-text, which has to recognise noisy MNIST digits before it can do arithmetic, against +6 for text-to-text working on clean one-hot input. Text-to-image outputs were scored with a separate CNN digit classifier that reaches 99.45% on real MNIST answer images, so the generated-image accuracies measure the generator rather than the classifier.

Sharpness and coverage are separate axes. The usual framing is that GANs are sharp and VAEs are blurry, which is true here but incomplete. A pixel-wise reconstruction loss forces the VAE to account for every training image, so it covers the data at the cost of averaging detail. An adversarial loss only asks that outputs be indistinguishable from real ones, which a single convincing face satisfies. The blur and the diversity have the same cause, and so do the sharpness and the collapse.

The VAE's KL term is small by construction. The implementation divides the KL divergence by the input dimension (64 × 64 × 3 = 12,288), so it contributes roughly 0.008 against a reconstruction loss near 0.51 — about 1.5% of the total. The latent space is barely pulled toward the prior, which is worth knowing when reading the sampled outputs: this model sits closer to a noisy autoencoder than a strongly regularised VAE.

• No quantitative sample metric. There is no FID, precision/recall or coverage measure, so the mode collapse is diagnosed from six samples and the loss curves rather than measured.
• Single run per configuration, no seeds. Every number comes from one training run, including the baseline-versus-deeper comparisons.
• Task 1 models were not tuned on equal footing — batch size, learning rate, latent dimension and epoch count all differ between the CAE, VAE and GAN, so the comparison is between three tuned systems, not a controlled ablation.
• Several Task 2 metrics are reported as ranges (image-to-text validation accuracy 0.7–0.79), which reflects reading them off training curves rather than a final evaluation pass.
• The Task 1 scripts hardcode Google Drive paths (/content/drive/MyDrive/...) and call drive.mount, so they run as written only in Colab.

The Anime Faces dataset (Kaggle, ~21,551 unlabelled images) is not redistributed here. The Task 1 scripts expect anime.zip containing data/images/, and load the first 20,000 files resized to 64×64 RGB, scaled to [0, 1] for the CAE and VAE and to [-1, 1] for the GAN.

Task 2 needs no download beyond MNIST, which arrives through tf.keras.datasets. The arithmetic dataset is generated in the script: 20,000 queries over all two-digit additions and subtractions, as five-character strings and as stacked MNIST digit images, with + and - signs drawn using OpenCV.

numpy
pandas
scipy
matplotlib
seaborn
scikit-learn
pillow
opencv-python
tqdm
tensorflow

pip install -r requirements.txt
python task2_seq2seq_rnns/rnn_arithmetic.py

Trained on Google Colab with a GPU. The three Task 1 scripts read from and write to Google Drive paths and call drive.mount; to run them elsewhere, change the extract_to, zip_file_path and savefig paths and drop the mount. rnn_arithmetic.py runs anywhere.

• Nallathambi Vethiappan
• Deepak Somesh K J
• Divyanshi Singh