scRNA Autoencoder Latent-Dimension Study

How should latent size (d) change as class complexity (K) changes in single-cell RNA-seq?

This project benchmarks AE, VAE, and scVI models on curated scRNA-seq datasets to study whether the optimal latent dimension can be predicted from the number of cell types. It uses a trust-aware evaluation framework that strictly separates ground-truth evidence from exploratory results.

Using curated evidence from scVI with negative binomial likelihood:

No universal d = f(K) scaling law was found. Instead, the data suggest dataset-dependent narrow bands: compact latent spaces (d=2-4) work for moderate-complexity datasets, while atlas-scale data favors d=10-12.

sc_autoencoder_project/
├── src/                   # Core package (models, training, evaluation, data)
│   ├── data/              #   Dataset loading, preprocessing, splitting
│   ├── models/            #   AE, VAE, scVI wrapper, loss functions
│   ├── training/          #   Training loop with early stopping, TensorBoard
│   ├── evaluation/        #   Clustering, reconstruction, batch metrics
│   └── utils/             #   Logging and tensor conversion helpers
├── scripts/               # CLI entry points
│   ├── run_experiment.py      # Main experiment orchestrator
│   ├── train.py               # Single model training
│   ├── smoke_test.py          # Pipeline validation
│   ├── preprocess.py          # Dataset preprocessing
│   ├── validate_pipeline.py   # End-to-end pipeline checks
│   ├── onboard_contract_h5ad.py  # Large-atlas onboarding
│   └── ...                    # Audit, inspection, re-evaluation tools
├── code_analysis/         # Post-hoc analysis pipeline (15 modules)
│   ├── common.py              # Shared plotting/loading utilities
│   ├── a00_inventory.py       # Data readiness audit
│   ├── a02_primary_d_vs_k.py  # Core d=f(K) analysis
│   ├── a14_publication_figures.py  # Publication-quality figures
│   └── ...                    # Cross-model, batch, stability, ELBO analyses
├── config/                # YAML experiment configurations
├── jobs/                  # SLURM job scripts for HPC execution
├── data/                  # Datasets (not tracked, downloaded on demand)
├── results/               # Experiment outputs (selectively tracked)
│   ├── global/tables/         # Core evidence & formula-fit CSVs
│   ├── datasets/              # Per-dataset figures, latents, tables
│   ├── analysis/              # Early analysis pipeline output
│   ├── analysis_runs/         # Timestamped analysis snapshots
│   └── code_analysis_runs/    # 15-module analysis pipeline output
├── checkpoints/           # Model weights (not tracked)
└── logs/                  # Training logs (not tracked)

# Clone and enter the repository
git clone https://github.com/annus3/scRNA-latent-AE.git
cd scRNA-latent-AE

# Create and activate environment
python -m venv ~/envs/scae
source ~/envs/scae/bin/activate

# Install package and all dependencies
pip install -e .

Requirements: Python >= 3.10, PyTorch >= 2.0, scvi-tools >= 1.0, Scanpy >= 1.9

python scripts/smoke_test.py --with_scvi

python scripts/preprocess.py --config config/default.yaml --dataset pbmc3k

python scripts/run_experiment.py \
  --config config/default.yaml \
  --datasets pbmc3k,paul15

python scripts/run_experiment.py \
  --config config/default.yaml \
  --report_only_csv results/global/tables/combined_curated_real_plus_phase4_enriched.csv \
  --auto_d_report

bash code_analysis/analysis.sh

Each analysis run creates a timestamped output folder under results/code_analysis_runs/.

The project was developed in four sequential phases, each building on the previous:

Phase 4 prerequisites: Large-atlas datasets must be downloaded and placed under $WORK/sc_autoencoder_project/data/raw/ before running. Run the onboarding script first to prepare each dataset, then the sweep:

# Example for AIFI atlas
sbatch jobs/phase4_aifi_onboard_light.sh
sbatch jobs/phase4_aifi_scvi_nb_reduced.sh

For large-scale runs on SLURM clusters, use the job scripts in jobs/:

# Validate the pipeline first (GPU, ~10 min)
sbatch jobs/smoke_test.sh

# Generic latent-dimension sweep (adapt partition/modules for your cluster)
sbatch jobs/slurm_sweep_generic.sh

# Parallel array sweep (12 tasks, one per latent dim)
sbatch jobs/sweep_array.sh

# Phase-specific runs (run in order)
sbatch jobs/phase1_pbmc3k_quick.sbatch
sbatch jobs/phase2_scvi_pbmc12k.sbatch
sbatch jobs/phase3_full_sweep.sh
sbatch jobs/phase4_aifi_scvi_nb_reduced.sh

Job scripts handle data staging via $WORK and $TMPDIR environment variables. Phase 4 configs reference ${SCAE_DATA_DIR} and ${SCAE_RESULTS_DIR} — these are exported automatically by the job scripts. If running outside SLURM, set them manually before invoking the phase configs.

The project enforces a strict split between trusted and exploratory evidence:

• Curated: datasets with verified ground-truth cell-type labels. All quantitative claims use only curated evidence.
• Exploratory: synthetic data or datasets with inferred labels. Used for pipeline validation only.

Trust is enforced programmatically — external metrics (ARI, AMI) are only computed when labels pass the trust gate.

The results/ directory is selectively tracked — only final outputs are included; raw tensorboard exports and intermediate runs are gitignored.

All tables are written to results/global/tables/:

The code_analysis/ pipeline produces a timestamped run under results/code_analysis_runs/ with 15 analysis modules:

MIT License. See LICENSE for details.

If you use this code, please cite:

Mohammad Annus, "Trust-Aware Latent Dimension Selection for Single-Cell RNA-seq Autoencoders," Biomedical Network Science Lab (BioNets), FAU Erlangen-Nürnberg, 2026.