RepDBmaker Pipeline

A Snakemake workflow for building taxonomically annotated protein sequence databases from public resources and custom genome collections.

• RepDBmaker Pipeline
  • Contents
  • Overview
  • Installation
    • Requirements
  • Docker
  • Quick start
  • Reproducibility
    • Pinned source snapshots
    • Reproducing a release
  • Troubleshooting
  • Documentation
  • Citation
  • License

RepDBmaker assembles protein sequence databases from multiple sources, annotates them with taxonomy, and builds searchable indices for Diamond, MMseqs2 and BLAST.

It draws on:

• prokaryotes from GTDB
• eukaryotes from EukProt, P10K, and UniProt
• viruses from NCBI Virus
• your own genomes or transcriptomes, alongside the public sources

with optional taxonomic clustering and contamination filtering along the way.

• Snakemake (v8.11.6)
• Conda or Miniconda
• Internet access for external downloads
• Sufficient disk space for genome and database files

Pick how the workflow schedules its jobs (local machine, SLURM, LSF, or a site-specific profile) in Choosing an executor.

If using --sdm conda, Snakemake will automatically create the following environments from workflow/envs/:

• workflow/envs/python.yaml: Python, pandas, matplotlib, polars
• workflow/envs/homology.yaml: diamond, mmseqs2, blast
• workflow/envs/utils.yaml: taxonkit, csvtk, ncbi-datasets-cli, newick_utils, seqkit, jq, and standard CLI tools (wget, tar, unzip, gzip)
• workflow/envs/R.yaml: R and visualization/taxonomy packages
• workflow/envs/krona.yaml: Krona (interactive charts)

Create all of them up front, without running the pipeline:

snakemake --configfile config/repdb.yaml --sdm conda --conda-create-envs-only

--configfile config/repdb.yaml is required here too: the Snakefile reads config["dbs"]["type"] while building its rule graph, before Snakemake even gets to --conda-create-envs-only.

These .yaml specs are intentionally loose (minimum bounds only where a feature requires it) so a fresh install resolves against current packages. To instead reproduce the exact package builds used for the published RepDB v1.0, a pin file (workflow/envs/<name>.linux-64.pin.txt) sits next to each .yaml and is picked up automatically by --sdm conda, no extra flag needed.

A Docker image is available at Docker Hub, built from the pinned envs so it reproduces the RepDB v1.0 toolchain exactly, without needing conda or Snakemake installed locally. From the repository root:

docker run --rm -t -v $(pwd):/app/data gmuttiirb/repdbmaker:v1.0 \
  snakemake --configfile config/default.yaml config/repdb.yaml --cores <N> \
  --directory /app/data --sdm conda --conda-prefix /conda-envs

Both flags are required: --sdm conda turns on each rule's conda: env, and --conda-prefix /conda-envs must be exactly this value to reuse the image's pre-built environments instead of rebuilding them from scratch. Add -n to preview the plan without running anything; -t just gets you Snakemake's usual colored output.

--configfile takes two files here, config/default.yaml explicitly: --directory changes the base Snakemake resolves relative paths against, which includes the Snakefile's own configfile: "config/default.yaml" directive. Passing only config/repdb.yaml after --directory /app/data makes Snakemake look for config/default.yaml inside /app/data (your mounted, otherwise-empty directory) instead of the image's own /app, silently failing to load it. Passing both explicitly sidesteps that.

For reproducible pulls, use the image's immutable digest instead of the mutable :v1.0 tag:

docker run --rm -t -v $(pwd):/app/data \
  gmuttiirb/repdbmaker@sha256:c3458c7c5dd8e1d1a7dcf7ea61d02b97f4807c309ade381c6a65a72c68fa628a \
  snakemake --configfile config/default.yaml config/repdb.yaml --cores <N> \
  --directory /app/data --sdm conda --conda-prefix /conda-envs

The workflow is two phases that meet at the universe, the enriched table of every available proteome (id, source, 7 ranks, completeness). Both phases read the same config file; pick the phase with the target:

build produces the universe first if needed, so it's self-contained; run sample on its own to stop at the universe and review it before building.

To just inspect the available proteomes first:

snakemake --configfile config/repdb.yaml -j 1 --until available_proteomes
# -> results/meta/available_proteomes.tsv, for picking a proteome subset

To build everything with the default config:

snakemake --configfile config/repdb.yaml -j 14

--configfile config/repdb.yaml is required even just for available_proteomes: the Snakefile reads config["dbs"]["type"] while building its rule graph, before it even knows which target you asked for, and config/repdb.yaml is what supplies dbs: on top of the Snakefile's own auto-loaded config/default.yaml.

For more, see Example commands: a fast smoke test on a tiny subset, building only a custom database, running on a scheduler, reproducing a release, and more.

Producing an actual versioned release (freezing the universe, staging assets, publishing to GitHub/Zenodo) is a longer process; see docs/releasing.md.

RepDBmaker supports reproducibility at three levels; pick the one your use case needs.

External sources drift, so pin the snapshots under versions: in config/repdb.yaml:

• gtdb: a specific release (e.g. release226), never latest.
• unieuk: the exported UniEuk taxonomy version.
• EukProt: the EukProt version.
• taxdump: a dated NCBI taxdump archive (or latest).

For sources without stable versioned hosting (UniProt reference-proteome release, RefSeq virus catalog, P10K), the universe below is what actually freezes them, so record the retrieval date alongside your run too.

resources/releases/v1/config.yaml is the self-contained build config for RepDB v1.0. It combines two independent routes; use either on its own, or both together as this config does.

Composition-level, via dbs.build.repdb.universe: because several sources are unversioned, re-running the full selection later may yield a different set of proteomes. Building from a frozen universe fixes that: taxonomy harmonization is skipped entirely (including for any custom proteomes configured), and the selection re-runs deterministically on the pinned composition. The frozen resources/releases/v1/universe.tsv is too large for git and ships as a GitHub release asset instead; download it into that path before running the command below.

Artifact-level, via dbs.build.repdb.zenodo: pins the exact sequences. The fasta, cluster table, decontamination report and taxdump are fetched from the RepDB v1.0 Zenodo deposit and checksum-verified instead of reassembled, so you get the exact published database rather than a re-derivation of it. See Getting RepDB v1.0 if that (or just downloading the data directly, no pipeline needed) is all you're after.

snakemake build --configfile resources/releases/v1/config.yaml --sdm conda -j 8

See docs/releasing.md for how a release like this gets produced, and where its other assets live.

Sometimes things can go wrong while downloading a proteome: rule db_stats fails if any gzipped fasta is malformed, blocking the database fasta until it's resolved.

cut -f1 results/dbs/<db>/genome_table.tsv | xargs -I {} sh -c 'gzip -t "{}" || echo "Failed: {}"'

Delete the problematic files and re-run the pipeline. If the problem persists, the files may be broken at the source or the current downloading script may be failing on them; we recommend excluding them and finding the most suitable alternative.

Everything beyond this README lives under docs/:

• Getting RepDB v1.0: download the published data directly, or fetch + build indices with the pipeline
• Example commands: a cookbook covering common tasks, from a fast smoke test to reproducing a release
• Choosing an executor: local, SLURM, LSF, site profiles
• Configuration: full config reference plus eukaryote downsampling
• Custom databases: adding your own database or proteomes
• Clustering: the additive cluster: block, how it works, repdb_clustered
• Decontamination: the cross-domain contamination filter
• Outputs and quality control: where everything lands, and the QC reports
• Utilities and benchmarking: helper scripts, the NR/PhyloDB comparison
• Reproducing the paper figures: the two-stage HPC → laptop notebook split, and the palette checks
• Building & releasing RepDB: the full release process

If you use RepDBmaker or RepDB, please cite:

Mutti G. and Gabaldón T. Automated reconstruction of reproducible protein databases with RepDBmaker. Protein Science (2026). DOI:

See the LICENSE file for details.