Pooled Morphological Profiling Workflow

A step-by-step data processing pipeline for pooled morphological profiling (including Cell Painting) data.

This worfkflow inputs morphological measurements from single cell objects and in situ sequencing (ISS, also referred to as sequencing by synthesis or SBS) measurements/barcode calls from a pooled screens. Measurements are made by CellProfiler software using a template pipeline (that can be modified as needed) including custom CellProfiler plugin modules for channel balancing and barcode calling.

Example datasets for use with this workflow can be found on the Cell Painting Gallery and include cpg0021-periscope and cpg0032-pooled-rare.

The workflow consists of eight steps (0-7):

0. Image QC. Image metrics from CellProfiler outputs are used to generate QC plots and reports. Includes checking for cell confluency, image focus, image saturation, and phenotyping images to genotyping images alignment.
1. Processing of SBS data. Reads in SBS foci data from CellProfiler outputs and generates per-cell barcode/gene assignment and per-site SBS metrics.
2. Merging SBS data and morphology data. Reads in phenotyping data from CellProfiler outputs and merges it with SBS data to generate a single-cell level dataset. Outputs a list of folders/sites that were used for single-cell dataset so that subsequent steps do not need access to the CellProfiler output data.
3. Summarizing SBS data. Creates QC plots and reports for SBS data and cell assignment. Generates a total barcode count summary for comparison to NGS.
4. Aggregating single cell profiles. Creates guide- and gene-level profiles by aggregating single-cell profiles using Pycytominer.
5. Normalizing profiles. Normalizes all profiles on a per-plate basis using Pycytominer.
6. Feature selection. Performs feature selection on normalized profiles using Pycytominer.
7. Explore. Performs preliminary exploration of biological relationships in the profiles output by the recipe.

The recipe includes creation of a log file for each step. The file is named after the step (e.g. 1.process-SBS.log) and saves in a logs/ folder. The file logs progress information, warnings, and uncaught exceptions.

The recipe supports running the workflow using Python in a conda environment or using Nextflow in either a conda environment or in a Docker.

This step can be skipped if you are using Nextflow with Docker.

Install conda. We use conda as an environment manager.

# Install computational environment
conda env create --force --file environment.yml

# Initialize the environment
conda activate pooled-profiling

Edit the Experiment configuration file to set parameters specific to your experiment. More information about the parameters can be found in the config documentation.

Note that there is also a Defaults configuration file that contains parameters that are less likely to need changing between experiments. These include default assumptions about the structure of this repository and CellProfiler naming conventions.

python run.py

If you would like to pass configuration files that have custom location or naming, you can do so with the following command:

python run.py --defaults-config path/to/defaults.yaml --experiment-config path/to/experiment.json

The same recipe steps can be run as a Nextflow pipeline (main.nf) instead of run.py. Each step is still one of the scripts/N.*.py files above; Nextflow just orchestrates which steps run and in what order, based on the perform_* flags in your experiment config.

Install Nextflow (see the Nextflow docs), activate the pooled-profiling conda environment from Step 1, then run:

nextflow run main.nf

To use custom configuration file locations:

nextflow run main.nf --defaults_config path/to/defaults.yaml --experiment_config path/to/experiment.json

Instead of installing the pooled-profiling conda environment locally (Step 1), you can run every step inside a Docker container built from the same environment.yml. Install Docker and make sure the daemon is running, then build the image once:

docker build -t pooled-profiling .

Then run the workflow with the docker profile instead of activating a conda environment:

nextflow run main.nf -profile docker

The container only contains the Python environment; your repo checkout and data directories (file_location, library_location) are bind-mounted in automatically based on your config files, so editing scripts or configs never requires rebuilding the image — only changes to environment.yml do.