Skip to content

add run_batch script formatted for CPG #179

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ErinWeisbart merged 19 commits into from
Nov 5, 2024
Merged

add run_batch script formatted for CPG #179

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
6416ad7
add run_batch script formatted for CPG
ErinWeisbart Aug 22, 2024
ccb124f
add run_batch_CPG to docs
ErinWeisbart Aug 22, 2024
a3ae1f7
run_batch_general to CLI, cleaned up
ErinWeisbart Aug 27, 2024
b695c03
black
ErinWeisbart Aug 27, 2024
d84b695
run_batch_general to CLI docs
ErinWeisbart Aug 27, 2024
dd15e19
typo
ErinWeisbart Aug 30, 2024
0575cb1
typo
ErinWeisbart Aug 30, 2024
cf11b2b
list
ErinWeisbart Aug 30, 2024
4c76aaa
typo
ErinWeisbart Oct 8, 2024
1cb103c
argument consistency
ErinWeisbart Oct 21, 2024
74876ee
more logical ordering
ErinWeisbart Oct 21, 2024
196050a
default empty
ErinWeisbart Oct 21, 2024
fe03c0f
make docs match argument cleanup
ErinWeisbart Oct 21, 2024
b47fbfe
make ints for checking
ErinWeisbart Oct 21, 2024
dabd821
cleanup from review
ErinWeisbart Oct 22, 2024
ceb6a42
add explicit well option
ErinWeisbart Oct 22, 2024
bd8d89c
add error catch
ErinWeisbart Oct 23, 2024
5d4c2b0
properly check no wells as ['']
ErinWeisbart Oct 24, 2024
afd3755
default default
ErinWeisbart Oct 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 60 additions & 4 deletions documentation/DCP-documentation/step_2_submit_jobs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,18 +38,74 @@ Job files that don't include it will use the default structure.
For large numbers of groups, it may be helpful to create this list separately as a .txt file you can then append into the job's JSON file.
You may create this yourself in your favorite scripting language.
Alternatively, you can use the following additional tools to help you create and format this list:
* `batches.sh` allows you to provide a list of all the individual metadata components (plates, columns, rows, etc).
* `batches.sh` allows you to provide a list of all the individual metadata components (plates, columns, rows, etc).
It then uses [GNU parallel](https://www.gnu.org/software/parallel/parallel_tutorial.html) to create a formatted text file with all the possible combinations of the components you provided.
This approach is best when you have a large number of groups and the group structure is uniform.

Example: for a 96-well plate experiment where one there are 3 plates and the experiment is grouped by Plate and Well, `batches.sh` would read:
`parallel echo '{\"Metadata\": \"Metadata_Plate={1},Metadata_Well={2}{3}\"},' ::: Plate1 Plate2 Plate3 ::: A B C D E F G H ::: 01 02 03 04 05 06 07 08 09 10 11 12 | sort > batches.txt`
* You may also use the list of groupings created by calling `cellprofiler --print-groups` from the command line (see [here](https://github.com/CellProfiler/CellProfiler/wiki/Adapting-CellProfiler-to-a-LIMS-environment#cmd) and [here](https://github.com/CellProfiler/Distributed-CellProfiler/issues/52) for more information).
Note that for job files that specify groupings in this way, the `output_structure` variable is NOT optional - it must be specified or an error will be returned.
`parallel echo '{\"Metadata\": \"Metadata_Plate={1},Metadata_Well={2}{3}\"},' ::: Plate1 Plate2 Plate3 ::: A B C D E F G H ::: 01 02 03 04 05 06 07 08 09 10 11 12 | sort > batches.txt`
* You may also use the list of groupings created by calling `cellprofiler --print-groups` from the command line (see [here](https://github.com/CellProfiler/CellProfiler/wiki/Adapting-CellProfiler-to-a-LIMS-environment#cmd) and [here](https://github.com/CellProfiler/Distributed-CellProfiler/issues/52) for more information).
Note that for job files that specify groupings in this way, the `output_structure` variable is NOT optional - it must be specified or an error will be returned.

## Alternate job submission: run_batch_general.py

We also support an alternate second path besides `submitJobs` to create the list of jobs - the `run_batch_general.py` file.
This file essentially serves as a "shortcut" to run many common types of stereotyped experiments we run in our lab.
Essentially, if your data follows a regular structure (such as N rows, N columns, N grouping, a particular structure for output, etc.), you may find it useful to take and modify this file for your own usage.
We recommend new users use the `submitJobs` pathway, as it will help users understand the kinds of information Distributed-CellProfiler needs in order to run properly, but once they are comfortable with it they may find `run_batch_general.py` helps them create jobs faster in the future.

As of Distributed-CellProfiler 2.2.0, `run_batch_general.py` has been reformatted as a CLI tool with greatly enhanced customizeability.
`run_batch_general.py` must be passed 5 pieces of information:

### Required inputs

* `step` is the step that you would like to make jobs for.
Supported steps are `zproj`, `illum`, `qc`, `qc_persite`, `assaydev`, and`analysis`
* `path_style` is the style of the input and output paths.

@bethac07 bethac07 Oct 22, 2024

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about making this an optional parameter, with default as default? I know you personally do a lot of CPG processing but I don't know that I think that's going to be a general-enough phenomenon to want to make this flag mandatory on every run.

@bethac07 bethac07 Oct 22, 2024

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(But I want to know what you think!)

@ErinWeisbart ErinWeisbart Oct 22, 2024

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess it depends on what we think of as the purpose of run_batch_general.py. Is it our internal resource that we go ahead and make public or is it a public resource that we use a lot internally? If the former then default as default makes sense, but I guess I was thinking of it as the latter, especially as we've been both curating tools that access the CPG and pushing as much data to CPG as we can - in that case cpg as default (or keeping it as a required parameter) makes the most sense.

Supported options are `default` or `cpg` (for Cell Painting Gallery structure).
All paths can be overwritten with flags (see below).
* `identifier` is the project identifier (e.g. "cpg0000-jump-pilot")

@bethac07 bethac07 Oct 22, 2024

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we have a non-cpg example also, please-and-thanks?

* `batch` is the name of the data batch (e.g. "2020_11_04_CPJUMP1")
* `platelist` is the list of plates to process.
Format the list in quotes with individual plates separated by commas (e.g. "Plate1,Plate2,Plate3")

@bethac07 bethac07 Oct 22, 2024

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably want to also specify commas without spaces


A minimal `run_batch_general.py` command may look like:
"""bash
run_batch_general.py analysis default 2024_05_16_Segmentation_Project 2024_10_10_Batch1 "Plate1,Plate2,Plate3"
"""

### Required input for Cell Painting Gallery

Runs being made off of the Cell Painting Gallery require the additional flag of `-- source <value>` to specify the identifier-specific source of the data.

@bethac07 bethac07 Oct 22, 2024

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--source not -- source

"""bash
run_batch_general.py analysis cpg cpg0000-jump-pilot 2020_11_04_CPJUMP1 "BR00116991,BR00116992" --source broad
"""

### Plate layout flags

* `--plate-format <value>`: can be `96` or `384` and will overwrite `rows` and `columns` to produce standard 96- or 384-well plate well names (e.g. A01, A02, etc.)
* `--rows <value>`: a custom list of row labels.
Will be combined with `columns` to generate well names.
* `--columns <value>`: a custom list of column labels.
Will be combined with `rows` to generate well names.
* `--no-well-digit-pad`: Formats wells without well digit padding (e.g. A1 NOT A01)
* `--sites <value>`: a custom list of sites (fields of view) to be analyzed

### Overwrite structural defaults

* `--outputstructure <value>`: overwrite default output structure
* `--outpath <value>`: overwrite default output path
* `--inputpath <value>`: overwrite the default path to input files

### Overwrite defaults (for runs using load data .csv's and .cppipe)

* `--pipeline <value>`: overwrite the default pipeline name
* `--pipelinepath <value>`: overwrite the default path to pipelines
* `--csvname <value>`: overwrite the default load data .csv name
* `--datafilepath <value>`: overwrite the default path to load data files

### Overwrite defaults (for runs using .h5 batch files)

* `--usebatch`: use h5 batch files instead of load data csv and .cppipe files
* `--batchfile <value>`: overwrite default batchfile name
* `--batchpath <value>`: overwrite default path to the batchfile
Loading