Park an archived project toolkit!

DISCLAIMERS:

• works only on BIOWULF or HELIX
• moves files to HPC-DME

When a project comes to a completion, most analysts have folders (or .tar files) on Biowulf/Helix which contain either:

• rawdata eg. Fastq files.
• processed data generated by CCBR pipelines and/or other downstream analysis custom tools.

The analyst can use parkit to park these folders directly on to HPCDME's CCBR_Archive object store vault. A typical project, say ccbrXYZ, can be parked at /CCBR_Archive/GRIDFTP/Project_CCBR-XYZ with collections "Analysis" and "Rawdata".

!!! note projark command is preferred for CCBR project arkiving

projark is the preferred interface for archiving and retrieving CCBR project data.

• deposit: archive a local project folder to HPC-DME.
• retrieve: pull archived objects back from HPC-DME to a local staging folder (default /scratch/$USER).

Version:

projark --version

Archive (deposit) example:

projark deposit --folder /data/CCBR/projects/CCBR-12345 --projectnumber 12345 --datatype Analysis

Archive (deposit) short-flag example:

projark deposit -f /data/CCBR/projects/CCBR-12345 -p CCBR-12345 -d Analysis -s 250 -k

Retrieve selected files example:

projark retrieve --projectnumber 12345 --datatype Analysis --filenames new.tar_0001,new.tar_0002 --unsplit

Retrieve selected files short-flag example:

projark retrieve -p CCBR-12345 -d Analysis -n new.tar_0001,new.tar_0002 -u

Retrieve full collection example (omit --filenames):

projark retrieve --projectnumber 12345 --datatype Analysis --unsplit

Useful deposit flags:

• -f, --folder <path>: input folder to archive.
• -p, --projectnumber <value>: project identifier.
• -d, --datatype <Analysis|Rawdata>: datatype (default Analysis).
• --tarname <name>.tar: override default tar name.
• -t, --tarname <name>.tar: override default tar name.
• -s, --split-size-gb <N>: split threshold/chunk size (default 500 GB).
• --staging-dir <path>: base folder for the tar, filelist, split parts and .md5 files (default $PROJARK_STAGING_DIR, else /scratch/$USER); files go in <path>/CCBR-<projectnumber>/<datatype>/.
• -k, --no-cleanup: keep the staging folder after successful transfer.

Useful retrieve flags:

• -f, --folder /path: local base folder (default <staging base>/CCBR-<projectnumber>).
• --staging-dir <path>: download to <path>/CCBR-<projectnumber>/<datatype>/ (cannot be combined with --folder).
• -p, --projectnumber <value>: project identifier.
• -d, --datatype <Analysis|Rawdata>: datatype (default Analysis).
• -n, --filenames a,b,c: retrieve specific objects.
• -u, --unsplit (alias --unspilt): merge downloaded split tar parts.

--projectnumber normalization:

• Accepts any non-empty value.
• Repeated leading prefixes ccbr, CCBR, Ccbr are removed (each may be followed by _, -, or nothing).
• Examples:
  • CCBR-1234 -> 1234
  • CCBR-abcd -> abcd
  • ccbr_ccbr-1234abc -> 1234abc

Path behavior:

• --folder FASTQ and --folder FASTQ/ are both accepted.
• Relative --folder values are resolved to absolute paths before use.

Staging location and safety checks:

• The staging base is --staging-dir, else $PROJARK_STAGING_DIR, else /scratch/$USER. A custom base must already exist and be writable.
• Before creating any HPC-DME collection, deposit estimates the tar and .filelist sizes and aborts if the staging filesystem lacks free space (about 2x the tar size when it will be split, plus the filelist and a margin; a marked staging folder from a previous run counts as reclaimable). retrieve runs a similar check before downloading.
• The check uses filesystem free space and cannot see per-user/group quotas (e.g. /data/$USER, /data/CCBR); check those with checkquota.
• deposit aborts if the staging folder <base>/CCBR-<projectnumber> overlaps the folder being archived.
• deposit marks its staging folder with a .projark_staging file and only deletes (on rerun or cleanup) folders that carry this marker. An existing unmarked CCBR-<projectnumber> folder (e.g. left by an older version with -k) makes deposit abort; check its contents and remove it, or use another --staging-dir.
• retrieve never writes the marker, and it refuses to download into a folder that has one, or into any folder inside it (a leftover deposit staging folder), because a later deposit would delete it. Remove the leftover folder or choose another --folder/--staging-dir.

Use a larger filesystem for staging (example):

export PROJARK_STAGING_DIR=/data/$USER/staging
projark deposit -f /data/CCBR/projects/CCBR-12345 -p 12345 -d Analysis

Runtime logging and notifications:

• Log lines are timestamped in ISO 8601 format (for example 2026-02-19T14:37:52-05:00).
• Completion/failure email is sent to $USER@nih.gov.
• Notification sender is NCICCBR@mail.nih.gov.

• On helix or biowulf you can get access to parkit by loading the appropriate mamba env

mamba activate /vf/users/CCBR_Pipeliner/db/PipeDB/miniforge3/envs/parkit_3

• HPC_DME_APIs package needs to be cloned and set up correctly. Follow this setup guide: https://ccbr.github.io/HowTos/docs/HPCDME/setup.html

• HPC_DM_UTILS environmental variable should be preset before calling parkit. It also needs to be passed as an argument to parkit_folder2hpcdme and parkit_tarball2hpcdme end-to-end workflows.

• HPC_DM_JAVA_VERSION minimum required value is 23 as of today.

• Run all operations from tmux, screen, or an Open OnDemand graphical session.

• Disclaimer: Open OnDemand is currently available only on Biowulf compute nodes, not directly on Helix. Since projark is Helix-only today, use tmux/screen on Helix; Open OnDemand support is future-facing until Helix access is available.

• If mamba is not already in your PATH, add the following block to your ~/.bashrc or ~/.zshrc, then run mamba activate /vf/users/CCBR_Pipeliner/db/PipeDB/miniforge3/envs/parkit_3:

# >>> mamba initialize >>>
# !! Contents within this block are managed by 'mamba shell init' !!
export MAMBA_EXE='/vf/users/CCBR_Pipeliner/db/PipeDB/miniforge3/bin/mamba';
export MAMBA_ROOT_PREFIX='/vf/users/CCBR_Pipeliner/db/PipeDB/miniforge3';
__mamba_setup="$("$MAMBA_EXE" shell hook --shell zsh --root-prefix "$MAMBA_ROOT_PREFIX" 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__mamba_setup"
else
    alias mamba="$MAMBA_EXE"  # Fallback on help from mamba activate
fi
unset __mamba_setup
# <<< mamba initialize <<<

parkit --help

Output:

usage: parkit [-h] [-v]
              {createtar,tarprep,checkapisync,syncapi,createmetadata,createemptycollection,deposittar,ls}
              ...

parkit subcommands to park data in HPCDME

positional arguments:
  {createtar,tarprep,checkapisync,syncapi,createmetadata,createemptycollection,deposittar,ls}
                        Subcommand to run
    createtar           create tarball(and its filelist) from a project
                        folder.
    tarprep             prepare tarball for upload.
    checkapisync        check whether HPC_DME_APIs is at its latest stable
                        release (or --ref)
    syncapi             move HPC_DME_APIs to its latest stable release (or
                        --ref) and generate a fresh token
    createmetadata      create the metadata.json file required for a tarball
                        (and its filelist)
    createemptycollection
                        creates empty project and analysis collections
    deposittar          deposit tarball(and filelist) into vault
    ls                  list data objects under any HPC-DME collection path

options:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit

CBIIT/HPC_DME_APIs no longer publishes GitHub tags or releases. Each release lives on a releases/X.Y.Z branch, and a release counts as stable once that branch is merged into master. Newer releases/* branches that are not merged yet are still in development.

By default, checkapisync and syncapi target the latest stable release: the highest-numbered releases/X.Y.Z branch merged into master. Versions are compared numerically (3.10.0 > 3.9.0), and names that are not exactly releases/X.Y.Z (for example releases/1.5.2-candidate) are ignored.

To use a different tag or branch, pass --ref or set HPC_DME_APIs_REF (also honored by the projark sync check). X.Y.Z is shorthand for releases/X.Y.Z:

parkit syncapi --ref 3.31.0          # releases/3.31.0
export HPC_DME_APIs_REF=3.31.0       # pin for syncapi, checkapisync and projark

Repo resolution order (both commands):

1. --repo /path/to/HPC_DME_APIs
2. HPC_DME_APIs environment variable
3. Parent directory of HPC_DM_UTILS (if it ends with utils)

Check whether your local HPC_DME_APIs clone is at the target ref (latest stable release, or --ref/HPC_DME_APIs_REF).

• Fetches master and all releases/* branches (or only the requested ref); your checkout is not changed.
• In sync when HEAD is at the target commit (or has identical content).
• Unless a ref is pinned, a local release newer than the latest stable release is also accepted, so you are never pushed to downgrade.
• Exit codes: 0 in sync, 1 out of sync, 2 configuration error (repo not found, unknown ref).

Example:

parkit checkapisync

Output:

HPC_DME_APIs=/data/$USER/HPC_DME_APIs
branch=releases/3.27.0
local=ebbdd298b24bd96fce306fd4d895edd563d2aa14
local_release=3.27.0 (latest stable release, branch)
target_sha=83222692c7e9fdbecd57731a6aeb96fae34bb519
target_release=3.30.1
OUT OF SYNC: local release 3.27.0 is older than releases/3.30.1. Run 'parkit syncapi' to update.

Move your local HPC_DME_APIs clone to the target ref and generate a fresh API token. You can also use this command just to refresh your token when the repo is already at the target.

What it does:

1. Resolves the repo and the target ref (latest stable release, or --ref/HPC_DME_APIs_REF).
2. Moves the clone to the target without creating merge commits:
   • branch: checks out a local branch of the same name tracking origin, fast-forwarding it if it already exists
   • tag: checks out the tag commit (detached HEAD)
   • never downgrades automatically: if your local release is newer than the latest stable release, the repo is left as is; an explicit --ref may downgrade (with a warning)
   • aborts if tracked files have uncommitted changes, or if the local branch has commits not on origin; untracked/ignored files such as utils/hpcdme.properties are kept
3. Verifies HEAD is at the target commit and warns if utils/version names a different hpc-cli major.minor.
4. Runs source <repo>/utils/functions && dm_generate_token, prompting for your password.

Exit codes: 0 success, 1 sync or token failure, 2 configuration error.

Example:

parkit syncapi

Output:

syncapi: starting HPC_DME_APIs sync and token refresh
HPC_DME_APIs=/data/$USER/HPC_DME_APIs
syncapi: [1/4] fetching release branches and resolving target ...
syncapi: target is releases/3.30.1 (latest stable release, branch)
syncapi: [2/4] moving local repo to target ...
syncapi: local releases/3.27.0 (release 3.27.0) -> target releases/3.30.1 (release 3.30.1)
syncapi: now at releases/3.30.1 (83222692c7).
syncapi: [3/4] locating HPC_DME_APIs functions file ...
syncapi: [4/4] generating a fresh token via dm_generate_token ...
syncapi: interactive prompt expected; enter your host password when asked.
Enter host password for user '<username>':
Now you are in sync and ready to run other parkit commands.

If token generation fails (for example ERROR MESSAGE: Access Denied: LDAP authentication failed), the repo has already been moved to the target, but you are not ready for data operations until a token is generated. Re-run parkit syncapi with valid credentials; if it still fails, inspect HPC_DME_APIs/utils/temp/log.

• Say you want to archive /data/CCBR/projects/CCBR-12345 folder to /CCBR_Archive/GRIDFTP/Project_CCBR-12345 collection on HPC-DME
• you can run the following commands sequentially to do this:

# create the tarball
parkit createtar --folder /data/CCBR/projects/ccbr_12345
# the above command will creates the following files:
# - ccbr_12345.tar
# - ccbr_12345.tar.md5
# - ccbr_12345.tar.filelist
# - ccbr_12345.tar.filelist.md5

# create an empty collection on HPC-DME
parkit createemptycollection --dest /CCBR_Archive/GRIDFTP/Project_CCBR-12345 --projectdesc "testing" --projecttitle "test project 1"
# the above command creates collections:
# - /CCBR_Archive/GRIDFTP/Project_CCBR-12345
# - /CCBR_Archive/GRIDFTP/Project_CCBR-12345/Analysis
# - /CCBR_Archive/GRIDFTP/Project_CCBR-12345/Rawdata

# create required metadata
parkit createmetadata --tarball /data/CCBR/projects/ccbr_12345.tar --dest /CCBR_Archive/GRIDFTP/Project_CCBR-12345
# if ccbr_12345.tar is rawdata then "--collectiontype Rawdata" argument needs to be added to the above commandline

# deposit the tar into HPC-DME
parkit deposittar --tarball /data/CCBR/projects/ccbr_12345.tar --dest /CCBR_Archive/GRIDFTP/Project_CCBR-12345
# if ccbr_12345.tar is rawdata then "--collectiontype Rawdata" argument needs to be added to the above commandline

# bunch of extra files are created in the process
ls /data/CCBR/projects/ccbr_12345.tar*

```bash
# delete the recently parked project folder contents including hidden contents
rm -rf /data/CCBR/projects/CCBR-12345/*

# copy filelist into the empty project folder for future quick reference
cp /data/CCBR/projects/ccbr_12345.tar.filelist /data/CCBR/projects/CCBR-12345/ccbr_12345.tar.filelist

# delete files created by parkit
rm -f /data/CCBR/projects/ccbr_12345.tar*

# test results with
dm_get_collection /CCBR_Archive/GRIDFTP/Project_CCBR-12345
# Done!

Output from ls /data/CCBR/projects/ccbr_12345.tar*:

/data/CCBR/projects/ccbr_12345.tar           /data/CCBR/projects/ccbr_12345.tar.filelist.md5            /data/CCBR/projects/ccbr_12345.tar.md5
/data/CCBR/projects/ccbr_12345.tar.filelist  /data/CCBR/projects/ccbr_12345.tar.filelist.metadata.json  /data/CCBR/projects/ccbr_12345.tar.metadata.json

We also have end-to-end slurm-supported folder-to-hpcdme and tarball-to-hpcdme workflows:

• parkit_folder2hpcdme
• parkit_tarball2hpcdme and
• projark [ recommended for archiving CCBR projects to GRIPFTP folder under CCBR_Archive ]

If run with --executor slurm this interfaces with the job scheduler on Biowulf and submitted individual steps of these E2E workflows as interdependent jobs.

parkit_folder2hpcdme --help

Output:

usage: parkit_folder2hpcdme [-h] [--restartfrom RESTARTFROM] [--executor EXECUTOR] [--folder FOLDER] [--dest DEST] [--projectdesc PROJECTDESC]
                            [--projecttitle PROJECTTITLE] [--rawdata] [--cleanup] --hpcdmutilspath HPCDMUTILSPATH [--version]

End-to-end parkit: Folder 2 HPCDME

options:
  -h, --help            show this help message and exit
  --restartfrom RESTARTFROM
                        if restarting then restart from this step. Options are: createemptycollection, createmetadata, deposittar
  --executor EXECUTOR   slurm or local
  --folder FOLDER       project folder to archive
  --dest DEST           vault collection path (Analysis goes under here!)
  --projectdesc PROJECTDESC
                        project description
  --projecttitle PROJECTTITLE
                        project title
  --rawdata             If tarball is rawdata and needs to go under folder Rawdata
  --cleanup             post transfer step to delete local files
  --hpcdmutilspath HPCDMUTILSPATH
                        what should be the value of env var HPC_DM_UTILS
  --version             print version

parkit_tarball2hpcdme --help

Output:

usage: parkit_tarball2hpcdme [-h] [--restartfrom RESTARTFROM] [--executor EXECUTOR] [--tarball TARBALL] [--dest DEST]
                             [--projectdesc PROJECTDESC] [--projecttitle PROJECTTITLE] [--cleanup] --hpcdmutilspath HPCDMUTILSPATH
                             [--version]

End-to-end parkit: Tarball 2 HPCDME

options:
  -h, --help            show this help message and exit
  --restartfrom RESTARTFROM
                        if restarting then restart from this step. Options are: createemptycollection, createmetadata, deposittar
  --executor EXECUTOR   slurm or local
  --tarball TARBALL     project tarball to archive
  --dest DEST           vault collection path (Analysis goes under here!)
  --projectdesc PROJECTDESC
                        project description
  --projecttitle PROJECTTITLE
                        project title
  --cleanup             post transfer step to delete local files
  --hpcdmutilspath HPCDMUTILSPATH
                        what should be the value of env var HPC_DM_UTILS
  --version             print version