The nci-lp-dragen partition as of June 2023 includes one dragen server configured to run the TSO500 ctDNA pipeline. It has been funded by NCI/CCR/Lab of Pathology until 30-Sep-2026.

Notes:

• Allocation is exclusive and includes all resources on the node. This is necessary since the Dragen's FPGA accelerator can only accomodate a single run at a time.
• Desipte this all memory and CPUs should be allocated explicitly because (a) exclusive allocation does not allocate all memory and (b) while exclusive allocation automatically allocates all CPUs not explicitly specifying them in job submission can distort priority calculations in a single node partition.
• Default walltime: 6h
• Max. walltime: 5d (Lab of Pathology members) or 8h (other users)
• Interactive sessions are not enabled
• Input and output of a pipeline run should be on the local /staging disk partition.

Changelog:

• 2024-09-16: installed ctDNA version 2.6.0
• 2024-07-25: installed solid version 2.5.3
• 2024-05-20: upgraded to OL8 and re-installed ctDNA 2.1.1

Create a batch script similar to the following:

#! /bin/bash
# this is dragen.sh

# set up paths etc
source /etc/profile.d/edico.sh

RUNPATH=/path/to/runs
RUNFOLDER=NNNNNN_XXXXXX_NNNN_XXXXXXXXXX
ANALYSIS="/staging/${RUNFOLDER}-ctDNA-$(date +%s)"
METRICS=${ANALYSIS}/Results/MetricsOutput.tsv
RESULTPATH=/path/to/results

# clean up after run
trap 'rm -rf "/staging/${RUNFOLDER}" "${ANALYSIS}"' EXIT

cp -r "${RUNPATH}/${RUNFOLDER}" /staging || exit 100
mkdir -p "${ANALYSIS}" || exit 101

# may need to specify --sampleSheet if not in the default location
DRAGEN_TSO500_CTDNA-2.6.0.sh --analysisFolder "${ANALYSIS}" \
    --runFolder "/staging/${RUNFOLDER}" \
2>&1 | tee "${ANALYSIS}/test.out" || exit 102

[[ -f "${METRICS}" ]] || printf "Missing metrics file\n"
grep COMPLETED_ALL_STEPS "${METRICS}"

# copy results back to working directory
cp -r "${ANALYSIS}" "${RESULTPATH}" || exit 103

And submit with

[user@biowulf]$ sbatch --mem=0 --cpus-per-task=64 --partition nci-lp-dragen --qos=compass dragen.sh
12345678

if you have access to the compass QOS which increases priority and max. walltime. Users without access to the QOS are limited to 8h max walltime.

Note that the $ANALYSIS folder is lager than the input with Logs_Intermediates taking up most the space. The script above could be modified to only transfer a subset of files back to shared storage.

Create a batch script similar to the following:

#! /bin/bash
# this is dragen.sh

# set up paths etc
source /etc/profile.d/edico.sh

RUNPATH=/path/to/runs
RUNFOLDER=NNNNNN_XXXXXX_NNNN_XXXXXXXXXX
ANALYSIS="/staging/${RUNFOLDER}-ctDNA-$(date +%s)"
METRICS=${ANALYSIS}/Results/MetricsOutput.tsv
RESULTPATH=/path/to/results

# clean up after run
trap 'rm -rf "/staging/${RUNFOLDER}" "${ANALYSIS}"' EXIT

cp -r "${RUNPATH}/${RUNFOLDER}" /staging || exit 100
mkdir -p "${ANALYSIS}" || exit 101

# may need to specify --sampleSheet if not in the default location
DRAGEN_TruSight_Oncology_500_ctDNA.sh --analysisFolder "${ANALYSIS}" \
    --runFolder "/staging/${RUNFOLDER}" \
2>&1 | tee "${ANALYSIS}/test.out" || exit 102

[[ -f "${METRICS}" ]] || printf "Missing metrics file\n"
grep COMPLETED_ALL_STEPS "${METRICS}"

# copy results back to working directory
cp -r "${ANALYSIS}" "${RESULTPATH}" || exit 103

And submit with

[user@biowulf]$ sbatch --mem=0 --cpus-per-task=64 --partition nci-lp-dragen --qos=compass dragen.sh
12345678

if you have access to the compass QOS which increases priority and max. walltime. Users without access to the QOS are limited to 8h max walltime.

Note that the $ANALYSIS folder is lager than the input with Logs_Intermediates taking up most the space. The script above could be modified to only transfer a subset of files back to shared storage.

Please send questions and comments to staff@hpc.nih.gov