Automatic Processing Pipelines with XNAT and REDCap Vanderbilt

Automatic Processing Pipelines with XNAT and REDCap Vanderbilt University 2013 -08 -20 Benjamin Yvernault, Bennett Landman, Brian Boyd, 1

Requirements At this point, you should know how to : – – – – Create project / sharing data Browse data and view image previews Search for projects, subjects, and data Download imaging data, processed, and stored meta-data Delete scans / sessions / projects Use the Pre-Archive Upload scans DICOMs Quality Assurance 2

After this short course, you will be able to: • • Create a project on REDCap Set up the design of your REDCap project Request API access for your project on REDCap What is a spider: - source code - Which spiders already exist? - Run spiders from your computer • Run a spider on your local machine • ACCRE: run spiders (status / failed / …. ) 3

Setting REDCap project for XNAT processing 4

REDCap: Setup the main instrument Variable Name: project_xnat -> type: Text Box Variable Name: subject_xnat -> type: Text Box Variable Name: experiment_xnat -> type: Text Box Variable Name: scan_type_xnat -> type: Text Box Variable Name: scan_sd_type -> type: Text Box Variable Name: script_name -> type: Text Box Variable Name: script_version -> type: Text Box Variable Name: pdf_report -> type: File Upload 5

REDCap: Upload main instrument Take time to create all this fields each time you have a new project. Variable Name: project_xnat -> type: Text Box Variable Name: subject_xnat -> type: Text Box Variable Name: experiment_xnat -> type: Text Box Variable Name: scan_type_xnat -> type: Text Box Variable Name: scan_sd_type -> type: Text Box Variable Name: script_name -> type: Text Box Variable Name: script_version -> type: Text Box Variable Name: pdf_report -> type: File Upload the main instrument dictionary as a first step. File location: http: //xnat. vanderbilt. edu -> Everything about spiders -> How to upload to REDCap -> link with the csv file. 6

REDCap: Download Shared Libraries 7

REDCap: API Key Ø To send data from XNAT to REDCap, you need to have an API key for your project on REDCap 8

REDCap Project Online Designer 9

Spiders … Ø Script executing a process on XNAT Ø Download images / data from XNAT Ø Execute a process on them Ø Copy the data to a directory Ø A specific spider to upload results on XNAT Ø All the spiders on http: //xnat. vanderbilt. edu Ø Source code in the project Masimatlab on NITRC Ø Three kind of Spiders : Ø Spider that creates results for a Scan Ø Spider that corresponds to an assessor Ø Infrastructure spiders 10

Scan Spiders Ø Converter Spider : ØSpider_dcm 2 nii_all. py (using dcm 2 nii) ØSpider_dcm 2 nii_all_v 2. py (using r 2 agui) Ø Changing attributes: ØSpider_set_scan_type. py ØSpider_Series_Description. py Ø Extracting/creating files: ØSpider_physlog. py ØSpider_Make_Preview_from_nifti. py 11

Process Spiders Ø This spiders can work on a scan level (one assessor per scan) or an experiment level (one assessor per session) Ø Spider that generates an assessor: Ø Spider_f. MRQA. py Ø Spider_VBMQA. py Ø Spider_dti. QA_v 2. py Ø Spider_label. Fusion. py Ø Spider_intra_sess_reg. py Ø Spider_nonrigi_reg_to_ATLAS. py Ø Spider_Freesurfer. Recon. All. With. QC. py Ø Spider_ON_segmentation. py 12

Infrastructure Spider Ø Spider_report_for_projects. py to create a report on projects (same than Xnatreport) Ø Spider_send_data_to_redcap. py to send data from complet processes on XNAT to REDCap Ø Spider_Process_Upload to upload the processes that finished. Any process Spiders will copy the results needed to a specific directory. You should run it when you need to upload processes. 13

Spider flow Spider_process_upload. py / Xnatprocessupload Python script Download data Spiders Folder Run process UPLOAD_SPIDER_DIR Any language script: - Matlab/SPM - Perl - shell script 14

How to run a spider? Ø Spiders can run on your local machine if you have the requirements. ØE. G: Most of the spiders use xvfb-run Øpython Spider_f. MRIQA. py -h to see the options ØYour. bashrc or. profile set up with your XNAT logs ØPython packages install Ø Specific python package to run spiders on ACCRE develop by our lab. 15

ACCRE and Spiders Ø Ø Ø Advanced Computing Center for Research & Education ACCRE is a Cluster for Vanderbilt University Python script running every night on the cluster Submitting one job per assessor Status on the assessor related to the job status: Ø Ø Ø Ø NEED_INPUTS-> the inputs files are not present on XNAT NEED_TO_RUN-> ready to be submitted on ACCRE JOB_RUNNING -> job submitted and running on ACCRE JOB_FAILED -> job stop with an error READY_TO_UPLOAD-> job ended and copy the results to the Upload directory UPLOADING-> Spider_Process_Upload running and uploading results READY_TO_COMPLETE-> a script will check and set the walltime and memory used by the job on ACCRE. Ø COMPLETE-> jobs done (processes finished and uploaded) with results on XNAT 16

ACCRE and CCI package Spider_process_upload. py / Xnatprocessupload Crontab job running every night a python script CCI Package Get assessors that need to run Folder UPLOAD_SPIDER_DIR Run jobs for each assessors (Spider flow on ACCRE) 17

Status sync between XNAT and REDCap 18

Interactive Assignment • Add to your. bashrc the line (or. profile on Mac OS): export UPLOAD_SPIDER_DIR=path • Create a project on REDCap • Upload the dictionary from http: //xnat. vanderbilt. edu – Everything about Spiders -> How to upload on REDCap • Download the shared libraries for f. MRIQA / VBMQA / dti. QA • Run the Spider VBMQA on one T 1 in the project you created in the last courses (it’s going to take 10 mins) • Look at the results and upload them using Xnatprocessupload • Delete the project 19