Fully Scalable and Turnkey CryoEM Cluster Solution with

Fully Scalable and Turnkey Cryo-EM Cluster Solution with Distributed Storage and GPU Computing Shaobai Li Blue. Jay Imaging Ltd. , Suzhou. single particle LLC. CONFIDENTIAL: Do not share or distribute without written permission. Slide #1 15 September 2020

Overviews • Data Storage • Data Processing Speed What is a good solution (hardware) for Cryo. EM data processing from an engineering point of view • Principle of Data-Specific Image Processing • Example of Tiny Architecture • Results of Test Cases CONFIDENTIAL: Do not share or distribute without written permission. Slide #2 15 September 2020

Integration Cryo. EM Users Workflow & System Level Tool Level Implementation Level Cluster Software Cluster Operation System Node Program/ Algorithm CPU/GPU Codes ALU/Memory/I O bandwidth CONFIDENTIAL: Do not share or distribute without written permission. Slide #3 15 September 2020

Single Particle Workflow Data Acquisition Motion Correction Electrons camera bit CTF Estimation Micrographs Particle Picking 2 D Classification Particles Cryo. EM data generated 3 D Classification Structure before reconstruction CONFIDENTIAL: Do not share or distribute without written permission. Slide #4 15 September 2020

Data Size (Cryo-EM End) • For a 4 k × 4 k camera 4 k × 4 B = 64 MB/micrograph (gain normalized, defect corrected ) • A Titan Krios for one day 64 MB/micrograph× 3000 micrographs/day = 190 GB/day • For a DDD camera, movie micrographs acquired (30 – 40) × 190 GB/day = 5. 6 -7. 4 TB/day • For super-resolution counting mode (× 4) • For new detectors, even higher throughput CONFIDENTIAL: Do not share or distribute without written permission. Slide #5 15 September 2020

Data Size (Reconstruction) • Resolution Target 20000 ribosome particles => Resolution 4. 7 (10000 CPU-hours using RELION) 200000 ribosome particles => Resolution 3. x (100000 CPU-hours using RELION) • Storage Challenge • Hot data, Cold data, Multiple Users (raw data, micrographs, particles, to be immediately processed or archived) CONFIDENTIAL: Do not share or distribute without written permission. Slide #6 15 September 2020

Data Processing Speed Full Micrographs 3 D reconstruction • Bottleneck of Speed IO bandwidth, computation power • Tradeoff of online/offline Processing • Parallel computing, Multiple Processors multiple processors, GPUs, new algorithms 7 CONFIDENTIAL: Do not share or distribute without written permission. Slide #7 15 September 2020

Data Processing Speed 10 nodes (with a total of 120 physical cores) For each node, 64 GB RAM, and an Intel(R) Xeon(R) CPU E 5 -2667 0 (@ 2. 90 GHz) benchmark time [hr] nr MPIs Class 3 D 23: 28 60 2 x Titan-X (Pascal) GPUs, 64 GB RAM, and a 12 -core i 7 -6800 K CPU (@3. 40 GHz). benchmark time [hr] nr MPIs Class 3 D 8: 52 3 Class 3 D 6: 21 3 Class 3 D 5: 23 1 Class 3 D 4: 29 3 Class 3 D 7: 31 1 Class 2 D 11: 02 3 4 x GTX 1080 GPUs, 64 GB RAM and an Intel(R) Xeon(R) CPU E 5 -2620 v 3 (@ 2. 40 GHz) benchmark time [hr] nr MPIs Class 3 D 5: 39 3 Class 3 D 3: 42 5 Single Quadro K 5200 GPU, 64 GB of RAM, and an Intel(R) Xeon(R) CPU E 5 -2687 W v 3 (@ 3. 10 GHz) benchmark time [hr] nr MPIs Class 3 D 13: 07 1 CONFIDENTIAL: Do not share or distribute without written permission. Slide #8 15 September 2020

Data Processing Speed • Why processing Cryo. EM data is time-consuming Conclusions • Firstly, image processing on Cryo. EM data is IO intensive • Secondly, it is also computation intensive for the same amount data load, compared to other applications of image processing CONFIDENTIAL: Do not share or distribute without written permission. Slide #9 15 September 2020

Data Processing Speed Data Flow Structure Input 1 Output 1 Input 2 Output 2 Input 3 Output 3 … … Input n Output n ratio of computation and IO = scale of computation/size of data required • Examples of large computational intensity auto correlation of an image would be half IO intensive of cross correlation of two images • The reason Relatively low SNR • Moreover, there are iterative steps CONFIDENTIAL: Do not share or distribute without written permission. Slide #10 15 September 2020

Online Processing Fast CPU Slow Memory Local Hard-disk Large Small Computations are here Your Models/Images/Micrographs/Particles are here • What does online/offline processing mean? • Fast storage and slow storage CPU can only talk to memory • Reuse / Reload CONFIDENTIAL: Do not share or distribute without written permission. Slide #11 15 September 2020

Online Processing For one to one data flow DDD Fast CPU Slow Memory One input(each frame of read out data) to one output (counted frame of 1 s and 0 s) Speed real time processing (speed of data transfer) Input 1 Output 1 Input 2 Output 2 Input 3 Output 3 … Global memory leak SSDs, Redundant Arrays … Write/Read asymmetric Local Hard-disk Input n Output n CONFIDENTIAL: Do not share or distribute without written permission. Slide #12 15 September 2020

Double Socket Multiple Core CPU NUMA (Non Uniform Memory Access) A computer memory design used in multiprocessing, where the memory access time depends on the memory location relative to the processor. Under NUMA, a processor can access its own local memory faster than non-local memory. Temporal Store CPU architectures cache to exploit locality of reference in memory accesses. When one of the copies of cache is changed, the other cache must reflect that change. It has a significant overhead for image processing. CONFIDENTIAL: Do not share or distribute without written permission. Slide #13 15 September 2020

Online Processing Fast Slow CPU Host Memory GPU Device Memory Local Hard-disk Small Large • Accessibility for central processor to memories of coprocessors • Online/Offline between host and device memory host memory (~100 GB), device memory (~10 GB) CONFIDENTIAL: Do not share or distribute without written permission. Slide #14 15 September 2020

Online Processing • Use pinned memory to achieve high bandwidth • Pay attention to the data size per transfer CONFIDENTIAL: Do not share or distribute without written permission. Slide #15 15 September 2020

Online Processing Fast Slow CPU Host Memory GPU Device Memory Local Hard-disk Small Large Motion Correction Input 1 Output 1 Fourier transform, power spectrum, cross correlation, Input 2 Output 2 binning and filtering Input 3 Output 3 … ROI box, additional buffer … Shift Estimation and Translation Aligning Input n Output n CONFIDENTIAL: Do not share or distribute without written permission. Slide #16 15 September 2020

Cluster Level Fast Slow CPUs Host Memories GPUs Device Memories Local Harddisks Additional options --j (number of threads, parallel computing) --pool (data size per transfer) --no_parallel_disc_io (distributive storage, more disks) --preread_images (fastest storage) --scratch_directory (fast storage) Output 1 Input 2 Output 2 Input 3 Output 3 … Speeding up without modifying the Input 1 … Load images after each iteration Remote Storage Input n Output n algorithm code? CONFIDENTIAL: Do not share or distribute without written permission. Slide #17 15 September 2020

Single Particle Storage Solution TRADITIONAL Single. Particle solution Application/Computing nodes Hot spot SAN Controllers/ NAS Gateways Disk array storage nodes • • Metadata High performance for read/write Easily scale from 30 TB to 300 PB Reliable file-level RAID for fast recovery and high utilization Self monitoring and easy user management CONFIDENTIAL: Do not share or distribute without written permission. Slide #18 15 September 2020

Screenshot of 40 billion files 19 CONFIDENTIAL: Do not share or distribute without written permission. Slide #19 15 September 2020

Overviews • Principle of Data-Specific Image Processing • Example of Tiny Architecture • Results of Test Cases CONFIDENTIAL: Do not share or distribute without written permission. Slide #20 15 September 2020

Test Case on Multiple Nodes Illustrative system architecture H Storage C C C 4 x GTX 1070 Storage in head node with redundant array of 8 x 8 T normal hard disk For each node, Intel Xeon E 5 -2680, 256 GB CONFIDENTIAL: Do not share or distribute without written permission. Slide #21 15 September 2020

Single. Particle Hybrid Cluster • Hybrid cluster with both CPU and GPU resources • CPU nodes use high density dual-socket servers with latest Intel Xeon Scalable CPUs • GPU nodes use 4 x NVIDIA Tesla P 100 cards and 8 x GTX 1080 Ti • 56 Gb Infiniband 10 Gb. E connections with storage nodes • Commercial cluster management software and fully supported stable Open. MPI/Slurm resource management CONFIDENTIAL: Do not share or distribute without written permission. Slide #22 15 September 2020

西交大RELION 2实例 CONFIDENTIAL: Do not share or distribute without written permission. Slide #23 15 September 2020

Results of Test Cases RELION 2 Class 3 D Benchmark Time in Minutes (less is faster) 140 120 100 80 60 40 20 0 1 Node (4 x GTX 1080) 1 Node (4 x GTX 1080 Ti) 1 Node (8 x GTX 1080 Ti) 1 Nodes (4 x GTX 1070) 2 Nodes (8 x GTX 1070) 3 Nodes (12 x GTX 1070) CONFIDENTIAL: Do not share or distribute without written permission. Slide #24 15 September 2020

Trusted by leading cryo. EM labs CONFIDENTIAL: Do not share or distribute without written permission. Slide #25 15 September 2020