The Anton Project at the National Resource for

The Anton Project at the National Resource for Biomedical Supercomputing HPC User Forum Houston, TX April 6, 2011 © 2011 Pittsburgh Supercomputing Center

Contributors • Joel Stiles, Ph. D, M. D. Director, NRBSC • Markus Dittrich, Senior Scientific Specialist NRBSC • Ralph Roskies, Scientific Director, PSC • Jim Kasdorf, Director of Special Projects, PSC © 2011 Pittsburgh Supercomputing Center

Disclaimer • Nothing in this presentation represents an official view or position of the U. S. National Institutes of Health, nor of D. E. Shaw Research, nor of the Pittsburgh Supercomputing Center nor of Carnegie Mellon University. All material is from public sources. All errors are my own (JK). © 2011 Pittsburgh Supercomputing Center

The PSC Biomedical Initiative • Founded in 1987 • Funded by U. S. National Institutes of Health – First extramural NIH supercomputing program • Renewed every five years • With 2006 grant, renamed: National Resource for Biomedical Super. Computing © 2011 Pittsburgh Supercomputing Center

National Resource for Biomedical Supercomputing • Performs leading-edge research in high performance computing and the life sciences • Fosters exchange between PSC expertise in computational science and biomedical researchers nationwide © 2011 Pittsburgh Supercomputing Center

National Resource for Biomedical Supercomputing • Computational biomedical research • Outreach to the national biomedical research community – – User training Software distribution Publications Online courses and workshop webcasts. © 2011 Pittsburgh Supercomputing Center

NRBSC Research Areas • Microphysiology • Volumetric visualization and analysis • Computational structural biology © 2011 Pittsburgh Supercomputing Center

Anton • Anton van Leeuwenhoek • (October 24, 1632 – August 26, 1723) • Perfected the microscope • “Father of microbiology” © 2011 Pittsburgh Supercomputing Center

Anton • Massively parallel supercomputer • Designed and built by D. E. Shaw Research (DESRES) • Special-purpose system for Molecular Dynamics (MD) simulations of proteins and other biological macromolecules © 2011 Pittsburgh Supercomputing Center

David E. Shaw • • B. S. / M. S. UCSD Ph. D. Stanford, 1980 Columbia CS Department until 1986 Formed D. E. Shaw Group in 1988 – “King Quant” (Fortune, Feb. 5, 1996) – Assets Under Management ~$20 B • Net worth: $2. 2 B (March 2011) © 2011 Pittsburgh Supercomputing Center

David E. Shaw • Founded DESRES, 2002 • Senior Research Fellow, Columbia University, 2005 • PCAST appointments, 1994 and 2009 • Council of Competitiveness Executive Committee • Fellow of AAAS – American Academy of Arts and Sciences – American Association for the Advancement of Science © 2011 Pittsburgh Supercomputing Center

D. E. Shaw Research (www. deshawresearch. com) • Independent research laboratory • Conducts basic scientific research in computational biochemistry • Involved primarily in: • The design of novel algorithms and machine architectures for high-speed molecular dynamics (MD) simulations of proteins and other biological macromolecules • The application of these simulations to basic scientific research in structural biology and biochemistry, and to the process of computer-aided drug design © 2011 Pittsburgh Supercomputing Center

The MD Challenge Molecular dynamics (MD) simulation allows scientists to study the microscopic dynamics of biomolecular systems and thus gain crucial insight into processes such as: • Protein folding • How drug molecules interact with proteins and nucleic acids • The mode of operation of many important cellular proteins such as ion channels or other transport proteins Importantly, most relevant biological events take place on a timescale of milliseconds (10 -3 s) or longer © 2011 Pittsburgh Supercomputing Center

Anton • Anton is a special-purpose supercomputer designed to specifically address the need of MD simulations to access the biologically-relevant realm of millisecond timescales. • Anton was designed to dramatically increase the speed of MD simulations. • Anton runs MD simulations fully in hardware (as opposed to running MD software on general purpose hardware). © 2011 Pittsburgh Supercomputing Center

Anton • Compared to the previous state-of-the-art Anton provides a speedup of ~100 x, rendering millisecond timescales attainable • Anton uses custom-designed ASICs and novel simulation algorithms © 2011 Pittsburgh Supercomputing Center

Anton Processing Node © 2011 Pittsburgh Supercomputing Center

Anton ASIC © 2011 Pittsburgh Supercomputing Center

A 512 -Node Anton Machine © 2011 Pittsburgh Supercomputing Center

Anton Benchmarks DHFR benchmark, simulated biological time) Computational platform single CPU codes Parallel supercomputers Anton (512 node machine) © 2011 Pittsburgh Supercomputing Center Speed (ns/day) ~1 ~100 17, 400

MD Simulations on Anton • Standard, unbiased MD runs in NVE, NVT, and NPT ensembles are possible. • Simulate proteins, lipids, water, RNA, DNA. • On a 512 -processor Anton, simulation systems with up to 120, 000 atoms can be simulated effectively. • Supports standard MD forcefields (CHARMM, AMBER, OPLSAA, . . . ). • Input is in Maestro format (similar to what is used in the DESRES Desmond code). © 2011 Pittsburgh Supercomputing Center

Anton Benchmarks Chemical system (PDB ID) System Size (# atoms) DHFR (5 dfr) 23, 558 a. SFP (1 sfp) 48, 423 11. 7 Fts. Z (1 fsz) 98, 236 5. 7 T 7 Lig (1 a 01) 116, 650 5. 5 Speed (μs/machine day) All simulations used 2. 5 -femtosecond time steps with long-range interactions evaluated at every other time step and a Berendsen thermostat applied every 100 time steps © 2011 Pittsburgh Supercomputing Center 17. 4

Anton @NRBSC/PSC • The availability of Anton at the National Resource for Biomedical Supercomputing (NRBSC) at PSC was made possible by a partnership with DESRES. • DESRES made available an Anton machine without cost for non-commercial research use by not-for-profit institutions. • The Anton at NRBSC is the only one available outside DESRES. © 2011 Pittsburgh Supercomputing Center

Anton @NRBSC/PSC • Operations are funded by a two-year $2. 7 million grant from the National Institute of General Medical Sciences (NIGMS), part of NIH. • This is the first time NIH has provided funds to make available a supercomputing system for national use. • Grand Opportunities (GO) grant mechanism from ARRA funds • One of only 14 GO awards from NIGMS © 2011 Pittsburgh Supercomputing Center

© 2011 Pittsburgh Supercomputing Center

Anton Hardware @NRBSC • 512 -node Anton production machine – Four cabinets, each with 128 special-purpose ASICs – Operational since October 2010 • 64 -node Anton test machine – One cabinet with 64 special-purpose ASICs – Operational since May 2010 Access to both Anton back-ends is via a Linux frontend running a standard PBS queuing system. © 2011 Pittsburgh Supercomputing Center

Anton Hardware @NRBSC • Anton. FS storage cluster – Dedicated to Anton MD trajectory storage – Lustre-based file system with ~450 TB of storage • Kollman analysis cluster – Dedicated to analysis of MD trajectories generated on Anton – Four nodes each with 12 cores (2 * 3. 33 GHz Intel Xeon Westmere EP X 5680 6 -core), 96 GB of RAM and one NVIDIA Tesla C 2050 card, QDR IB © 2011 Pittsburgh Supercomputing Center

Anton Allocations • In spring of 2010 NRBSC solicited proposals from the scientific community; 67 proposals were received. • Proposal review and assignment of allocations was conducted by a committee convened by the National Research Council at the National Academies of Science. • 15 proposals were awarded 100, 000 node-hours each during the first phase of the project (October 2010 through March 2011). © 2011 Pittsburgh Supercomputing Center

Anton Allocations • 32 proposals were awarded ~50, 000 node hours each during the second phase of the project (April 2011 through September 2011). © 2011 Pittsburgh Supercomputing Center

Science Projects on Anton The range of science topics proposed for simulation on Anton is quite diverse: • Studies of membrane channels and transporters • Protein folding • Design and evaluation of novel enzyme catalysts • DNA sequencing • Protein dynamics • Signal transduction • Force field development • …. © 2011 Pittsburgh Supercomputing Center

Science Projects on Anton Currently (as of March 2011), the investigators of phase 1 are wrapping up their simulations and are conducting preliminary analyses of their generated data. © 2011 Pittsburgh Supercomputing Center

References • “New Architectures for a New Biology”, Martin Deneroff, February 22, 2007 www. nersc. gov/projects/SDSA/meetings/SIAM_CS E 07/SIAM_2 -07 -desres. pdf • “Anton, a special-purpose machine for molecular dynamics simulation”, D. E. Shaw, et al. , Communications of the ACM, Volume 51, Issue 7, July 2008 • “Herculean Device for Molecular Mysteries”, John Markoff, The New York Times, July 8, 2008 © 2011 Pittsburgh Supercomputing Center

References • “Millisecond-scale molecular dynamics simulations on Anton”, D. E. Shaw, et al. , SC’ 09 Proceedings • “Exploiting 162 -Nanosecond End-to-End Communication Latency on Anton”, Ron O. Dror, et al. , SC 10 Proceedings © 2011 Pittsburgh Supercomputing Center

References • “Atomic-Level Characterization of the Structural Dynamics of Proteins”, D. E. Shaw, et al. Science 15 October 2010: Vol. 330 no. 6002 pp. 341 -346 • “Custom-Built Supercomputer Brings Protein Folding Into View”, Robert F. Service, Science 15 October 2010: 308 -309. © 2011 Pittsburgh Supercomputing Center

References • “Anton Supercomputer Proves Its Mettle”, Chemical & Engineering News, October 14, 2010 http: //pubs. acs. org/cen/news/88/i 42/8842 notw 1. htm • “Supercomputer sets protein-folding record”, Nature, October 14, 2010, http: //www. nature. com/news/2010/101014/full/news. 2010. 541. html © 2011 Pittsburgh Supercomputing Center

Jim Kasdorf kasdorf@psc. edu www. nrbsc. org © 2011 Pittsburgh Supercomputing Center