Computational chemistry packages efficient usage issues Jemmy Hu

Computational chemistry packages (efficient usage issues? ) Jemmy Hu SHARCNET HPC Consultant Summer School June 3, 2016 /work/jemmyhu/ss 2016/chemistry/

Chemistry Software managed by SHARCNET Electronic Structure Theory Molecular Dynamics Gaussian (09, 03) -commercial AMBER -commercial ADF/Band -commercial Gromacs NWChem CPMD Quantum Expresso NAMD ORCA Gamess/Firefly Abinit User/group owned software (commercial) Molpro VASP Turbomole CHARMM

Nature of the software Software Parallel Purpose Gaussian Threaded (Open. MP) http: //www. gaussian. com/g_prod/g 09. htm ADF MPI https: //www. scm. com/product/adf/ NWChem MPI http: //www. nwchemsw. org/index. php/Main_Page Quantum Expresso MPI http: //www. quantum-espresso. org/ VASP MPI https: //www. vasp. at/index. php/aboutvasp/59 -about-vasp ORCA MPI https: //orcaforum. cec. mpg. de/

General policy for software • Open Software used by 2 or more groups – add to SHARCNET software stack with software usage instruction/example webpage online https: //www. sharcnet. ca/my/software • Software used by one group – user installs in his/her own space with help from SHARCNET if necessary • SHARCNET owned commercial software – free • Self owned software – user’s response for license issues • Software installed as module package, setup in your bash configuration file, i. e. , . bash_profile under /home/userid/ module load gaussian/g 09_D. 01

module load packages • Login to a cluster, e. g. , ssh user. ID@saw. sharcnet. ca • Open the shell configuration file, e. g. , vi. bash_profile • Add the module load lines at the end of the file module load gaussian/g 09_D. 01 module load adf/2014. 05 module load nwchem/6. 5 module load espresso/5. 1. 1 module load orca/3. 0 • • • Save the file Exit Re-login Goto your /work directory: cd /work/user. ID Copy the files cp –rf /work/jemmyhu/ss 2016/chemistry.

System Info Parallel Computer Memory Architectures Hybrid Distributed-Shared Memory (Cluster solution)

Size of Threaded Jobs on specific system https: //www. sharcnet. ca/my/systems System Nodes CPU/Node Memory GB (threads) (Max for full cpus usage) orca Opteron 16 or 24 30 saw Xeon 8 15 brown Xeon 8 16 Job uses less than the max threads and memory on a node will be preferable

Gaussian 09 issues • • Latest version: g 09 -D. 01 Input sample File location Memory Disk Restart Utilities https: //www. sharcnet. ca/my/software/show/21

G 09 input file: simplest case #p b 3 lyp/6 -311++g(2 df, p) opt freq << CO-6311++g(2 df, p)—B 3 LYP-Opt-Freq >> 0 1 c o 1 co co=1. 1281 (leave an blank line at the end) File name: CO-b 3 lyp-1. com

Run job, file structure • Real jobs should be submitted to the threaded queue using sqsub command • First check the files under your /work and /scratch directory open two windows, one for /work, one to view files in /scratch • For testing purpose today, we run jobs from command line on the dev. node on saw (saw-dev 1, …, saw-dev 6) ssh saw-dev 4 • run g 09 CO-b 3 lyp-1. com >& CO-b 3 lyp-1. log & • Output file CO-b 3 lyp-1. log • Runtime files (5 of them, . rwf is big) /scratch/user. ID • • • Job uses defaults [1 cpu, minimum memory (256 MB? )] Only. log output file will be generated in the run directory Runtime files will be removed automatically once finished successfully, otherwise they will stay.

Specify checkpoint file: name. chk %chk=CO-b 3 lyp-1 chk #p b 3 lyp/6 -311++g(2 df, p) opt freq << CO-6311++g(2 df, p)—B 3 LYP-Opt-Freq >> 0 1 c o 1 co co=1. 1281 Input file: CO-b 3 lyp-1 chk. com Run on dev: g 09 CO-b 3 lyp-1 chk. com >& CO-b 3 lyp-1 chk. log & Files: CO-b 3 lyp-1 chk. log, CO-b 3 lyp-1 chk. chk

G 09 input file: CO-b 3 lyp-4. com, use 4 cpus %nosave %mem=500 MB %chk=CO-b 3 lyp-4 %nproc=4 #p b 3 lyp/6 -311++g(2 df, p) opt freq << CO-6311++g(2 df, p)—B 3 LYP-Opt-Freq >> 0 1 c o 1 co co=1. 1281

Run Job in the queue %mem=500 MB (--mpp=2 x%mem, memory per job, for all cpus) %nproc=4 (-n 4) Submit command is sqsub -q threaded -n 4 -r 1. 0 d –mpp=2 g -o CO-6311 -4. out g 09 CO-6311 -4. com What’s happening afterwards? - sqjobs to see job status, job. ID - If runs, where are the files? . log and. chk files are generated in the run directory runtime files, . rwf, etc will be in /scratch/your. ID/job. ID/ - Failed? remove the runtime files, rm –rf /scratch/your. ID/job. ID. correct input file, re-run

Input file with named. rwf: CO-b 3 lyp-4 rwf. com %rwf=CO-b 3 lyp-4 rwf %nosave %mem=500 MB %chk=CO-b 3 lyp-4 rwf %nproc=4 #p b 3 lyp/6 -311++g(2 df, p) opt freq << CO-6311++g(2 df, p)—B 3 LYP-Opt-Freq >> 0 1 c o 1 co co=1. 1281 The named. rwf file will be generated in the current directory

Checkpointing 1. . chk can be used to restart an Optimization job, or used for property (freq, etc) calculation afterwards. 2. In G 09, . rwf can be used to restart a frequency calculation.

Restart (see examples) • Restart opt job from. chk %No. Save %chk=Cu_test. chk %mem=500 mb %nproc=4 #B 3 LYP/DGDZVP Opt Geom=All. Check Guess=Read (one blank line) • Restart freq from. rwf %rwf=/scratch/jemmyhu/Cu_test. rwf %No. Save %chk=Cu_test. chk %mem=500 mb %nproc=4 #p restart (one blank line)

Gaussian Utilities • Utilities formchk cubegen freqchk …… use the same version direct run from command line • NBO

Review • Input file, memory (%mem) /cpu (%nproc) • sqsub (-n, --mpp) , sqjobs, sqkill • Use job info from scheduler • File location • Failed jobs • File cleanup

ADF • ADF is frequently used for studying transition metal complexes and molecules with heavy atoms. ADF offers unique capabilities to predict molecular properties of organic electronics materials. • ADF, BAND, DFTB • https: //www. scm. com/

Use ADF on SHARCNET • Available on major clusters, viz-stations • Versions: 2013. 01, 2014. 05 • Setup, add module load line to. bash_profile, i. e. , module load adf/2014. 05 • Examples /opt/sharcnet/adf/2014. 05/examples

Default settings [jemmyhu@saw-login 1: ~/tests/test_ADF/saw/test_adf] module show adf/2014. 05 ---------------------------------/opt/sharcnet/modules/adf/2014. 05: module-whatis Provides ADF 2014. 05 binary for x 86_64 Centos 6 systems conflict adf setenv ADFROOT /opt/sharcnet/adf setenv ADFHOME /opt/sharcnet/adf/2014. 05 setenv ADFBIN /opt/sharcnet/adf/2014. 05/bin setenv ADFRESOURCES /opt/sharcnet/adf/2014. 05/atomicdata setenv SCMLICENSE /opt/sharcnet/adf/2014. 05/license setenv SCM_TMPDIR /scratch/jemmyhu setenv SCM_USETMPDIR yes prepend-path PATH /opt/sharcnet/adf/2014. 05/bin ----------------------------------

Two input options https: //www. sharcnet. ca/help/index. php/ADF/BAND • name. inp (one step a time) use a runscript ‘ADF’ or ‘BAND’, to bypass the runtime files to /scratch/username/jobid/ sqsub -q mpi --nompirun -n 8 --mpp=2 g -r 1. 0 d -o GO_H 2 O-adf. log ADF GO_H 2 O-adf. inp sqsub -q mpi --nompirun -n 8 --ppn=4 --mpp=2 g -r 1. 0 d -o GO_H 2 O-adf. log ADF GO_H 2 O-adf. inp sqsub -q mpi --nompirun -n 8 --mpp=2 g -r 1. 0 d -o name_band. log BAND name_band. inp

/home/jemmyhu/work/ss 2016/chemistry/examples/adf GO_H 2 O-adf. inp

Input for multi-steps: name. run sqsub -q mpi --nompirun -n 8 --ppn=4 --mpp=2 g -r 1. 0 d -o NMR_B 3 LYP. log. /NMR_B 3 LYP. runtime files are under /scratch/username/

Restart • TAPE 13 It serves as a checkpoint file. TAPE 13 can be used to restart the calculation at a point not too far before the fatal condition occurred TAPE 13 is upgraded during the calculation but discarded upon normal termination. To keep TAPE 13, -use the SAVE key • TAPE 21 Upon normal (i. e. program-controlled) termination of a calculation, the TAPE 21 result file can be used for restart purposes When a crash occurs, however, chances are that TAPE 21 has not correctly been closed and that its data structure is inconsistent

ADF GUI • GUI software to run on viz-workstations vdi-centos 6. user. sharcnet. ca viz-uwo 10. sharcnet. ca viz-uwo 11. sharcnet. ca • Turn on X 11 locally, then login to he workstation • adfview

https: //www. sharcnet. ca/help/index. php/NWCHEM

RUN nwchem on SHARCNET [jemmyhu@saw 333: ~/work/ss 2016/chemistry/examples/nwchem] pwd /home/jemmyhu/work/ss 2016/chemistry/examples/nwchem [jemmyhu@saw 333: ~/work/ss 2016/chemistry/examples/nwchem] ls h 2 o_hu. nw sqsub -q mpi -n 4 -r 3. 0 d --mpp=4 g -o h 2 o_hu. log nwchem h 2 o_hu. nw

https: //www. sharcnet. ca/help/index. php/ESPRESSO

https: //www. sharcnet. ca/help/index. php/ORCA

Research group owns the license Installation guideline https: //www. sharcnet. ca/help/index. php/VASP

What else? • Question? • Discussion?