Introductory concepts Computer etiquette Jon Goss http aimpro

Introductory concepts: Computer etiquette Jon Goss http: //aimpro. ncl. ac. uk MMG Skills Lecture Series

Outline Get organised Consistency vs efficiency Compute machine vs file server Hierarchical calculation strategy Restarting calculations Throughput vs turn-around time Interacting with a batch queue 2 MMG Skills Lecture Series

Get organised You are in an area of study that has the potential to produce vast amounts of data Why do we need to store it? It is crucial that you adopt a comprehensive, structured filing system from the very start. Alpha-numerical file naming with index files – could be chronological. Potentially many files per directory. Hierarchical set of directories based on application Try to keep few files per directory. You’ll probably have to revise your filing system over time. Acid test: 3 When asked to locate a file, can you do it within a few seconds? Minutes? Hours? Days? MMG Skills Lecture Series

Consistency vs efficiency When you begin researching an application, you may adopt a parameter set from some previous calculations to be consistent with them The foundation work may already be there Bulk material: basis sets, lattice constants, convergence tests… This may allow direct comparison between previous and current calculations Binding / reaction energies Marker method calculations for electrical levels Structures for better starting points It reduces the probability of trivial errors – it does not eliminate them. 4 MMG Skills Lecture Series

Consistency vs efficiency Consistency is good, but not at any cost: cost The old parameter set may be inefficient in terms of CPU time per calculation. New parameters may be better designed (e. g. optimised basis sets) for this specific project In the long run it may be better to spend time at the start setting your parameters as this gives you a higher level of confidence in the accuracy of the results, and grounds you in the fundamentals of the calculations You may well find that you have to repeat large numbers of calculations if you have to revise from a sub-optimal setup… Calculating the same basic values twice is unlikely to be exemplary efficiency 5 MMG Skills Lecture Series

Consistency vs efficiency What ever you do, it should be agreed with all involved in the project, and you should choose your strategy carefully at the start: Think! 6 MMG Skills Lecture Series

Compute machine vs file server This distinction is part of “organisation”, and is important in: Avoiding duplication Avoiding loss of data Allowing access for others involved in the project Keeping access to important files for yourself Some computers are there for calculations: calculations Rhodes, Verity, Braid, Hector, HPCx, … Other machines are designed to store files: files Trueman, Snufkin… 7 MMG Skills Lecture Series

File storage We need to answer three questions: Why do we need to store files? Which files do we need to keep? How can we minimise file-space usage? 8 MMG Skills Lecture Series

File storage: why? Why do we need to store files? Scientific ethics require us to be able to back up our claims! They are one of our chief resources for future research. 9 MMG Skills Lecture Series

File storage Which files do we need to keep? Always AIMPRO standard output, plus Bandstructures: bandst. out, bandt. plt EELS / OA: bandst. out, dieln output files Mulliken: bandt. out NEB: maybe res. neb AIMVIEW: bandst. out, dump files (careful with these) DDS: maybe derivs. txt Do. S: bandst. out , dos. dump Keep all parts of a run If you have to restart a relaxation, for example, keep all parts. What don’t we keep in our permanent filing: Restart dump files! fort. 99, standard error files (e. g. aim 2. 3. 01 b. sh. e 18371) Aimpro input files (dat, pseudo-pots, hgh-pot, bandst. dat, …) unless they are modified specifically for this run, run and cannot be recreated from the aimpro output. 10 MMG Skills Lecture Series

File storage How can we minimise file-space usage? Most files we generate are ASCII They can be reduced in size by compressing them (we’ll look again at this in another lecture): bzip 2 <file> (You can learn about this by typing man bzip 2 on snufkin. ) A typical aimpro output file may be reduced in size by 80% without any loss of data ‘Bzipped’ files can be viewed, gres’d and even edited. You have a quota on most machines If you exceed it, you’ll be unable to do very much You may be prevented from logging into the machine! If there is no quota, you may fill the disk space and affect all other users 11 MMG Skills Lecture Series

Calculation strategy: hierarchy of costs The majority of the AIMPRO computational time is taken in obtaining total energies: The self-consistent cycle. Generate Ĥ Obtain ρout Obtain ρin=ρout? Done We focus on this part of the calculation for speed. 12 MMG Skills Lecture Series

Calculation strategy: hierarchy of costs Commonly, a goal may be reached in a sequential method, minimizing computational effort… …we want to maximise the amount of science we can do so we need to be able to answer the following questions: How do the decisions we make in constructing a data file, and how we run the job affect the time & efficiency? Which factors are most important? 13 MMG Skills Lecture Series

Calculation strategy: hierarchy of costs Number of atoms Number of different species Self-consistency method Number of basis functions Number of exponents Maximum orbital angular momentum Initialisation charge density basis Number of k-points Location of k-points Spin state Plane-wave basis Number of processors The amount of memory available 14 K-point parallelism Number of symmetry operations DIIS Real-space build The amount of vacuum (molecules and surfaces) Number of images in a NEB Number of k-points in a band structure Number of bands in an EELS run Number of energies in an EELS run Number of atoms included in the derivatives MMG Skills Lecture Series

Calculation strategy: hierarchy of costs Number of atoms Number of different species Self-consistency method Number of basis functions Number of exponents Maximum orbital angular momentum Initialisation charge density basis Number of k-points Location of k-points (real/complex) Spin state Plane-wave basis (vacuum) Number of processors The amount of memory available 15 K-point parallelism Number of symmetry operations DIIS (extensive parallelism) Real-space build The amount of vacuum (molecules and surfaces) Number of images in a NEB Number of k-points in a band structure Number of bands in an EELS run Number of energies in an EELS run Number of atoms included in the derivatives MMG Skills Lecture Series

Calculation strategy: hierarchy of costs For an SCF step: The time for an energy scales as nα, α~2→ 4 where n is the dimension of the Hamiltonian. Going from real arithmetic to complex (at a general k-point), the time increases by a factor of ~½-1 order of magnitude. Sampling the Brillouin-zone mp 23 generally includes 4 (complex) points for a cubic cell. Spin polarisation doubles the time taken The number of SCF steps for an energy may increase with spin-polarisation. Compare the time for an energy for a pppp, pdpp, ddpp and dddd basis. Compare the time for pppp, gamma point, spin averaged, and dddd, 4 complex k-points, spin polarised. 16 MMG Skills Lecture Series

Calculation strategy: hierarchy of costs The most common calculation we perform is a structural relaxation. If we do not have an accurate starting structure, structure it is likely that the optimisation will require multiple optimisation iterations. Why is this likely to be the case? The number of structural iterations is approximately independent of how we run AIMRPO, provided the calculation is performed within a ‘reasonable’ set of parameters (i. e. not necessarily convergent ones). What does this tell us about the energy surface? A structural iteration generally takes more SCF steps when we’re far from the structural minimum. This is related to recycling charge densities from previous structures – why does this affect the number of SCF cycles? 17 MMG Skills Lecture Series

Calculation strategy: hierarchy of costs Example: Substitutional gold in silicon. We want to know the symmetry, donor and acceptor levels. We need to check the convergence with Cell size Basis Pseudo-potential – whether the 5 d electrons are included in the valence or not How do we start? Sketch out a strategy to get all the data we need. 18 MMG Skills Lecture Series

Calculation strategy: hierarchy of costs Basis Design a hierarchical basis set sequence: C 44 G* → pdpp → dddd How do we know where we can start? And how far we need to go? 19 MMG Skills Lecture Series

Calculation strategy: hierarchy of costs Sampling k 1 Start with simplest viable sampling scheme: k 2 Gamma-point, mp 23, … k 3 k 4 20 Use k-point parallelism k 4 k 3 k 2 3 k 1 2 k 2 k 1 The Hamiltonians for different kpoints are diagonalised in parallel, rather than in serial parameter{use_kpar} If nk<np, then np must be an integer multiple of nk. If nk>np, then this is not the case. k 1 How do we know what is “viable”? viable How does this improve efficiency? What are the potential pitfalls? k 1 MMG Skills Lecture Series

Calculation strategy: hierarchy of costs Spin polarisation For spin-polarized problems, first relax spin averaged – this makes the calculation run twice as fast. What does spin averaged / polarised mean? Can we always relax S=0? Is it always really helpful? 21 MMG Skills Lecture Series

Calculation strategy: hierarchy of costs Supercell size Start with a small unit cell and embed it in larger ones: 64 → 216 → 512 → 1000 atoms Use an anchor point Take care over symmetry What are the implications for timing? 22 MMG Skills Lecture Series

Calculation strategy: hierarchy of costs Starting structures continued… Use the structure obtained from one charge state to start others. Recycle similar systems: e. g. use a phosphorus structure to start an arsenic one. If you’ve already run in LDA and you now want it GGA, scale the structure according to the ratio of standardized lengths (typically lattice constants). What sort of errors might these short-cuts lead to, if any? 23 MMG Skills Lecture Series

Calculation strategy: hierarchy of costs Recycle the charge-density! When restarting an incomplete relaxation, or restarting at the end of the relaxation to get some analytical data (e. g. AIMVIEW dumps, bandstructures, EELS spectra. . . ) Use a “restart-dump”! restart-dump What does the restart dump contain? restart{make-dump} restart{load-dump, override-positions} How can the restart be used? How much time might this save? 24 MMG Skills Lecture Series

Restarting calculations: more generally Your default mode of operation is to always write to a restart dump using: restart{make-dump, file=dump. xxxxx} You might store them all in one place filespace{/scratch/njpg/DUMP-FILES} This is what I do! The xxxxx is a unique identifier associating the dump-file with the run. The restart dump files may be very large in terms of disk-usage! Restarts also exist for other calculations: NEB This will be discussed in detail elsewhere in this course Energy second derivatives with respect to position You should check the on-line documentation 25 MMG Skills Lecture Series

After all this… Sometimes we have to bite the bullet apple(? !). 26 MMG Skills Lecture Series

Throughput vs turn-around time When running a parallel job, job there is a scaling penalty: Doubling the number of nodes will NOT half the time (in general). Why is this? It WILL prevent other people from using the nodes. You will constantly balance throughput (the number of jobs completed per day) with turn-around time (wall time from start to job completion). In general: throughput is maximised by adopting the minimum number of nodes for the job, given the batch queue time limits turn-around time is minimised by adopting larger numbers of nodes (subject to scaling). Minimising the turn-around time is anti-social behaviour Big brother is watching you. 27 MMG Skills Lecture Series

Interacting with a batch queue system The batch queue systems differ from machine to machine. It is important that you familiarise yourself with: Memory per node CPUs/cores per node Time limits Scheduling priority Disk-usage Scaling (interconnect) Reliability How do these relate to the preceding discussion? 28 MMG Skills Lecture Series

Interacting with a batch queue system Every day, check all of your jobs Running Finished For those still running, check to see: Is all well? Can/should this run be terminated? When might this be? Maximise the available resources for all users For those finished: Restart them, if need be Incomplete relaxation Analysis File them carefully 29 MMG Skills Lecture Series

Concluding thoughts Be organised Be hardware aware Be efficient Be sociable THINK about what you’re doing. 30 MMG Skills Lecture Series