Introductory concepts PERL for AIMPRO users Jon Goss

  • Slides: 49
Download presentation
Introductory concepts: PERL for AIMPRO users Jon Goss http: //aimpro. ncl. ac. uk MMG

Introductory concepts: PERL for AIMPRO users Jon Goss http: //aimpro. ncl. ac. uk MMG Skills Lecture Series

Outline l l What is PERL? Why do you need to know? What common

Outline l l What is PERL? Why do you need to know? What common uses are there? The actual stuff through case studies: – Data file format conversion: l – Filters to unix/linux commands: l – E. g. qsub, q AIMPRO data analysis: l 2 o 2 x, x 2 o, Rescale. OTB E. g. gres, ETOT, FORCES, CHAINS, Sand. D MMG Skills Lecture Series

What is PERL? l Perl is a scripting programming language: – – – 3

What is PERL? l Perl is a scripting programming language: – – – 3 Interpreted rather than compiled (c. f. fortran and C++) Can be object oriented Sometimes linked to web systems Very “c-like” in structure, and similar to unix/linux shell environments Easy to do simple things MMG Skills Lecture Series

Why should you care? l l 4 As your studies are almost entirely based

Why should you care? l l 4 As your studies are almost entirely based on computer generated, ASCII data files, especially those produced from AIMPRO, learning a little about how to manage the numbers will be essential. Spending a little time early on to develop some skills in data analysis will save you time in the long run. MMG Skills Lecture Series

Why perl rather than something else? l l I use perl Perl is designed

Why perl rather than something else? l l I use perl Perl is designed with text-file manipulation in mind: c. f. C and Fortran. – l l If you use a compiled language (c, c++, fortran…) you are more dependent on the compiler and libraries (non-standards, compability…). However, if you need to do some real number crunching, then you are probably better off using something else. – 5 Especially regular expression matching (coming later) (Note, you can use perl to do these things. ) MMG Skills Lecture Series

Common uses l I use perl on a daily basis, writing scripts of a

Common uses l I use perl on a daily basis, writing scripts of a few lines to do something simple, and throwing it away: – – l Less frequently I write longer programs to do more sophisticated things that I expect to do again: – – 6 generating a number of similar files where only the lattice constants of the systems change (Zn 3 P 2). converting general basis vectors into spherical polars converting hyperfine tensor output from AIMPRO to a postscript graphics file, or a POVray input file. gres (general aimpro output interigator) batch queue summary FORCES, CHAINS, Sand. D, … MMG Skills Lecture Series

My “permanent” scripts on Rhodes: AB 2 exp aim 2 res Average. OTB B

My “permanent” scripts on Rhodes: AB 2 exp aim 2 res Average. OTB B 2 P 2 centre_super_cell CND dsub F Get. Lattice. Ratios HFi 2 POV Last. Mod Make. Fake. DDRESTART Monkey nsub Provec Randomize. OTB Redistribute res 2 nebxyz rsub Swap. Usage TIME XYZTransformation 7 ABC-Alp. Bet. Gam_to_abcdef analysis Average. OTBpoint bandst 2 ps CHAINS Count. Down echerr FE 2 plot Get. PSOptions HFi 2 ps latest Make. Group. Run neb 2 xyz o 2 x q Randomize. XYZ Redistribute. HGH RES 2 ROMP Sand. D table 2 sortedlist tofin Add. KFPARAM Anglular. Deviation b bun Change. Procs crit ETOT FORCES gres Homer latex 2 ps Make. HGH NEBSaddle. Points 2 xyz Plot. Basis. Params qme Recentre Redistributelib Rescale SCFTIMES TETOT x 2 o Add. Norm available B Centre. NEB CHECK Do. S ETOT 3 Get. HGHOptions Harvest. Charges iter LOADS. EXE makesphere new plt 2 ps qsub Recentre. XYZ res 2 neb RESv. EGY State. Eq Tidy. NEB XYZrotator MMG Skills Lecture Series

Case studies: Rescale, o 2 x and x 2 o l 8 In AIMPRO

Case studies: Rescale, o 2 x and x 2 o l 8 In AIMPRO data files, we use a standard format to describe the (relative) positions of the atoms in the simulation cells. This is what is sometimes referred to as “otb”: Atom number Atomic species Neighbour 1 Neighbour 2 Neighbour 3 Neighbour 4 x y z 1 1 12 16 45 109 0 0 0 2 2 12 32 41 79 1 0 0 MMG Skills Lecture Series

Case studies: Rescale, o 2 x and x 2 o l l l 9

Case studies: Rescale, o 2 x and x 2 o l l l 9 The atom number and the lists of four neighbours are only information, and do not affect the calculation The atomic species is a number corresponding to a chemical species listed elsewhere in the data file. The (x, y, z) may be in a variety of units, but we’ll assume that they’re atomic (1 a. u. ≈0. 529Å). MMG Skills Lecture Series

Case studies: Rescale, o 2 x and x 2 o l l l However,

Case studies: Rescale, o 2 x and x 2 o l l l However, this is not a well known standard More common is “xyz”, and this is one format supported by molecular structure plotting software (e. g. minimol, rasmol, jmol, xmol, etc. ) The simplest form of xyz format is as in the following example: 216 Structure data (comment line) 10 C 0. 587548 1. 276483 29. 385792 C 3. 098373 1. 239844 28. 981111 … MMG Skills Lecture Series

Case studies: Rescale, o 2 x and x 2 o l l 11 We

Case studies: Rescale, o 2 x and x 2 o l l 11 We therefore commonly want to convert between the data format you have to pass to AIMPRO, and a form you can visualise using minimol – this is prime perl-script territory. In order to write such a script we need to know some basics… MMG Skills Lecture Series

Case studies: Rescale, o 2 x and x 2 o Perl basics l A

Case studies: Rescale, o 2 x and x 2 o Perl basics l A scalar variable in a perl script (a number, string etc) is stored as a term such as “$a” – l l l 12 The dollar sign in this context indicates a scalar variable. An array (lets say a vector) is stored as “@a”. There are default scalars and vectors: $_ and @_, which are set when a perl script function returns a value or vector that is not specified. These must be looked after! Command line values are stored in a vector “@ARGV”, and the name of the script (as invoked) is stored in “$0”. MMG Skills Lecture Series

Case studies: Rescale, o 2 x and x 2 o Perl basics l To

Case studies: Rescale, o 2 x and x 2 o Perl basics l To look at the contents of a file you need to open it, or pass it as “standard input”: l l – or for standard input: l – 13 open INPUT, “<data. file” or die “I could not open data. filen”; ; $a=<INPUT>; $a=<>; Each example is reads a single line. MMG Skills Lecture Series

Case studies: Rescale, o 2 x and x 2 o Perl basics l Perl

Case studies: Rescale, o 2 x and x 2 o Perl basics l Perl has a number of mechanisms forming a “loop”. – – l l l 14 while foreach (aka for) The while command checks a logical condition and ends to loop when a false is returned. The foreach operates on an array or list. For following script we’ll be using the both case. MMG Skills Lecture Series

Case studies: Rescale #!/usr/bin/perl -w $f=0. 5; while(<>){ split; for(6. . 8){$_[$_]*=$f} print(join” ”,

Case studies: Rescale #!/usr/bin/perl -w $f=0. 5; while(<>){ split; for(6. . 8){$_[$_]*=$f} print(join” ”, @_, ”n”); } 15 MMG Skills Lecture Series

YIKES 16 MMG Skills Lecture Series

YIKES 16 MMG Skills Lecture Series

Case studies: Rescale l 17 I’ve introduced several more ideas here… #!/usr/bin/perl -w $f=0.

Case studies: Rescale l 17 I’ve introduced several more ideas here… #!/usr/bin/perl -w $f=0. 5; while(<>){ split; for(6. . 8){$_[$_]*=$f} print(join” ”, @_, ”n”); } MMG Skills Lecture Series

Case studies: Rescale l I’ve introduced several more ideas here… – 18 Note the

Case studies: Rescale l I’ve introduced several more ideas here… – 18 Note the use of braces and semicolons. #!/usr/bin/perl -w $f=0. 5; while(<>){ split; for(6. . 8){$_[$_]*=$f} print(join” ”, @_, ”n”); } MMG Skills Lecture Series

Case studies: Rescale l I’ve introduced several more ideas here… – – – 19

Case studies: Rescale l I’ve introduced several more ideas here… – – – 19 The computer needs to be told how to interpret the script, and this line tells it two things: Perl lives in /usr/bin The “-w” flag tells it to write out warnings to the shell where the script is being run (this is very important when you’re debugging). #!/usr/bin/perl -w $f=0. 5; while(<>){ split; for(6. . 8){$_[$_]*=$f} print(join” ”, @_, ”n”); } MMG Skills Lecture Series

Case studies: Rescale l I’ve introduced several more ideas here… – – split is

Case studies: Rescale l I’ve introduced several more ideas here… – – split is a perl function that takes a string and “splits” it into its constituent parts. Perl has lots of such useful functions As invoked it looks at “$_” and splits it into the parts separated by “white space”, returning the output to @_; More generally: l 20 #!/usr/bin/perl -w $f=0. 5; while(<>){ split; for(6. . 8){$_[$_]*=$f} print(join” ”, @_, ”n”); } @b=split(“xxxx”, $a) MMG Skills Lecture Series

Case studies: Rescale l I’ve introduced several more ideas here… – – “for(a. .

Case studies: Rescale l I’ve introduced several more ideas here… – – “for(a. . b)” means all integer values between a and b, inclusive. In this case we have 6. . 8 which means 6, 7, 8. Because there is no variable stated, the values are assigned to $_. We could have written #!/usr/bin/perl -w $f=0. 5; while(<>){ split; for(6. . 8){$_[$_]*=$f} print(join” ”, @_, ”n”); } for $x (6. . 8) 21 MMG Skills Lecture Series

Case studies: Rescale l I’ve introduced several more ideas here… – – – 22

Case studies: Rescale l I’ve introduced several more ideas here… – – – 22 For an array, @a, we can ask for a particular value in this way. $a[0] is the first element, $a[1] the second and so on. In this case we have The index of the final element can be determined using the (scalar) $#a. #!/usr/bin/perl -w $f=0. 5; while(<>){ split; for(6. . 8){$_[$_]*=$f} print(join” ”, @_, ”n”); } MMG Skills Lecture Series

Case studies: Rescale l I’ve introduced several more ideas here… – – – 23

Case studies: Rescale l I’ve introduced several more ideas here… – – – 23 This is a shorthand for “multiply yourself by the value in $f. This is not a mathematical expression! It is the same as $_[$_]=($_[$_]*$f) #!/usr/bin/perl -w $f=0. 5; while(<>){ split; for(6. . 8){$_[$_]*=$f} print(join” ”, @_, ”n”); } MMG Skills Lecture Series

Case studies: Rescale l I’ve introduced several more ideas here… – 24 This is

Case studies: Rescale l I’ve introduced several more ideas here… – 24 This is actually a compound of two new functions. #!/usr/bin/perl -w $f=0. 5; while(<>){ split; for(6. . 8){$_[$_]*=$f} print(join” ”, @_, ”n”); } MMG Skills Lecture Series

Case studies: Rescale l I’ve introduced several more ideas here… – – – 25

Case studies: Rescale l I’ve introduced several more ideas here… – – – 25 “join” is a function which joins together the elements of an array and returns a scalar. The first argument is the “glue”, in this case a space. The rest is a list of items, in this case the contents of @_ and this “n” object, which is a “newline” character (there are several special string characters like this all with “”). #!/usr/bin/perl -w $f=0. 5; while(<>){ split; for(6. . 8){$_[$_]*=$f} print(join” ”, @_, ”n”); } MMG Skills Lecture Series

Case studies: Rescale l I’ve introduced several more ideas here… – 26 “print” means

Case studies: Rescale l I’ve introduced several more ideas here… – 26 “print” means pass to the standard output (by default the shell window) #!/usr/bin/perl -w $f=0. 5; while(<>){ split; for(6. . 8){$_[$_]*=$f} print(join” ”, @_, ”n”); } MMG Skills Lecture Series

Case studies: Rescale l This is a pretty basic, and fundamentally flawed way to

Case studies: Rescale l This is a pretty basic, and fundamentally flawed way to do this (although with some care it will work). – – 27 You are assuming that the “standard input” is correctly formatted otb. The scaling factor is “hard-wired”. MMG Skills Lecture Series

Case studies: Rescale #!/usr/bin/perl -w $f=$ARGV[0]; $file=$ARGV[1]; open INPUT, “<$file” or die “I cannot

Case studies: Rescale #!/usr/bin/perl -w $f=$ARGV[0]; $file=$ARGV[1]; open INPUT, “<$file” or die “I cannot open $filen”; while(<INPUT>){ @a=(split); if($#a!=8){die “Wrong number of elements detected on line $. n”} foreach $b (6. . 8){$a[$b]*=$f} print(join” ”, @a, ”n”); } close(INPUT); 28 MMG Skills Lecture Series

Case studies: Rescale #!/usr/bin/perl -w if($#ARGV!=1){die “Wrong number of arguments usedn”} $f=$ARGV[0]; $file=$ARGV[1]; if(-e

Case studies: Rescale #!/usr/bin/perl -w if($#ARGV!=1){die “Wrong number of arguments usedn”} $f=$ARGV[0]; $file=$ARGV[1]; if(-e $file){ open INPUT, “<$file” or die “I cannot open $filen”; }else{ die “file ‘$file’ does not exist”; } while(<INPUT>){ @a=(split); if($#a!=8){die “Wrong number of elements detected on line $. n”} for $b (6. . 8){$a[$b]*=$f} print(join” ”, @a, ”n”); } close(INPUT); 29 MMG Skills Lecture Series

Case studies: Rescale #!/usr/bin/perl -w if($#ARGV!=1){die “Wrong number of arguments usedn”} $f=$ARGV[0]; if($f!~/^(d*. ?

Case studies: Rescale #!/usr/bin/perl -w if($#ARGV!=1){die “Wrong number of arguments usedn”} $f=$ARGV[0]; if($f!~/^(d*. ? d+|d+)$/){ die “scaling factor not a numbern”; } $file=$ARGV[1]; if(-e $file){ open INPUT, “<$file” or die “I cannot open $filen”; }else{ die “file ‘$file’ does not exist”; } while(<INPUT>){ @a=(split); if($#a!=8){die “Wrong number of elements detected on line $. n”} for $b (6. . 8){$a[$b]*=$f} print(join” ”, @a, ”n”); } close(INPUT); 30 MMG Skills Lecture Series

Case studies: o 2 x and x 2 o Audience Participation l What are

Case studies: o 2 x and x 2 o Audience Participation l What are the differences in the operation of these two converters relative to – – l Rescale? each other? Sketch out the script for x 2 o… 1 2 3 4 5 1 1 1 0 39 38 35 27 0 29 33 30 26 0 0 25 0 0 28 0. 00000 6. 5000000000 0. 00000 3. 2500000000 0. 0000000000 6. 500000 3. 250000 56 #Set no. 1 (Final) (Initial) Fe 0. 00000000000 Fe 3. 43965211850 0. 000000 Fe 3. 43965211850 Fe 0. 000000 3. 43965211850 Fe 1. 71982605925 31 0. 00000000000 3. 43965211850 1. 71982605925 MMG Skills Lecture Series

Case studies: linux command filter l We can also open a file-handle on a

Case studies: linux command filter l We can also open a file-handle on a linux/unix command rather than a file: – l This example would execute the list files command “ls –l” and pass the standard output to the file-handle “Q”. – 32 e. g. open Q, “ls -l|” or die Actually there are better, perl function based methods to look at the contents of a directory, but never mind MMG Skills Lecture Series

Case studies: linux command filter l The example wasn’t very obviously useful, but there

Case studies: linux command filter l The example wasn’t very obviously useful, but there are often instances of standard output from a shell command being unhelpful because – – – l l 33 there’s too much and some disappears off the screen the width means that text is wrapped and difficult to interpret there’s lots of irrelevant information and you really only want a small part of it. Sometimes we sort this out with an “alias” in the shell (e. g. “alias ll ‘ls –l’ ”). More complicated manipulation may be performed with the use of perl… MMG Skills Lecture Series

Case studies: linux command filter “q” on Verity l prior name user state submit/start

Case studies: linux command filter “q” on Verity l prior name user state submit/start at queue slots ja-task-ID The standard way to examine job-ID --------------------------------------------------------0. 60500 aim 2. 3. 0 b 2 a 6909644 r 09/21/2008 22: 47: 50 master. q@verity. ncl. ac. uk the contents of the batch queue 15621 15638 0. 50500 aim 2. 3. 0 b 2 a 6912698 r 09/22/2008 16: 41: 22 master. q@verity. ncl. ac. uk 0. 50500 aim 2. 3. 0 b 1 n 1101440 r 09/22/2008 21: 37 master. q@verity. ncl. ac. uk on Verity would be something 15642 15645 0. 50500 aim 2. 3. 0 b 1 n 1101440 r 09/23/2008 01: 44: 29 master. q@verity. ncl. ac. uk 15646 0. 50500 aim 2. 3. 0 b 1 n 1101440 r 09/23/2008 12: 55: 25 master. q@verity. ncl. ac. uk along the lines of “qstat”, the 15650 0. 50500 aim 2. 3. 0 b 2 a 6909644 r 09/24/2008 14: 11: 32 master. q@verity. ncl. ac. uk 15652 0. 60500 aim 2. 3. 0 b 2 a 6909644 r 09/23/2008 12: 33: 01 master. q@verity. ncl. ac. uk output from which is not very 15653 0. 60500 aim 2. 3. 0 b 2 a 6909644 r 09/23/2008 13: 21: 17 master. q@verity. ncl. ac. uk helpful: 15655 0. 50500 aim 2. 3. 0 b 2 a 6912698 r 09/24/2008 14: 12: 07 master. q@verity. ncl. ac. uk 15668 0. 60500 aim 2. 3. 0 b 2 a 6909644 15670 0. 60500 aim 2. 3. 0 b 2 a 6909644 15671 0. 60500 aim 2. 3. 0 b 2 a 6918314 15672 0. 60500 aim 2. 3. 0 b 2 a 6918314 15680 0. 60500 aim 2. 3. 0 b 2 a 6909644 15656 0. 50500 aim 2. 3. 0 b 2 a 6912698 15657 0. 50500 aim 2. 3. 0 b 2 a 6909644 15658 0. 50500 aim 2. 3. 0 b 2 a 6909644 15659 0. 50500 aim 2. 3. 0 b 1 n 1101440 15662 0. 50500 aim 2. 3. 0 b 2 a 6912698 15663 0. 50500 aim 2. 3. 0 b 2 a 6912698 15664 0. 50500 aim 2. 3. 0 b 2 a 6912698 15665 0. 50500 aim 2. 3. 0 b 2 a 6912698 15666 0. 50500 aim 2. 3. 0 b 2 a 6912698 15667 0. 50500 aim 2. 3. 0 b 2 a 6912698 15674 0. 50500 aim 2. 3. 0 b 1 n 8156276 15675 0. 50500 aim 2. 3. 0 b 1 n 8156276 15676 0. 50500 aim 2. 3. 0 b 1 n 8156276 15677 0. 50500 aim 2. 3. 0 b 1 n 8156276 15678 0. 50500 aim 2. 3. 0 b 1 n 8156276 15679 0. 50500 aim 2. 3. 0 b 1 n 8156276 34 r 09/23/2008 20: 50: 21 master. q@verity. ncl. ac. uk r 09/24/2008 04: 33: 40 master. q@verity. ncl. ac. uk qw 09/24/2008 11: 25: 00 3 qw 09/24/2008 11: 47: 04 3 qw 09/24/2008 13: 57: 30 3 qw 09/23/2008 12: 04: 20 2 qw 09/23/2008 12: 22: 55 2 qw 09/23/2008 12: 23: 22 2 qw 09/23/2008 12: 55: 16 2 qw 09/23/2008 14: 16: 17 2 qw 09/23/2008 14: 16: 57 2 qw 09/23/2008 14: 19: 15 2 qw 09/23/2008 15: 02: 51 2 qw 09/23/2008 15: 03: 51 2 qw 09/23/2008 15: 04: 46 2 qw 09/24/2008 12: 21: 53 2 qw 09/24/2008 12: 21: 54 2 qw 09/24/2008 12: 21: 55 2 qw 09/24/2008 12: 21: 57 2 qw 09/24/2008 12: 21: 58 2 qw 09/24/2008 12: 22: 00 2 MMG Skills Lecture Series 3 2 2 2 3 3

Case studies: linux command filter “q” on Verity l 35 The perl script “q”

Case studies: linux command filter “q” on Verity l 35 The perl script “q” takes output from qstat (in various ways) and turns it into something (at least I find) useful: Queue ===== Job. ID ===== 15621 15638 15642 15645 15652 15646 15653 15668 15670 15655 ===== 15671 15672 15680 15656 15657 15658 15659 15662 15663 15664 15665 15666 15667 15674 15675 15676 15677 15678 15679 ===== data: ===== user ===== Abdusalam Mariam Richard Abdusalam Abdusalam Mariam ===== Khaled Abdusalam Mariam Abdusalam Richard Mariam Mariam Rob Rob Rob ====== time ====== 2. 8 d 2. 0 d 1. 8 d 1. 7 d 1. 2 d 20. 5 h 12. 8 h 3. 2 h ====== 5. 9 h 5. 6 h 3. 4 h 1. 2 d 1. 1 d 5. 0 h ====== == #n == 2 1 1 1 2 2 2 1 1 == 2 2 2 1 1 1 1 == == ST == r r r == qw qw qw qw qw == ========================== dir =================================================== ************* *** 24 nodes *** 0 nodes TNA *** 10 nodes dsbld *** -2 nodes free *** 22 nodes queued ************** *** my R nodes: 0 *** my Q nodes: 0 ************** MMG Skills Lecture Series

Case studies: linux command filter “q” on Verity Queue ===== Job. ID ===== 15642

Case studies: linux command filter “q” on Verity Queue ===== Job. ID ===== 15642 15645 15646 ===== 15659 ===== data: ===== user ===== Richard ========= time ====== 1. 8 d 1. 7 d 1. 2 d ====== == #n == 1 1 1 == == ST == r r r == qw == ========================== dir ========================== /scratch/n 1101440/T 3 /scratch/n 1101440/T 4 /scratch/n 1101440/T 5 ========================== /scratch/n 1101440/T 1/T 2_1 ========================== ************* *** 24 nodes *** 0 nodes TNA *** 10 nodes dsbld *** -2 nodes free *** 21 nodes queued ************** *** my R nodes: 3 *** my Q nodes: 1 ************** 36 MMG Skills Lecture Series

Case studies: linux command filter “q” on Verity l l 37 How is this

Case studies: linux command filter “q” on Verity l l 37 How is this done? We will not look at the gory details, but here’s the script… MMG Skills Lecture Series

Case studies: linux command filter “q” on Verity #!/usr/bin/perl # Initialise variables: &Set. Defaults;

Case studies: linux command filter “q” on Verity #!/usr/bin/perl # Initialise variables: &Set. Defaults; #Check the command-line options: &Command. Line; #For each jobid, get the relevant data: &Get. Job. Data; &Get. JOBIDOrder; &Queue. Usage; This is clearly not a very complicated script: in fact, does it do anything at all? #Do some output if($Settings{"Print. Queue"}){&Print. Queue} if($Settings{"QSummary"}){&Print. Queue. Summary} if($Settings{"Node. Summary"}){&Print. Node. Summary} if($Settings{"User. Summary"}){&Print. User. Summary} 38 MMG Skills Lecture Series

Case studies: linux command filter “q” on Verity l Of course it does, but

Case studies: linux command filter “q” on Verity l Of course it does, but the guts are structured in a series of functions or subroutines, which are then called using the syntax such as – l We’ve also now seen a new data structure called a hash. This is a bit like an array, but the index is not necessarily a number: – – l l l 39 $name{first}=“Jonathan”; $name{last}=“Goss”; A hash is denoted by a percent: %name; These structures can be very useful in organising data which is linked in some complex way. You can interrogate a hash using the “keys” function: – l &Do. Something(a list of arguments) “keys %name” returns a list of the elements of the hash, in this case (“first”, “last”). Returning to “q”, we’ll look at one subroutine as an illustration… MMG Skills Lecture Series

Case studies: linux command filter “q” on Verity sub Get. Job. Data{ open IN,

Case studies: linux command filter “q” on Verity sub Get. Job. Data{ open IN, "qstat -r -f -ne |" or die; my $comp=0; while(<IN>){ my @stuff=split; #Get the number of nodes: if($stuff[0] eq "Requested"){ if($jobid){${$data{$jobid}}{"Nodes"}=$stuff[3]-1} else{die "Error - no job id"} }elsif(/parallel. q@comp(. . )/){ $comp=$1; } 40 MMG Skills Lecture Series

Case studies: linux command filter “q” on Verity (it goes on…) elsif(/^s*d+s+/){ $jobid=$stuff[0]; if(${$data{$jobid}}{"user"}){

Case studies: linux command filter “q” on Verity (it goes on…) elsif(/^s*d+s+/){ $jobid=$stuff[0]; if(${$data{$jobid}}{"user"}){ #For the slaves, gather which nodes they're running on: ${$data{$stuff[0]}}{"Comp"}. =" $comp"; $nodes++; #Once for each job: }else{ ${$data{"users"}}{$stuff[3]}++; if($stuff[4]eq"r"||$stuff[4]eq"t"){${$data{$stuff[3]}}{"jobs running"}++} else{${$data{$stuff[3]}}{"jobs queued"}++} ${$data{$jobid}}{"user"}=$stuff[3]; ${$data{$jobid}}{"state"}=$stuff[4]; ${$data{$jobid}}{"time"}=&dt(@stuff[5. . 6]); ${$data{$jobid}}{"CWD"}=""; if( (${$data{$jobid}}{"state"} eq "r") || (${$data{$jobid}}{"state"} eq "t") ){ push @{$data{"Preserve. Qstat. Order. Running"}}, $jobid; }else{ push @{$data{"Preserve. Qstat. Order. Queued"}}, $jobid; } if($Settings{"user"} eq $stuff[3]||$Settings{"All. Users"}){push @{$data{"My. Jobs"}}, $jobid} } close(IN); #For the relevant user, get some additional data: for(@{$data{"My. Jobs"}}){Get. Job. Dir($_)} 41 } MMG Skills Lecture Series

Case studies: linux command filter “q” on Verity l l 42 So what should

Case studies: linux command filter “q” on Verity l l 42 So what should you expect to “take home” from the past two slides? Certainly not the details. You should have a feel for the scope of the use of perl scripts to do useful things… And know that there are plenty of working examples from which you can copy parts and learn how to adapt them for your own purposes… MMG Skills Lecture Series

Case studies: linux command filter “qsub” on Verity l l l 43 Another example

Case studies: linux command filter “qsub” on Verity l l l 43 Another example is the batch queue submission procedure. The submission of a job to the batch queue potentially commits a substantial fraction of our computation resources to do your bidding. An error may result in wasted resources or the job failing for a trivial reason. It is therefore important that the chances of it doing something useful are high Hence the “qsub” script that I use (I do not require you to do the same, but why not? ). MMG Skills Lecture Series

Case studies: linux command filter “qsub” on Verity l Again, the script is written

Case studies: linux command filter “qsub” on Verity l Again, the script is written as a header listing a number of calls to subroutines: #!/usr/bin/perl -w &Defaults; &Command. Line(@ARGV); &Check. Options; if(!$Switches{'nochange'}){ &Parse. DATAfile; &Modify. DATAfile; } if(!$Switches{'tidyonly'}){&Submit} exit; 44 MMG Skills Lecture Series

Case studies: linux command filter “qsub” on Verity: more of a hash 45 ###################################

Case studies: linux command filter “qsub” on Verity: more of a hash 45 ################################### # Set default values: ################################### sub Defaults{ # These three hashes should be the only global parameters: %Settings=( "exec" => "/users/njpg/bin/AIM", "invokation" => $0, "nodes" => 0 ); # %Strings=( parameters=> "", filespace => "filespace{/scratch/njpg/DUMP-FILES}n", rsb => "parameter{use_real_space_build}n!parameter{gvec_cutoff=30}n" ); # %Switches=( 'normalize' => 0, 'override' => 0, 'tidyonly' => 0, 'nochange' => 0, 'rsb' => 0, 'kpar' => 0, 'gga' => 0 ); } MMG Skills Lecture ################################### Series

AIMPRO Filters: FORCES (audience participation, handout) l l 46 This (along with many others)

AIMPRO Filters: FORCES (audience participation, handout) l l 46 This (along with many others) is an example of a script that couples AIMPRO data with a graphics representation. When a structure is being relaxed, AIMPRO outputs information regarding the changes in total energies and the forces on each atom. FORCES digests this data, prepares summary, plottable, temporary data files, and a gnuplot script. Look through the script and identify what you do not understand… MMG Skills Lecture Series

AIMPRO Filters: FORCES – what you see 47 MMG Skills Lecture Series

AIMPRO Filters: FORCES – what you see 47 MMG Skills Lecture Series

Resources l l l Cannibalise existing scripts Books The web: – – – 48

Resources l l l Cannibalise existing scripts Books The web: – – – 48 http: //www. comp. leeds. ac. uk/Perl/ http: //www. tizag. com/perl. T/ http: //www. perl. com/pub/a/2000/10/begperl 1. html MMG Skills Lecture Series

Exercise l The best way I fund to learn programming is not seeing, but

Exercise l The best way I fund to learn programming is not seeing, but doing: – – You should each identify a simple problem of data analysis or production that is a real need in your research. Write the code and see if it works. l – 49 Use existing scripts as source materials. Get together to discuss your scripts and identify better solutions. MMG Skills Lecture Series