Python for Beginners Part 1 Ferdinand JamitzkyLRZ de

Python for Beginners (Part 1) Ferdinand. Jamitzky@LRZ. de

Executable Pseudo-Code for x in range(10): y=2*x if x==0: print("x is zero") elif x>5 and x<10: print("x is between 5 and 10") else: print(f"twice {x} = {y}") “Python is executable pseudocode. Perl is executable line noise. ” (– Old Klingon Proverb)

Zen of python (20. 2. 1991 -? ) ● ● ● Beautiful is better than ugly Explicit is better than implicit Simple is better than complex Complex is better than complicated Readability counts “There should be one (and only one) obvious way to do it“

Python in a nutshell

python as seen from the orbit External Libraries Basic Language and Syntax Python 2 Python Cython 3 Num. Py Dask Pandas Sci. Py Python Builtin Libraries Strings, IO Web, XML Extensions os, zip Spark Blender Jupyter Tensor Flow

Python Syntax ● basic syntax - import, for, if, while, functions, comprehensions ● advanced syntax - classes, try, except, raise, global, nonlocal ● expert syntax - generators, yield, with, lambda, coroutines, async ● builtin data types - int, double, complex, strings - lists, tuples, arrays, sets - dictionaries

Keywords (more than 90% code) import lib as name from lib import n as n [expr for it in list if cond] while condition: else: if condition: else: def function: if_true if cond else if_false ”””doc string””” return value for iterator in list: pass class name: break def __init__(self): continue def method(self):

Keywords (less than 10% code) raise name lambda var: expression try: except name: finally: @decorator def fun with expression as var: global variable nonlocal variable async def fun -> ann: assert condition yield value yield from generator await expression

basic rules of the game ● ● ● indentation matters! # denotes comments lists start from 0 ● ● file type matters (*. py)! directory hierarchy matters! for x in range(10): y = 2*x+1 # here comes the output: print(f"y = {y}") print("finished loop")

basic rules of the game Modules can be defined either by filename or by folder name. $ python # module by filename >>> import myfile # module by folder name >>> import mymod # call: >>> myfile. myfunc() hello >>> mymod. myfunc() world $ ls myfile. py mymod/__init__. py myfile. py: def myfunc(): print(“hello”) mymod/__init__. py: def myfunc(): print(“world”)

Variable names ● Variable names can consist of: Länge = [10, 20, 30, 40, 50] #Berechne Mittelwert ΣLänge = 0 for L in Länge: ● Variables have to start with Alphabetic or Underscore ΣLänge = ΣLänge + L e. g. this is a valid vaiable name: µ = ΣLänge / len(Länge) _sumÖfÄll_µラ 0 print(f"Mittelwert = {µ}") #this is valid: ● Best practices: Greek letters can be helpful for math notation. ラーメン = "ramen" However, for readability and π = 3. 14159 portability you better stick to pure ASCII. jalapeño = "a hot pepper" - Alphabetic (also Greek or Umlauts) - Numbers - Underscore _

types, lists, tuples and dicts ● Python has the following number types: - int, long, float, complex ● Strings - Normal Strings: "this", 'this' Multiline Strings: """this""", '''this''' Unicoda and Byte Strings: u'this', b'this' Format Strings: f'x={x}', f'{x=. 3 f}' ● Lists and tuples - a = [1, 2, 3] is a list, - b = (1, 2, 3) is a tuple (immutable) ● Dictionaries aka Associative Arrays - a = { ’one’: 1, ’two’: ”zwei”} is a dict

Package Managers

package managers python has a plentitude of package managers and package formats (contradicts zen of python), so don’t get confused ● ● easy_install, virtualenv (dead) pip (alive, default package manager for python) conda (state of the art) Data formats: - wheel (official package format PEP 427) - egg (old package format)

pip ● ● $ $ simple packages management tool for python comes preinstalled with python complementary to conda packages are called *. whl (wheel) pip pip install Some. Package==1. 0. 4 'Some. Package>=1. 0. 4' --upgrade Some. Package # # latest version specific version minimum version upgrade

conda ● ● ● conda is a package manager in user space. tool to create isolated python installations it allows you to use multiple versions of python substitutes virtualenv (dead since 2016) multiple versions - miniconda (free) - anaconda (commercial) - intel (numpy using MKL, intel. mpi ● works on linux, MS-win, mac. OS ● packages are provided by channels (anaconda, conda-forge, bioconda, intel, . . . )

Anaconda or Miniconda? Choose Anaconda if you: ● Are new to conda or Python. ● Like the convenience of having Python and over 150 scientific packages automatically installed at once. ● Have the time and disk space---a few minutes and 300 MB. ● Do not want to individually install each of the packages you want to use. Choose Miniconda if you: ● Do not mind installing each of the packages you want to use individually. ● Do not have time or disk space to install over 150 packages at once. ● Want fast access to Python and the conda commands and you wish to sort out the other programs later.

conda $ $ $ $ conda conda create –n my_env python=3. 6 create –p <mydir> --clone root --offline install –c conda-forge scipy=0. 15. 0 list search numpy update –all info numpy

Python @ LRZ Several interpreters are available: • OS python /usr/bin/python -> don' t use! • module load python(2. 7_intel) -> old • python(3. 5_intel) -> OK! • python(2. 7_anaconda_mpi) -> brings own mpi • python(3. 5_anaconda_nompi) -> no mpi at all • download miniconda and install your own

Best practice: Python @ LRZ ● On each node there is a system python installed. Don't use it! ● Use the module system: $ module avail python ---------------- /lrz/sys/share/modules/files/tools -------python/2. 7_anaconda_nompi python/2. 7_intel(default) python/3. 5_intel python/3. 6_intel Version Source MPI Remark 2. 7 anaconda nompi deprecated 2. 7 intel Intel MPI legacy 3. 5 intel Intel MPI deprecated 3. 6 intel Intel MPI current

Generate your own python LRZ environment ● LRZ uses the conda package manager for python libraries. In the default module only a minimal set of libraries is provided. $ module load python/3. 6_intel $ conda create –n py 36 python=3. 6 $ source activate py 36 or $ conda create –p $HOME/conda/py 36 --clone root –-offline # offline also works on Super. MUC-NG $ source activate $HOME/conda/py 36 then $ conda install scipy=0. 15 $ conda list

miniconda and conda clone ● ● Sometimes conda wants to change files in /lrz/sys which is not allowed for ordinary users use miniconda: - free minimal installer for conda - small, bootstrap version of Anaconda - includes only conda, Python and a small number of other useful packages - use the conda install command to install 720+ additional conda packages from the Anaconda repository. or ● use conda clone - clone an existing conda installation $ conda create –p <mydir> --clone root --offline

Super. MUC-NG: Method 1 Using conda pack Step 1: On the machine with internet connection (local linux workstation or Super. MUCCloud instance) ● Install conda in a VM on the Super. MUC-Cloud ● Install the packages that you need into a new environment ● install the conda-package $ conda install conda-pack ● pack environment my_env into my_env. tar. gz $ conda pack -n my_env ● then copy the tar file to Super. MUC-NG Step 2: On Super. MUC-NG (no internet connection to outside) ● Unpack environment into directory my_env $ mkdir -p my_env $ tar -xzf my_env. tar. gz -C my_en ● Activate the environment: $ source my_env/bin/activate

Super. MUC-NG: Method 2 Via local HTTP proxy and reverse SSH tunnels: Step 1: create a proxy server local> pip install --upgrade --user proxy. py local>. local/bin/proxy. py --port 8899 Step 2: reverse SSH tunnel local> ssh -R 8899: localhost: 8899 skx. supermuc. lrz. de Step 3: create local environment sm-NG> module load python/3. 6_intel sm-NG> conda create -p $HOME/mypython_3. 6 --clone root –offline Step 4: edit. condarc on Super. MUC-NG to provide the proxy server proxy_servers: http: //127. 0. 0. 1: 8899 https: http: //127. 0. 0. 1: 8899 ssl_verify: False

Syntax

import The import statement, which is used to import modules whose functions or variables can be used in the current program. There are four ways of using import: >>> >>> import numpy from numpy import * import numpy as np from numpy import pi as Pie

While loop err=0. 1 n=0 while err>1. e 6: x=iter(x) err=error(x) n+=1 if n>1000: break elif err>0. 1: n=0 continue else: print("loop finished")

for loops for i in list: do_something_with(i) if cond 1(i): break elif cond 2(i): continue else: print("Loop correctly finished")

loopless computing (advanced) # prerequisites: def iterate(func, tol=1. e-5, x=1): while True: yield x # suspend execution y = func(x) if abs(y-x) < tol: return y # finalize function x = y # here comes the program: [x for x in iterate(func = lambda x: (x+9/x)/2, x=10)] [10, 5. 45, 3. 5506880733944954, 3. 042704026361998, 3. 0002996732267952, 3. 0000000149658455]

file i/o ● text files dd=open(”data. txt”). readlines() ● print lines [x[: -1] for x in open(”data. txt”, ”r”). readlines()] ● pretty print from pprint import pprint(dd) ● binary files xx=open(”data. bin”, ”rb”). read() xx. __class__

interaction with the shell make script executable: $ chmod u+x myscript. py: #!/usr/bin/python #!/usr/bin/env python 2. 7 import sys print "The name of the script: ", sys. argv[0] print "Number of arguments: ", len(sys. argv) print "The arguments are: " , str(sys. argv) in larger scripts use the argparse library

Data Structures

Data structures ● Numbers are mapped to maximal system types: byte, int (unlimited length), float (=double), complex ● Lists of data strutures. Each one of them is numbered, starting from zero. You can remove values from the list, and add new values to the end. ● Strings are just like lists, but you can't change their values. ● Tuples are immutable lists. The values that you give it first up, are the values that you are stuck with for the rest of the program. ● Dictionaries aka associative arrays aka key-value stores

number types ● Python has the following number types: - int, long, float, complex >>> x=0 >>> x=123456789012345 >>> x**2 152415787532388367504953347995733866912056239 9025

basic types: long vs float >>> x=123456789012345 >>> float(x)**12 1. 2536598767934103 e+289 >>> float(x**12) 1. 2536598767934098 e+289 >>> x**12 125365987679340988385155987957344620719772763 435558412643918634708860008684622476289189408 122904124025079348898207042504644463778641104 140990841878266383680568044115362044043884095 444413842891790950870476081757908423384415448 872287884941281209197912958987211967647326426 09051396426025390625

basic types: complex Imaginary and complex numbers are built in: >>> 1 j**2 #imaginary unit (-1+0 j) >>>(1+1 j)**4 #4 th root of -4 (-4+0 j) >>> 1 j**1 j # i to the i (0. 20787957635076193+0 j) >>> import cmath >>> cmath. log(-1) 3. 141592653589793 j # pi

Lists >>> >>> >>> 2 >>> [2, x=[1, 2, 3] x. append(“one”) y=x y[0]=2 x[0] # list methods # list name is a pointer # index starts at 0 x. append(x) # append pointer to list x 2, 3, 'one', [. . . ]]

Strings python 3 has Unicode strings (vs python 2 byte strings) - "this", 'this' """this""", '''this''' u'thiß' b'this' f'this' - string interpolation (masks): >>> x=1 >>> f"x = {x}" >>> f"1 + 1 = {1+1}" #single line #multi line #explicite unicode #byte array #format string -> "x = 1" -> "1 + 1 = 2"

Indexing strings and lists >>> a="1234" a[0] -> 1 a[0: ] -> 1234 a[0: -1] -> 123 a[0: : 2] -> 13 a[: : -1] -> 4321 a[-1: : -2]-> 42 >>> b=[1, 2, 3, 4] b[0] -> 1

String methods ● split strings >>> dd="a b c d" >>> dd. split() ']a', 'b', 'c', 'd[' ● join strings >>> " ". join(']a', 'b', 'c', 'd']) ● combine both >>> " ". join([ "<"+x"/>" for x in dd. split()]) >'a/> <b/> <c/> <d'</

lists and tuples are immutable lists >>> a=(1, 2, 3) >>> a[1]=3 -> error reason for tuples: faster access

list comprehensions ● a list is defined by square brackets ● a list comprehension uses square brackets and for >>> x=[1, 2, 3, 4, 5] >>> y=[ i for i in x] >>> “ ”. join([s. split(“n”) for s in open(“file. txt”). readlines()]) >>> import random. uniform as r >>> np=1000000 >>> sum([(r(0, 1)**2+r(0, 1)**2 < 1) for i in range(np)])/np*4. 3. 141244

dicts dictionaries aka associative arrays aka key/value stores >>> a={‘one’: 1, ‘two’: 2. 0, ‘three’: [3, 3, 3]} dictionary comprehensions: >>> {i: i**2 for i in range(4)} {0: 0, 1: 1, 2: 4, 3: 9} >>> a. keys() >>> a. values()

for loops with dicts you can loop over a dict by: >>> knights = {'gallahad': 'the pure', 'robin': 'the brave'} >>> for k, v in knights. items(): . . . print(k, v) or >>> {k+” ”+v for k, v in knights. items()} >>> [k+” ”+v for k, v in knights. items()]

arrays are lists with the same type of elements there exists a special library for numeric arrays (numpy) which never made it into the official distribution. they serve as an interface to c-code. If you need numerical arrays use the numpy library (see below)

sets are unordered lists. They provide all the methods from set theory like intersection and union. Elements are unique. >>> {1, >>> >>> x=set((1, 2, 3, 4, 1, 2, 3, 4)) x 2, 3, 4} x & y x | y x-y x ^ y

python 2 vs python 3 ● why python 3? - you need Unicode? - you want to use generators (yield) and other new features? ● why python 2? - many lists are iterators in py 3 (range, filter, zip, map, . . . ) - many old packages do not (yet) have a python 3 version python 2 is now officially deprecated and no longer maintained! ● use 2 to 3 converter (or 3 to 2 for backwards) $ 2 to 3 myprog. py 19. 12. 2021 Leibniz-Rechenzentrum 47

Functions

functions ● keywords ● doc strings ● specials: - 19. 12. 2021 lambda functions yield, yield from, generators annotations, decorators async, await, . . . Leibniz-Rechenzentrum 49

functions: keywords def myfun(a, b=1, c=[1, 2], *args): “decription goes here” return a, b, c, args >>> myfun(0) (0, 1, [1, 2], ()) >>> myfun(0, c=2) (0, 1, 2, ()) >>> myfun(0, 1, 2, 3, 4) (0, 1, 2, (3, 4))

functions: lambda functions Lambda functions are unnamed functions which can be defined on the fly f 1 = lambda x: x+1 def f 2(x): return x+1 f = lambda *x: x >>> f(”one”, 2, [])

Putting it all together Compute prime numbers up to 30 def isprime(n): return n not in [x*y for x in range(n) for y in range(n)] print([n for n in range(2, 30) if isprime(n)])

Classes

Object oriented programming in a nutshell ● Everything in python is an Object (numbers too) ● Objects: instances of classes ● Classes: blueprints for objects ● Methods: functions attached to objects ● Classes can inherit "blueprints" from other classes >>> a=[] >>> type(a) >>> print a. __class__ >>> print dir(a) >>> a. __doc__

class definition class point 2 d(object): def __init__(self, x=0, y=0): self. x = x self. y = y def move(self, dx=0, dy=0): self. x += dx self. y += dy return self def __repr__(self): return f"Point at x={self. x}, y={self. y}" 19. 12. 2021 Leibniz-Rechenzentrum 55

inheriting from class object ● Python 2 had 2 styles of classes - "classic" style classes - "new" style classes ● Python 3 only has "new" style classes ● In Python 2: always inherit from object explicitly. ● In Python 3: inherit from object if you are writing code that tries to be Python agnostic, that is, it needs to work both in Python 2 and in Python 3.

class usage >>> p 0=point 2 d() >>> p 1=point 2 d(x=1) >>> p 2=point 2 d(3, 4) >>> p 0. move(1, 2) Point at x=1, y=2 >>> p 3 = p 1. move(dx=2). move(dy=3) >>> print(p 3) Point at x=3, y=3 19. 12. 2021 Leibniz-Rechenzentrum 57

magic methods ● function names with leading and trailing underscores are special in python ("magic methods") >>> str(a) is translated to: >>> a. __str__() and >>> a+b >>> a. __add__(b) >>> f(x) >>> f. __call__(x)

Subclassing # library. py # user. py class Base: def bar(self): return "bar" from library import Foo class Derived(Base): def baz(self): return 'baz'+self. bar() Derived. bar is inherited from Base class

data classes (python 3. 7) from dataclasses import dataclass @dataclass Data. Class. Card: rank: str suit: str = 'Spades' >>> queen_of_hearts = Data. Class. Card('Q', 'Hearts') >>> queen_of_hearts. rank 'Q'

Advanced Topics

Advanced topics ● ● ● try-except decorators with yield async

try except using try you can catch an exception that would normally stop the program x=range(10) y=[0]*10 for i in range(10): try: y[i]=1. /x[i] except: y[i]=0.

@decorators are syntactic sugar for applying a function and overwriting it. @mydecorator def myfunc(): pass is the same as def myfunc(): pass myfunc = mydecorator(myfunc)

decorators aka macros @mymacro def ff(y): return y*2 def mymacro(func): return lambda *x: ”Hey! ”+str(func(*x)) def mymacro(somefunc) def tempfunc(*x): return ”Hey! ”+str(somefunc(*x)) return tempfunc

with statement motivation The with statement allows for different contexts with EXPR as VAR: BLOCK roughly translates into this: VAR = EXPR VAR. __enter__() try: BLOCK finally: VAR. __exit__() VAR must be an instance of a class (aka context manager) that implements the methods __enter__ and __exit__

with statement examples You need a context manager (has enter and exit methods) Examples: ● opening and automatically closing a file ● html requests ● database transactions ● temporary option settings ● Thread. Pool. Executor ● Open. MP directives ● log file on/off ● cd to a different folder and back ● set debug verbose level ● tensorflow ● change the output format or output destination with open("/etc/passwd") as f: df=f. readlines() from urllib. request import urlopen with closing(urlopen('http: //www. lrz. de')) as page: [print (line) for line in page] with Thread. Pool. Executor(workers=1) as ex: future=ex. submit(pow, 323, 1235) print(future. result()) A=pymp. shared. array((100, ), dtype='uint 8') with pymp. Parallel(4) as p: for index in p. range(0, 100): A[index] = 1 z= tf. multiply(x, y) with tf. Session() as sess: out = sess. run(z) print(out) with redirect_stdout(sys. stderr): help(pow)

generators ● range(10000) would generate a list of 10000 number although they would later on not be needed. ● generators to the rescue!! ● only generate what you really need ● new keyword: yield (instead of return) <<<def create. Generator: (). . . yield “one“. . . yield 2. . . yield [3, 4]. . . >>> a=create. Generator() >>> next(a) “one“

Generator comprehensions ● like list comprehensions, but computed only when needed >>> a=(i**4 for i in range(8)) >>> next(a) 0 >>> next(a) 1 >>> list(a) [16, 81]

Generator classes class Gen: def __init__(self): self. i=0 def _iter__(self): return self def __next__(self): self. i+= 1 sleep(1) return self. i A generator class must implement the methods __next__ and __iter__

new in python 3. 7: asyncio and await import asyncio async def main(sec): print('Hello. . . '+str(sec)) await asyncio. sleep(sec) print(str(sec)+'. . . World!') # Jupyter Python 3. 7+ loop = asyncio. get_event_loop() loop. create_task(main(10))

new in python 3. 8: await in the interpreter $ python 3 -m asyncio REPL 3. 8. 0 Use "await" directly instead of "asyncio. run()". Type "help", "copyright", "credits" or "license" for more information. >>> import asyncio >>> await asyncio. sleep(10, result='hello') hello

new in python 3. 8: Assignment expressions ● There is new syntax : = that assigns values to variables as part of a larger expression. ● It is affectionately known as “the walrus operator” due to its resemblance to the eyes and tusks of a walrus. (: =) Usage: x: =10 Assigns 10 to x and returns 10 at the same time e. g. y=(x: =10)

new in python 3. 8: Assignment expressions Examples: # initialization during computation if (n : = len(a)) > 10: print(f"List too long ({n} elements, >10") # cumulative sums of list a [cs: =a[0]] + [cs: =cs+a[i] for i in range(1, len(a))] # 10 fibonacci numbers [x: =[1, 1]] + [x : = [x[1], sum(x)] for i in range(10)] Try to limit use of the walrus operator to clean cases that reduce complexity and improve readability.