Windows System Programming using Python Mark Hammond mhammondskippinet

Windows System Programming using Python Mark Hammond mhammond@skippinet. com. au OReilly Open Source Python Conference August 1999, Monterey, CA Aug 24, 1999 Windows System Programming using Python

About this Presentation l Most content taken directly from upcoming book for O’Reilly Python Programming on Win 32 By Mark Hammond and Andy Robinson l http: //www. ora. com/catalog/pythonwin 32/ O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 2

Who this talk is for? l Existing Python programmers – Even those without any Windows experience should follow this without problem. l Although there is not enough time to explain the relevant Windows APIs l Existing Windows Programmers – Even without Python experience, you should immediately see the similarities between your existing language. O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 3

Python and Windows l Each Python version comes with an installer package for Windows. l Standard Python port contains all crossplatform Python features, but very few Windows specific features. l Python for Windows extensions contains many useful Windows extensions for Python. l http: //www. python. org/windows O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 4

Python Windows Extensions l Includes: – Pythonwin: MFC based GUI environment and IDE/Debugger – win 32 com: Interfaces Python and COM – win 32 Extensions: Interfaces to native Win 32 API. l Official releases can be found at http: //www. python. org/windows l Extensions home is at http: //starship. python. net/crew/mhammond O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 5

System Level Programming? l For this talk, we define system level programming as working with low-level features of Windows Files, Pipes, Processes, Threads, Services, Event Log and so forth. l Python and similar languages really not suitable for device-driver type development, and other more system-like Systems Programming! O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 6

Why not just use Python? l Python has excellent native support for files, processes, threads etc. l These features are typically limited to those defined by ANSI C. – Many advanced Windows features are not exposed using these interfaces. – Standard implementation of some of the standard library functions leaves a little to be desired in a Windows environment. O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 7

Portable Process Control (1 of 3) l Standard Python functions all work – Just often not quite how we would like! l os. system() import os os. system(“notepad C: \autoexec. bat”) Problems – Creates a new console window when run from a GUI. – Waits for process to terminate. O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 8

Portable Process Control (2 of 3) l os. execv family – Doesn’t search system path, and doesn’t parse command lines os. execv("c: \Winnt\notepad. exe", ("c: \autoexec. bat", ) ) – Does clobber your existing process - the call to os. execv() never returns! O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 9

Portable Process Control (3 of 3) l os. popen() >>> file = os. popen("echo Hello") >>> file. read() 'Hello12' – Works fine from Windows NT console programs, but fails miserably from a GUI! – win 32 pipe module in the Win 32 extensions provides a working replacement. O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 10

Better Process Control (1 of ) l win 32 api module provides some highlevel, Windows specific functions. l win 32 api. Win. Exec() – Very similar to os. system(), but overcomes limitations. – >>> import win 32 api – >>> win 32 api. Win. Exec("notepad") – Optional parameter allows you to specify the Window’s initial state (eg, minimized) O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 11

Better Process Control (2 of 2) l win 32 api. Shell. Execute() – Typically opens “documents” - eg, execute “foo. doc”, and (typically) Word will open. – Finer control over the new process. – Can also execute arbitrary executables - not limited to documents. – For example, to print a specific document: win 32 api. Shell. Execute(0, "print", "My. Document. doc", None, "", 1) O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 12

Ultimate Process Control (1 of 2) l win 32 process module exposes the low level Win 32 API. l Full support for Create. Process, Create. Process. As. User, Create. Thread etc. l Full support for Windows Handles – Files can be passed as stdin/out/err – Process and thread handles are waitable using the win 32 event module. O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 13

Ultimate Process Control (2 of 2) l Able to set thread and process priority and affinity levels – Very handy for bugs that only appear in multiprocessor machines. l Able to do all this for both existing and new processes. l Process usage samples included in distribution. O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 14

Introduction to our Sample l Full sample Windows NT Service – Provides event log and performance monitor information. Clients connect using Named Pipes – Less than 75 lines of code for all this functionality. – Still too big to present in one hit l Selected excerpts included in slides l Full code on CD, and at http: //starship. python. net/crew/mhammond/conferences/ ora 99 O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 15

Portable Files and Pipes l Python has excellent built-in file support – Inherits platform stdio support - wont bother discussing them here. l Native Windows files only useful for highly -advanced features, such as: – Overlapped IO – IO Completion Ports – Named Pipes – NT Security requirements O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 16

Native Files (1 of 4) l win 32 file. Create. File() used for most file operations – Create and open regular files, memory mapped files, etc. – Takes seven parameters - c. f. open()’s two! – Returns a Py. HANDLE object l Can be passed to any Python API wrapper expecting a handle. l Auto-closed when last reference removed. O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 17

Native Files (2 of 4) l Overlapped IO for asynchronous operations – File opened for overlapped IO requires a Windows event object. – All IO operations return immediately. – Event object signalled when IO complete l Great for high-performance servers – Simple to support multiple concurrent file operations per thread. O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 18

Native Files (3 of 4) l NT Completion ports for even better asynchronous control – Windows manages associating the completed IO operation with a connection – More complex to use. – Often requires a state-machine implementation. – Offers excellent performance - Microsoft’s recommended architecture for scalable, highperformance servers. O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 19

Native Files (4 of 4) l Full support for NT Security – Default security can be used by passing None – Many real-world applications require explicit security configuration l Full support for Windows Impersonation – Processes can automatically impersonate the remote pipe client. – Impersonate any user given their password. O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 20

File Sample (1 of 2) l Overlapped IO is used self. overlapped = pywintypes. OVERLAPPED() # create the event to be used. self. overlapped. h. Event = win 32 event. Create. Event(None, 0, 0, None) l Special security for pipes is needed for services sa = win 32 security. SECURITY_ATTRIBUTES() # Allow full access! sa. Set. Security. Descriptor. Dacl ( 1, None, 0 ) O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 21

File Sample (2 of 2) l Create pipe and connect to client pipe. Handle = win 32 pipe. Create. Named. Pipe(pipe. Name, open. Mode, pipe. Mode, win 32 pipe. PIPE_UNLIMITED_INSTANCES, 0, 0, 6000, # 6 second timeout. sa). . . hr = win 32 pipe. Connect. Named. Pipe(pipe. Handle, self. overlapped) O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 22

Windows NT Services l Similar concept to a Unix daemon l A few special requirements – Must respond to asynchronous commands from NT to (e. g. ) Shutdown – Must be capable of reporting status to NT l NT has built-in UI for configuring and controlling services. O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 23

Python controlling Services l Full exposure of the Windows NT Service Control Manager API – Start, Stop, Pause Services, Install or Remove services, etc l win 32 serviceutil module makes it simple >>> win 32 serviceutil. Stop. Service("Messenger") (32, 3, 0, 0, 0, 6, 20000) >>> win 32 serviceutil. Start. Service(. . . ) >>> O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 24

Implementing Services l Simple to implement Services in Python – Smallest service is around 16 lines of code! l Includes debug support, self-installation, and fully controllable from Windows NT l Few more lines needed to make it something useful : -) l Simply sub-class Service base-class, and implement control features – Minimum required is Stop. Service – Trivial to implement most controls O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 25

Windows NT Event Log l Central place for programs to log information. – Ideal for Services - can not present effective GUIs l Benefits to programmer – Built in transaction and thread safety, maximum size ability, etc. l Benefits to Administrator – Central log of messages, and third party tools to help analysis. O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 26

Python reading the Event Log l Complex native API from win 32 evtlog l Simpler interface from win 32 evtlogutil – Define Feeder function def Dump. Record(record): print ”Got event ID”, record. Event. ID – And feed it! win 32 evtlogutil. Feed. Event. Log. Records( Dump. Record) Got Event ID -2147483645 O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 27

Python writing the Event Log l More complex than reading - event sources must be registered. l win 32 evtlog and win 32 evtlogutil used here too l Python Service framework also supports simple event log writing O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 28

Writing the Event Log Sample l Our sample uses the Service Framework functions which makes it trivial l Events are logged by ID, rather than explicit text. – Our sample uses a built-in ID to log a “service starting” message import servicemanager. Log. Msg( servicemanager. EVENTLOG_INFORMATION_TYPE, servicemanager. PYS_SERVICE_STARTED, (self. _svc_name_, '')) O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 29

NT Performance Monitor l Built-in NT tool for monitoring an applications performance. l Application must be written to provide this data. l API designed for smallest impact on program supplying data – Moves the burden to the collecting application. – Not trivial to work with O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 30

Python reading from Perf. Mon l Useful for overcoming limitations in built- in tool – For example, sample the data hourly and log to a longer-term database. – Useful for reading standard information about another process. l Process ID, Memory usage, etc. l win 32 pdhutil module for reading data – Good sample code in the source file O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 31

Python supplying Perf. Mon l Complex installation and setup procedure. – C. h file and custom. ini file necessary at application installation – Supported by the Python Service framework l Quite trivial to use once running – Simply increment a counter - Performance Monitor handles conversions to required units. O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 32

Perf. Mon Sample l Installation more than we can cover here – See Init. Perf. Mon() method l Working with counters trivial – Increment our connections counter each connection self. counter. Connections. Increment() l Perfmon manages converting to connections-per-second automatically. O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 33

Our Sample in Detail (1 of 8) l Pipe. Service 2. py – The service implementation. l Pipe. Service 2_install. h Pipe. Service 2_install. ini – Required for performance monitor installation – Most services don’t need this! l Pipe. Service. Client. py – Sample client to connect to our service. O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 34

Our Sample in Detail (2 of 8) l Service functionality in Pipe. Service class – Base class handles most of the grunt – We simply supply service name and other optional attributes class Pipe. Service( win 32 serviceutil. Service. Framework): _svc_name_ = "Python. Pipe. Service" _svc_display_name_ = "A sample Python. . . " O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 35

Our Sample in Detail (3 of 8) l We respond to an NT Service. Stop request by telling NT we are stopping, and setting a Windows Event def Svc. Stop(self): # Before we do anything, tell the # SCM we are starting the stop process. self. Report. Service. Status( win 32 service. SERVICE_STOP_PENDING) # And set my event. win 32 event. Set. Event(self. h. Wait. Stop) O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 36

Our Sample in Detail (4 of 8) l Overlapped IO means we can wait for either a connection, or our Stop request win 32 pipe. Connect. Named. Pipe(pipe. Handle, self. overlapped). . . # Wait for either a connection, or # a service stop request. wait. Handles = self. h. Wait. Stop, self. overlapped. h. Event rc = win 32 event. Wait. For. Multiple. Objects( wait. Handles, 0, timeout) O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 37

Our Sample in Detail (5 of 8) l Install our service simply by executing the service script – If not for Performance Monitor, the command-line would be simple: C: Scripts> Pipe. Service 2. py install – But Perf. Mon needs an extra arg C: Scripts> Pipe. Service 2. py --perfmonini=Pipe. Service 2_install. ini install Installing service Python. Pipe. Service to. . . Service installed O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 38

Our Sample in Detail (6 of 8) l Once installed, we can start the service – Start from Control Panel, or using the script itself C: Scripts> python. exe Pipe. Service 2. py start l And start a client test session C: Scripts> python. exe Pipe. Service. Client. py Hi there The service sent back: You sent me: Hi there O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 39

Our Sample in Detail (7 of 8) l We will have Event Log records. . . O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 40

Our Sample in Detail (8 of 8) l And Performance Monitor Data. O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 41

Summary l Python library flexible, rich and portable enough for many tasks l Low-level Windows programming achieved by using extensions that expose the raw Windows API l This talk should have shown: – An overview of how this programming is done in Python. – How simple it is to do complex Windows tasks. O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 42

More Information l Python and Python Documentation – http: //www. python. org/doc l Python for Windows Extensions – http: //starship. python. net/crew/mhammond – http: //www. python. org/windows – Reference Manuals and Samples included with the distributions O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 43

Thanks for coming l Mark Hammond – mhammond@skippinet. com. au – http: //starship. python. net/crew/mhammond O’Reilly Python Conference Aug 24, 1999 Windows System Programming using Python 44