Functional Programming and XQuery M D Metadata Solutions

Functional Programming and XQuery M D Metadata Solutions Dan Mc. Creary President Dan Mc. Creary & Associates dan@danmccreary. co m (952) 931 -9198

Outline • Historical background • Imperative vs. Functional programming • How Functional programming is different from Imperative programming • Strategies for thinking in functional terms M D Copyright 2007 Dan Mc. Creary & Associates 2 of N

Two Approaches to Computation 1940 s… John von Neumann Alonzo Church Manage state with a program counter. Make functions act like math functions. Which is simpler? Which will scale to 10, 000 CPUs? M D 3 Copyright 2010 Dan Mc. Creary & Associates

Imperative vs. Functional Imperative Programming Functional Programming Focus on a discrete series of sequential steps that update the state of a system Focus on independent functions that transform data with no regard to state management M D Copyright 2010 Dan Mc. Creary & Associates 4 of N

Different Approaches Imperative Programming Functional Programming 1. Parameters are data, not functions 2. Variables are changeable 3. Use of state to manage calculations 4. Any object can change the state of any other object at any time 5. Use of sequential steps 1. Functions are used as parameters to functions 2. Variables are "immutable" 3. Outputs only depend on the inputs 4. No "side effects" 5. Use of recursion M D Copyright 2010 Dan Mc. Creary & Associates 5 of N

Different Thinking Sequential Processing • The output of any step can be used in the next step Parallel Processing • Each loop of a FLOWR statement is an independent thread M D Copyright 2010 Dan Mc. Creary & Associates 6 of N

No Crosstalk Between Threads • No cross-thread communication is possible • No intermediate results of any thread can be used in any other thread • No "side-effects" M D Copyright 2010 Dan Mc. Creary & Associates 7 of N

No Execution Order Guaranteed Start Order 3 1 2 Execution Time 5 sec 2 sec 3 sec • Each thread may run on a different CPU • Threads may each need input that may be ready at a different time • Threads may take different amounts of time to complete M D Copyright 2010 Dan Mc. Creary & Associates 8 of N

Referential Transparency • Any expression can be replaced with its value without changing the program behavior • Promotes the ability to – Verify (testability) – Optimize (performance) – Parallelize • Easier to write tools to automate testing of functional systems See http: //en. wikipedia. org/wiki/Referential_transparency_(computer_sc M D Copyright 2010 Dan Mc. Creary & Associates 9 of N

current-date() • The current-date() function is not transparent because it depends on the state of a clock • If you run it tomorrow it will return a different value M D Copyright 2010 Dan Mc. Creary & Associates 10 of N

Program Counter Orientation data program • Program counter steps through instructions Program • Branch instructions State change the address of the Program Counter Program • Any program could Counter update any data • Arrays started with item "0" which was a memory offset M D Copyright 2010 Dan Mc. Creary & Associates 11 of N

All Computation Is Transformation ('a', 'b', 'c') Lists and Items ('A', 'B', 'C') transformati on Other Lists And Items • Functional programming was based on a theory called "lambda calculus" • It postulated that all computing was a form of data transformation • Lists were "logical" and the first item in a list was called "1" http: //en. wikipedia. org/wiki/Lambda_calculus M D Copyright 2010 Dan Mc. Creary & Associates 12 of N

Leaky Pipes vs. Solid Pipes Procedural Programming Data In Functional Programming Data In Data leakage Side Effects Data leakage Solid Steel Sides Data leakage M D Data Out Copyright 13 Data Out 2010 Dan Mc. Creary & Associates

Functional Programming y = f(x) • Computer programs are like mathematical functions • Developers do not manipulate states and variables (things that change value), but focus entirely on transformations of the input data • Functions that take other functions as input parameters • Makes it very easy to build modular programs • Software written in FP languages tend to be very concise and easy to port to parallel systems • Functions are treated as first class citizens http: //en. wikibooks. org/wiki/Computer_programming/Functional_programming M D Copyright 2008 Dan Mc. Creary & Associates 14

Idempotence Transactions yn = f(xn) • Transactions that they can be applied multiple times without changing the result. • For any input xn you will always get exactly the same output yn M http: //en. wikipedia. org/wiki/Idempotence D Copyright 2010 Dan Mc. Creary & Associates 15 of N

Side Effects Side effect y = f(x) Side effect • Ideally the same inputs will ALWAYS give you the same output • Side-effects are when something in the environment changes and you get unexpected results • Side effects make things very difficult to run in parallel threads M D Copyright 2010 Dan Mc. Creary & Associates 16 of N

Constant Example y = math: pi() • y should always return the same result • There are no inputs M D Copyright 2010 Dan Mc. Creary & Associates 17 of N

Simple Function Example y = add(1, 2) • Y should always return the same result of "3" since the values of 1 and 2 are "static" M D Copyright 2010 Dan Mc. Creary & Associates 18 of N

Transform Example y = transform(x) • Y should always return the same result since the values of 1 and 2 are "static" • Example: count( ('a', 'b', 'c') ) M D Copyright 2010 Dan Mc. Creary & Associates 19 of N

Document Transforms y = transform( doc('my-file. xml') ) • Some transforms are more complex • XML to HTML or PDF • What about a "collection" of documents? M D Copyright 2010 Dan Mc. Creary & Associates 20 of N

Collection Transforms y = transform( collection('/db/inputs') ) • What if there are 10, 000 input documents? • What if you have an 8 core CPU? • Can you distribute the load evenly? M D Copyright 2010 Dan Mc. Creary & Associates 21 of N

Whats Wrong With This Picture? x=x+1 When you start and stop this thread, you need to know exactly where you stopped in order to restart it again and not increment the counter twice. M D What if it fails half way through the pipe? What if you were incrementing an external object? 22 Copyright 2010 Dan Mc. Creary & Associates

Traditional Procedural Loops for (i=0; i < 5; i++) { n = n + i; } M • Each loop can gather information from a prior loop • The state of the loop must be carefully maintained • If you stop a loop you must store the entire state of the context • If any part fails you must restart the entire batch D Copyright 2010 Dan Mc. Creary & Associates 23 of N

XQuery FLOWR Loops a b c d e let $sequence : = ('a', 'b', 'c'…) for $item in $sequence return my-transform($item) • Each loop is an independent parallel execution stream • You can not use the RESULTS of a in b M D Copyright 2010 Dan Mc. Creary & Associates 24 of N

Compositionality • The meaning of a complex expression (function) is determined by the meanings of its constituent expressions (functions) • The secret to building highly reusable web services is to build new services that are composed of other finer grained services (reuse) • The most reliable and testable enterprise services leverage "composed" web services http: //en. wikipedia. org/wiki/Principle_of_compositionality M D Copyright 2010 Dan Mc. Creary & Associates 25 of N

Map Reduce M D 26 Copyright 2010 Dan Mc. Creary & Associates

Transformation on 10, 000 CPUs M D 27 Copyright 2010 Dan Mc. Creary & Associates

Hadoop M D 28 Copyright 2010 Dan Mc. Creary & Associates

Functions as Data y = f 1(f 2, x) M • Functional programming frequently requires you to pass a function as an argument to a function • In the example above function 2 is a parameter for function 1 • These are known as "Higher Order Functions" or HOFs • HOFs were not part of the XQuery "1. 0" W 3 C specification but have been added to e. Xist and other systems since 1. 2 • There is some speculation that HOFs can make XQuery functions much more flexible for some types of transforms D Copyright 2010 Dan Mc. Creary & Associates 29 of N

Functions as Parameters A user defined function for filtering resources from the zip file. compression: unzip( $zip-data as xs: base 64 Binary, $entry-filter as function, $entry-filter-param as xs: any. Type*, $entry-data as function, $entry-data-param as xs: any. Type*) item()* A user defined function for storing an extracted resource from the zip file. M D 30 Copyright 2010 Dan Mc. Creary & Associates

Composability • Composability is the ability to create new functions that refer to other functions • Function "Choreography" • Functions can be web services • Web services can call each other • REST services can be cached M D Copyright 2010 Dan Mc. Creary & Associates 31 of N

Philosophy of Computation Dan's Favorite Quote: "All computation is transformation. " M D Copyright 2010 Dan Mc. Creary & Associates 32 of N

Examples of Transformation • Compilers – Transform a high-level language to a machine code • Brain – Transform electrical patterns in the cornea to visual object recognition M D Copyright 2010 Dan Mc. Creary & Associates 33 of N

"On Intelligence" • The Brain's neo-cortex transformation layers • Each layer transforms signals http: //www. numenta. com/htm-overview/education/HTM_Cortical. Learning. Algorithms. pd M D Copyright 2010 Dan Mc. Creary & Associates 34 of N

Future Intel 80 Core CPU • 80 Core chips are in development today • What happens when your watch has 1, 000 core CPUs? • What programming style will be most useful? Mobile devices with multi-core chips M D Copyright 2010 Dan Mc. Creary & Associates 35 of N

Resources • Sample of passing functions as parameters to the zip module – http: //en. wikibooks. org/wiki/XQuery/Get_zi pped_XML_file • Wolfgang's demo – http: //demo. existdb. org/exist/xquery/call. xql/source – http: //demo. existdb. org/exist/xquery/call. xql M D Copyright 2010 Dan Mc. Creary & Associates 36 of N

Labs • Write a program that takes a function as a parameter – Use the util: function() function – http: //demo. existdb. org/exist/functions/util/function • Write an program that takes a zip file in one collection and puts another collection – http: //en. wikibooks. org/wiki/XQuery/Get_zi pped_XML_file M D Copyright 2010 Dan Mc. Creary & Associates 37 of N

Thank You! Please contact me for more information: • • • Metadata Management Services Web Services Service Oriented Architectures Business Intelligence and Data Warehouse Metadata Registries Semantic Web Dan Mc. Creary, President Dan Mc. Creary & Associates Metadata Strategy Development dan@danmccreary. com (952) 931 -9198 M D Copyright 2007 Dan Mc. Creary & Associates 38 of N