Application Integration The devil is in the details

Application Integration ”The devil is in the details” 1

Content • Why is integration an issue? • How do I recognize a decent integration architecture when I see one? • Could there possibly be something called an Integration Model? • Problems you might run into as an application integrator! • Saved by an Application Integration methodology! • Corus Quicklink – What sort of hack is that? What the heck does it do? 2

Why is integration an issue? • Demand of new and improved business processes • Time To Market • Protection of IT investments • The dream of the Connected Economy 3

Why doesn’t standards solve the integration problem? • Having many standards is like having none • Standards does not protect you from semantical differences • Standards are not the solution to the system legacy • They are however a goal to aim for. 4

Why are there so many buzzwords in this area? • Application integration has become HOT • No generally accepted problem definitions • Complete mixup in terminology – – – Standards Infrastructures Architectures APIs Protocols Products 5

What is application integration? • • Seemless interaction between applications Doing the unintended Old thing, new requirements Both but independently information and access 6

Real life integration scenario 7

How do I recognize a decent integration architecture when I see one? 8

Rocky road to integration. . . • Complexity increases heavily with number of integrated systems • Heavy impact on the integrated systems • System dependencies reduces availability for integrated systems • Hard to support and change existing integration solutions • Integration crosses organization boundaries with “political” discussions as a result 9

Complexity O(n 2) System A System B System E System C System D 10

Complexity O(n) System A System B System E System C System D 11

Distributed Business Logic System A System B System E System C System D 12

Central Business Logic System A System B System E System C System D 13

Synchronous delivery System A System B System E System C System D 14

Asynchronous delivery System A System B System E System C System D 15

Distributed knowledge System A System B System E System C System D 16

Central Knowledge Base System A Knowledge base System E System B System C System D 17

Ad-hoc integration model System A System B System C System E System D 18

Integration model with common view System A System B System C System E System D 19

Could there possibly be something called an Integration Model? 20

Modelling reality Picture of a Forest Sun Organisms 1. . 1 Area 1. . 1 Trees Animals 0. . * Model of a Forest 0. . * Litter Forest Lumber harvest 21

Use of a model • Declared semantics of information objects • Declared syntax of information objects • But then there is operational semantics which is uncaptured by the model because it occurs in the eyes of the beholder 22

Operational Semantics Program ----------------------- Organisms 1. . 1 Area 1. . 1 0. . * Trees Animals 0. . * Litter Forest 23

Integration Model • Contains all application views of information objects + common reference view • Describes business rules in the form of – – Integrity checks Transformations Filters. . 24

Integrity checks • Can be local to an information object IF car. Type = ”SPORTSCAR” THEN horse. Power > 200 • Can also be referential i. e. depending on other information objects Car. driver EXISTS IN Driver. License. Register. driver 25

Transformations • Vital to a integration model because they describe how to go from one information view to another • Note, transformation can (and often do) ”fork out” or ”fork in” i. e. one information element in one view correlate to several in another information view. 26

Syntax transformations Sales Support view Customer: • Name 30 chars • Phone number is mandatory • Identified by a string Project management view Customer: • Name 20 chars • Phone number is optional • Identified by a number Finance department view Customer: • Name 30 chars • Identified by Phone number Common view Customer: • Name 30 chars • Identified by internal id • Phone number optional • Any other attribute 27

Semantical transformations Sales Support view Customer: • Created at first contact • Never deleted Project management view Customer: • Created at project start • Deleted at project end Finance department view Customer: • Created at first invoice • Deleted 10 years after last invoice Common view Customer: • Common lifecycle for all views which implies new state attributes 28

Filters • Filters are criterias describing when information in different views are eligible for delivery to applications. 29

Integration Model 30

Use-cases • • Use-cases are good practice in modelling a integration scenario Use-cases translates to an chain of operations in the integration model 31

Error handling • Since there are no good receiver of errors in an integration scenario. Error situations and their counter actions must be modelled • It is common that the error situation usecases outnumbers the normal operation use-cases (if not, you might have neglected some possible error situations) 32

Problems you might run into as an application integrator! 33

Some important problem cases • Multiple master synchronisation • Asynchronous information correlation • Message operation calculations 34

Multiple master synchronisation System A Loops Customer System A Different latency times Customer System B Integration environment Once per hour Customer System B Integration environment Customer Once per second 35

Asynch. information correlation System A Once per hour Customer System B Integration environment Order with valid customer System C Order Once per second 36

Message operation calculations Bad create System A Customer System B Integration environment Good update System A Customer Bad create Integration environment Customer Good create System B Customer ”Do nothing” Good remove 37

Problem summary • Technical by nature • Asynchronous communication gives extra complexity (Asynch comm is however necessary to have loose integration) • The solutions of these problems involves the integration model 38

Saved by an Application Integration Methodology! 39

IS-supported business processes Find product Place order Marketing Pay Use product Selling Sales Production Productio n Deliver Business plan Invoicing Web portal Customer Product def. order PDM Finance Product def. custom er Information systems plan CRM Paying custom er OA 40

Top-down approach Order_header order_id customer status From process-analysis to mapping bit and bytes, preferably with the same tool and methodology! number(20) varchar(30) varchar(1) g Order member n pi ap varchar(10) varchar(30) ru l es nd m a 41

Methodology anatomy feedback Integration process Business analysis Implementation Utilities and common resources 42

Anatomy; continued Integration process Overall control of an integration project. Assures that relevant benefit is delivered to customer Business analysis Assure that the integration will be relevant according to business goals. The business analysis is independent on the integration tool used. Implementation Defines an efficient way to implement the ”whats” of the business analysis Data access Subpart of implementation, dealing with connectivity between applications. - How can applications be accessed? Information content Subpart of the implementation, dealing with transformation and interpretation of information - How should information be handled when transferred between applications. 43

A closer look at business analysis feedback Integration process • • • Business analysis Implementation Goal model Process / activity model Utilities and common resources Conceptual model Application-Activity-Object model Relatively small part of projects and when it is necessary! 44

Business analysis Workshops • Current state • Future state • Investigate the need for integration • Evaluate for solution 45

To create a relationship between organization intentions and the objectives for integration Goal model «goal» Business continuance «quantitative goal» Increased profit facilitates «mean» Focus on core business «quantitative goal» Minimize costs prerequisite for «sub-goal» Supply chain management enables «mean» Business automation Is part of contradictory Mitigates problem «mean» 1) Obtain Web-system that facilitates XML-order/invoice 2) integration of order/invoice between Web and financial appl. «problem» No XML order/invoice capabilities in financial application 46

Process/activity model Buy Marketing Pay Use product Customer Sales Sell Produce Productio n Deliver Invoicing Finance Identify organization business processes and their interaction 47

Process/activity model; continued Sell register customer Identify prospects Prepare prospects place order Identify: • the activities within the processes • the need for information • where the information is available • information produced • where the created information is stored 48

Conceptual model Organization Customer Person is placed by Order • • • contains Product Defining the participating business objects and their associations Optional UML class diagram 49

Application-Activity-Object diagram Object source OA WEB CRM WEB register customer CRM register customer Object target Product balance product customer OA product developmentproduction customer PDM place order Activity performed in OA customer product PDM WEB CRM PDM 50

Business analysis; continued Strict integration focus: • Eliminate activities where no information flow is prevalent • Order between activities are not important • Where activities can be performed are important • (potential) source / target of information are important 51

Result of business analysis • Consistent view on integration need - or alternatives to integration! • Integration goals and means mapped against organization intentions or visions • Input to implementation resulting in transparency between analysis and implementation application-activity-object diagram application collaboration diagram • Conceptual model - optional, used for consistency control 52

A closer look at implementation feedback Web-site Integration process Business analysis Implementation Utilities and common resources Implementation Information content Data access 53

Application collaboration diagram nfo i g lin il r, b me o t us C • • • ) nf i g lin bil Web. Portal y pa ( o te da Finance system Spedition order (shipping date) Storage system Integrated applications described as subsystems Only data flow (normally asynchronous) Integration objects or their sub-sets No attributes or similar details Collaborations outside integration for visualization 54

Integration syntax and semantics Sales department: Customer is: • anyone which may be involved in business • 30 chars in name • phone number is a non-mandatory attribute • identified by a string Finance department: Customer is: • Legal entity which has been involved in a business transaction • 25 chars in name • identified by organisation number (mandatory) Reference: Project management: Customer is: • Anybody within an active project • 20 chars in name • phone number is a mandatory attribute • identified by a number Customer is: • anyone which may be involved in business • 30 chars in name • identified by internal key (mapped against a string, phone number and a number) 55

Information content Integration model Finance system Sale support system Proxy model Business rules Reference model Project management 56

Use case diagrams - for requirement specifications Publish use case «extend» Application A Subscribe use case Application B Actors: integrated applications Publish use case: responds to creation, update or deletion of integration objects in publishing application with C, U or D in ref. model Subscribe use case: C, U or D of integration objects in subscribing client application as a result of C, U or D in ref. model 57

Use cases; continued Publish from WEB Use case Scenario (condition) Result A customer enters member information - Customer is created Order is entered Related to a customer Order is created Not related to a customer Error case Reference model Use case Scenario (condition) Result Customer is created - FIN: null Order is created - LOG: Order is shipped FIN: Billing info is created 58

Compilation of integration model • Proxy models and reference model defines static structure of information (relations between integration objects, i. e. referential integrity) • Data flows in application collaboration diagram defines which transformations must be created • Use cases describes the behavior of the transformations (and events for integration) • Attribute mapping is a part of defining the transformations (may be separated later on) 59

Class diagrams for integration model po 1 Customer * -Organization no -Name p 1 0. . * Project * po 3 po 2 po 5 1 po 6 Project -Customer -Contact person -Project no 0. . * po 3 Project po 4 Proxy model A Customer -Organization no -Name po 4 Reference model Integration model: - Proxy model - Referential integrity constraints - Reference model - Post-operations po 7 Proxy model B 60

Corus Quicklink What sort of hack is that ? What the heck does it do? 61

Where does it come from? • Based on experiences made from a number of integration projects • Theoretical reasoning • Demands from customers, implementers and maintenance staff 62

Corus Architecture i BUILDER server i BUILDER • i CONTROLLER • i CONFIGURATOR • i USER HTTP Server Repository Servlet Engine Generators Participant UI layer HTTP Server i. DM ODS Servlet Engine Participant UI layer App layer DB layer i. DM App layer i ENGINE DB layer 63

Integration object - Quicklink atom View Pre OP Table Message trigger Post OP History log Event log 64

Operational Data Store . . . 65

System access Client application B Client application A Java VM Integration Engine ODS Plug-in Generic INI file Log file queue table API Dialog Manager Java VM Dialog Manager Plug-in Publishing adaptor Subscribing adaptor Generic queue table INI file Log file 66

Feature example - Pending events • Solves the problem with inconsistent messages – what can the publishing application do with an error? • A pended message stays in the message table and postoperation is not performed. • Can self-heal Integration engine Subscribing system Publishing system Subscribing system 67

Feature example - Consistency monitor • Autonomous process checks consistency • Can release pending events CM Integration engine Subscribing system Publishing system Subscribing system 68

The Corus Integration Process Order process Information Model Production IA Information Model digital Data Model Application 1 Data Model Application 2 digital WAN IS/IT // Kundinfoadaptor // for system sys. A procedure send_kundinfo { for columns 1 to 3 do { store Kunder. col(i); } … //user part begin call datum_check //user part end } Generator Repository 69

Corus Methodology (Implementation) 70

Goal model for integration automation Competetive integration Prerequisite for Facilitates Solving all types of integration Rapid implementation Prerequisite for Facilitates Flexibility and advanced functionality Contradictional to Simplicity leveraged by Methodology 71

Basic principles of Corus methodology • Top-down approach – from process, all the way down to the bits and bytes • Iterative development and successive implementation • Transparency – no hand-over – single, minimum documentation – all deliverables persistent in one single repository • Problem split – technology vs. business – decisions will be made by correct persons • Plug-able. Can be adopted to RUP or PROPS • Middle of the road (solve 80 %) • Standards where possible – UML for documentation and visualization 72

But note! • The methodology is not absolutely necessary for integration with Corus Quicklink – A solution can be obtained ad-hoc – Quicklink is central . . though highly recommended 73