IBM Research Legacy Revitalization Through Web Services Howard

IBM Research Agenda § Objectives and common challenges § Enabling technologies § Approaches Analysis and planning Connecting Refactoring Transforming 2 Legacy Revitalization Through Web Services | Web Services Council © 2003 IBM Corporation

IBM Research Objectives and common challenges | May 22, 2003 | Web Services Council

IBM Research Objective – responsive, flexible business systems 4 Legacy Revitalization Through Web Services

IBM Research e-business on demand is the target “ An enterprise whose business processes – integrated end-to-end across the company and with key partners, suppliers and customers – can respond with speed to any customer demand, market opportunity or external threat. ” IBM corporate presentation on e-business on demand 5 Legacy Revitalization Through Web Services | Web Services Council © 2003 IBM Corporation

IBM Research Challenge #1: size and complexity of legacy apps Client A Client B Applications 760 995 CICS regions 35 155 Transactions 17, 000 ? Online datasets 3, 300 25, 600 Batch jobs 27, 000 91, 900 Job steps 260, 000 816, 600 1, 370, 000 5, 342, 000 31, 000 70, 000 Data elements declared 15, 000 59, 993, 000 Data elements referenced 3, 000 13, 367, 000 Source, copy, JCL, PROC members 91, 000 292, 900 DD statements Compilation units 6 200 applications and their 1, 700 file dependencies in batch … don’t ask about database dependencies or online… Legacy Revitalization Through Web Services | Web Services Council © 2003 IBM Corporation

IBM Research Challenge #2: lack of cohesion Redundancy and coupling are also problems. 7

IBM Research Challenge #3: technology limitations Technology change and lack of technology standards are also factors. /** * @deprecated Replaced in JDK 1. 3 */ //REPRO EXEC PGM=IDCAMS, PARM= EXEC CICS SEND MAP (‘DMHMZ 88’) Technical constraints live on in applications long after they’re removed from the infrastructure. Application-specific capabilities may not exploit runtime infrastructure (e. g. security) 8 “Note that because some of the standards are still not complete, the stable API will definitely be different from its current form in Xerces-J 1. 4. 4. This is your chance to give us feedback on the features that are important to you, and let us know whether the APIs that we are providing are the right ones. Please direct your feedback to the Xerces-J mailing list. ” Legacy Revitalization Through Web Services | Web Services Council AMODE 24, RMODE ANY © 2003 IBM Corporation

IBM Research Challenge #4: skills Skills are hard to develop and maintain. Many people skilled in legacy technologies are reaching retirement age. Few schools teach traditional batch and transaction application development. 7, 234, 574 Essential Best Practices for IT Professionals 2453 th Edition Guaranteed to make your head spin! The new technologies aren’t easy, either. 9 Legacy Revitalization Through Web Services | Web Services Council © 2003 IBM Corporation

IBM Research Enabling technologies | May 22, 2003 | Web Services Council © 2003

IBM Research Runtimes and tools for development and testing § Web. Sphere family, including § § § § 11 Application Server, Portal Server, Edge Server … Messaging (e. g. JMS and MQ) Connectors and adapters (e. g. JCA) Flows and complex adapters (e. g. Web. Sphere Business Integrator) Connectivity to traditional environments (e. g. CICS and IMS gateways) Bypass of screen handling (e. g. CICS bridge, i. Series Web. Facing) 390 3 GL Java interoperability (e. g. COBOL, PL/I) 390 3 GL XML support (e. g. COBOL and PL/I) § Testing (e. g. Web. Sphere Studio, § § § Rational) J 2 EE, Web and Java development (e. g. Web. Sphere Studio, Rational) Connector development, flow composition (e. g. Web. Sphere Studio) Connectors to terminal applications (e. g. Host Publisher) Business modeling, component composition (Web. Sphere Business Integrator) 390 XML enablement (Web. Sphere Studio) Understanding existing applications (Web. Sphere Studio Asset Analyzer) Legacy Revitalization Through Web Services | Web Services Council © 2003 IBM Corporation

IBM Research Tools for legacy analysis and transformation § Legacy-specific tools Extract COBOL code to subroutines (e. g. Relativity’s Rescue. Ware, IBM’s Web. Sphere Studio Asset Analyzer) Legacy apps Extract physical application models, Subroutines; Create connectors Identify program statements (“business rules” related to selected data or statements (see above) § Web and Web Services tools Create connectors to legacy screens or transactions (IBM Web. Sphere Studio Application Developer – Integration Edition, IBM Host Publisher) Create business and technical models, compose applications from services (IBM Web. Sphere Business Integrator) § But the legacy tools have gaps identified by review, pilots Rearchitecture planning including data consolidation; isolating code in large, complex programs; reaggregation of fragments into components; aligning apps to business 12 Legacy Revitalization Through Web Services | Web Services Council © 2003 IBM Corporation

IBM Research Approaches | May 22, 2003 | Web Services Council © 2003 IBM

IBM Research Analysis and planning § Scope is widely variable, depends on business objective, i. e. “How can my Web application retrieve status information from IMS transaction XYZ? ” “I need to get a consolidated view of a customer’s payments and balances to our different business units. ” “We need to outsource non-core processing. What can go, and what needs to stay? ” “How do we test this? ” § Answering these questions requires a combination of top-down and bottom§ § 14 up analysis, and mapping between them Web Services are a means to achieving business ends, not ends unto themselves Analysis and planning in the small may be done using existing skills and methods Analysis and planning in the large tie into Application Portfolio Management Let’s look back at some best practices from j. Start Legacy Revitalization Through Web Services | Web Services Council © 2003 IBM Corporation

IBM Research j. Start – Project Definition Workshop § Understand the structure and the dynamics of the organization in which a system is to be deployed § Capture business and IT drivers § Evaluate a collection of potential projects to understand their problem domain and identify business value of Web services Define and scope the business objectives of a particular project Develop a business use case model for a particular project and capture initial set of requirements § Ensure that critical success factors such as management and financial commitments are identified § Deliverables: Project Description/Scope/Objectives Initial Use Cases and with High Level Descriptions Initial Requirements, Dependencies and Assumptions System Context Diagram 15 Legacy Revitalization Through Web Services | Web Services Council © 2003 IBM Corporation

IBM Research j. Start – Project Readiness Workshop § Define Feature and Supplementary System Requirements Establish agreement with the customer and other stackholders on what the system should do Capture detailed requirements (functional and non-functional) § Perform System Use Case Analysis Define system behavior Define the boundaries of the system Identify users and external systems to be interfaced § Logical Architecture Analysis Map requirements/drivers to IBM Patterns for e-Business Review solution alternatives based on IBM Patterns for e-Business Identify flow of events Refine dependencies and assumptions § Deliverables Use case diagrams and specifications Architecture document Component model Activity and interaction diagrams Initial class diagrams Create “Statement of Work” documents 16 Legacy Revitalization Through Web Services | Web Services Council © 2003 IBM Corporation

IBM Research Combining top-down and bottom-up analysis Target processes Target data Target architecture Target skills Mapping, gap analysis Current processes 17 Current data Current applications Legacy Revitalization Through Web Services | Web Services Council Pla n Current skills © 2003 IBM Corporation

IBM Research Bottom-up impact analysis §Batch JCL Provides program execution, dataset creation and use, database use JCL Parsing Tools Works for package software without source Scheduler information adds information on data flow §Online Transaction/File Data Element CICS, IMS Parsing Tools Program or Copy Source Data Record Definitions Provides program execution, dataset and database use §Application Program Source Provides data record definitions, data store I/O, call relationships, data element use Typically COBOL, PL/I and assembler §Database Definitions Program Parsing Tools Program Job Step File, Database Transaction PSB/DBD Parsing Tools Provides linkage between source and database (IMS PSB) 18 Legacy Revitalization Through Web Services | Web Services Council © 2003 IBM Corporation

IBM Research Bottom-up analysis results § Bottom-up analysis yields a bill of materials, with ability to drill down to the element and statement level § When used in conjunction with mapping, we create a foundation for assessing technical impact of business change 19 Legacy Revitalization Through Web Services | Web Services Council © 2003 IBM Corporation

IBM Research Mapping approach Business Models Business Processes Project Database IBM Web. Sphere Studio Asset Analyzer with IBM Research extensions Mapping Data Models WSAA Scan application source, JCL and online environments (e. g. Web. Sphere Studio Asset Analyzer) 20 Legacy Revitalization Through Web Services | Web Services Council © 2003 IBM Corporation

IBM Research Mapping tools help the project team relate the data in master files and databases to the data in the target models Mapping is not limited to data. Mapping of procedure -- business functions, transactions. services or methods -can be done to the program or paragraph/block level 21 Person Name Mailing Address 10 CUSTOMER-NAME. 15 LNAME PIC X(30). 15 MI PIC X. 15 FNAME PIC X(15). 10 CUST-ADDR. 15 STREET PIC X(40). 15 APT PIC X(5). 15 CITY PIC X(30). Name First. Name Middle. Name Last. Name Address Street. Address Postal. Code Mapping data is stored in the project database, where it is used as an anchor for impact analysis, i. e. "What is affected if I change customer information capture, update and reporting? " Legacy Revitalization Through Web Services | Web Services Council © 2003 IBM Corporation

IBM Research Strategy, partitioning and planning The mapping and impact analysis information can be used to validate the service and component models: ƒ Are all of new model's components traceable back to the legacy systems, or known to be untraceable? ƒ Are all of the data and functions of the legacy systems accounted for in the new models, or known to be irrelevant? The desired disposition and strategies need to be defined for each affected file/database and collection of programs, then recorded in the project database. 22 The interfaces and interdependencies between affected components are used to partition the overall effort into manageable work units A detailed legacy transformation plan is created for each work unit Replace Reuse Retire Refurbish Legacy Revitalization Through Web Services | Web Services Council © 2003 IBM Corporation

IBM Research Connecting § Tools for JCA, JMS style connectors to callable/message-oriented transactions § Tools for screen connectors § Tools to build Web Services from Java § Tools to help XML enable COBOL programs (viewlet demo) 23 Legacy Revitalization Through Web Services | Web Services Council © 2003 IBM Corporation

IBM Research XML support in Enterprise COBOL Local-Storage Section. 1 business-datastructure. 2 coordinates occurs 5 times. 3 x pic 9(4)v 9(4) binary. 3 y pic 9(4)v 9(4) binary. 3 z pic 9(4)v 9(4) binary. Linkage Section. 1 XML-Interface. 2 XML-length pic 9(9) binary. 2 XML-Bytes pic x(1048576). XML Procedure Division. call 'inbound' using business-datastructure XML-length xml-bytes call 'busprog' using business-datastructure call 'outbound' using business-datastructure XML-length XML-bytes XML 24 goback Legacy Revitalization Through Web Services | Web Services Council XML to data structure converter Business program Data structure to XML converter © 2003 IBM Corporation

IBM Research Refactoring § § Refactoring is changing the implementation without changing the function This is a well-accepted practice in the Java world, but not in the legacy world Refactoring requires disciplined approaches to version management and testing Examples of legacy refactoring: 1. Isolating I/O into called subroutines 2. Extracting callable subroutines from programs 3. Removing obsolete programming constructs 4. Restructuring program control 5. Standardizing data names and copybook use 6. Separating screen handling from business logic § § § 25 Tools support is widespread for 1 and 2; 3 -5 are typically niche tools that have not had broad adoption; 6 is hard, even with current tools. Tools support is limited for test planning and creating test cases for use by regression testing and coverage-based test tools. Web and Web Services tools facilitate legacy refactoring by creating a stable public interface behind which refactoring can occur Legacy Revitalization Through Web Services | Web Services Council © 2003 IBM Corporation

IBM Research Refactoring example Monolithic edit and update programs processed multiple transactions and multiple database record types. Extensive use of complex conditional tests based on transaction type. No data encapsulation. Driver Rearchitected programs provided identical function while reducing overall complexity, increasing testability, maintainability and encapsulation. Router Gap analysis was simplified; database migration and wrapper construction were enabled. Example: One large program was split into 23 separate transaction handlers plus supporting routines. They represented core update processes and data in the legacy system. 26 Legacy Revitalization Through Web Services | Web Services Council Transaction Processors Shared Routines Database Access © 2003 IBM Corporation

IBM Research Refactoring example The monolithic program masked the relationships between business transactions and data. The rearchitected program allowed mapping at a very granular level, whichsimplified gap analysis and transition planning. Gap analysis could use the transaction-to-data relationships to leverage the mapping. Rearchitecture had additional immediate benefits of improved maintainability and testability. Driver Router Transaction Processors Database Access 27 Database Access Legacy Revitalization Through Web Services | Web Services Council Transaction Processors Database Access © 2003 IBM Corporation

IBM Research Refactoring facilitating transformation Rearchitected programs provided a better starting point for specifying changes to the legacy system. The programs that were rearchitected contained the key update logic for key master files. Other, lower-priority programs were not changed. Database Access Driver The new system models and business priorities were used to determine which legacy data were to change. Router The mapping from new system models to legacy data was used to identify the elements, records, and programs that would be affected by the change. The legacy system inventory allowed impact analysis across multiple applications, so interface requirements could be identified quickly and thoroughly. Transaction Processors Database Access The impact was reviewed, alternatives were reviewed and estimated, and a transition strategy was selected. 28 Legacy Revitalization Through Web Services | Web Services Council © 2003 IBM Corporation

IBM Research Transforming § Changing the function of the legacy system, e. g. Adding new interfaces, connecting to new channels Consolidating redundant programs and data Implementing packaged software Integrating new technology (rules engines, workflow) Retiring software and data Porting, rewriting § Driven by business priorities, business cases § Alignment with component-based business architectures, service oriented architectures 29 Legacy Revitalization Through Web Services | Web Services Council © 2003 IBM Corporation

IBM Research “Simplify to Succeed” § "To move to this [component- based business] architecture, existing systems need to be re -aligned and possibly augmented with new facilities to match component boundaries, dividing, extending and retracting their scope as necessary. " § "Align your IT portfolio and architecture with your business architecture. Ensure that the transformation is business-driven and has tight linkages to IT. " 30 Current complexity and overhead … ROI Legacy Revitalization Through Web Services | Web Services Council Organizational Size and Complexity © 2003 IBM Corporation

IBM Research Transformation example §Problem Client in retail industry needed to consolidate redundant applications for business flexibility and customer service Legacy Applications §Approach Business Processes Legacy systems inventory Gather data by identifying functions and data Repository Data Models Identify function and data overlap Identify redundancy Find best-in-class system components Conduct application workshops Plan to reduce redundancies 31 Legacy Revitalization Through Web Services | Web Services Council © 2003 IBM Corporation

IBM Research Transformation example (continued) Redundant Functions - same business function represented multiple times §Enabling Technology Inventory collection tools Relational repository Data model reverse-engineering and mapping tools Application 1 H D §Results Consolidated multiple file-based structures to DB 2 32 Application J 3 A Testing tools Restructured and consolidated applications software: more programs – a new driver and smaller and simpler business logic modules --for easier addition of new transaction types I B D Application 2 L C K E E M Unique Functions - one function performed by by one application Common Functions - same function performed by multiple applications Legacy Revitalization Through Web Services | Web Services Council © 2003 IBM Corporation