Exchange Design Best Practices Tools for Successful Flow

Exchange Design Best Practices Tools for Successful Flow Design and Implementation 1

Flow Design & Implementation Life Cycle 1. 2. 3. 4. 5. 6. 7. Planning Flow architecture design Determine data elements Draft Schema and submit for initial review Develop flow components (sender/receiver) Implement and test Document and submit for final review and publishing 2

Planning Phase • Determine goals of the exchange – Often a regulatory reporting requirement involved (Submit) – Consider data publishing model • Expose queries • Make receiver “Come and get it” • Still satisfies regulatory requirements – Consider other uses for the data – make the exchange flexible! • Ad hoc uses of data (human) • Support warehousing by any partner (machine)3

Flow Architecture Design Phase • Determine how to chunk data – Schema and Data Serviceswork hand in hand – Compartmentalize Schema by data module – Enables returning portions of data • Get. Facility. By. Name(“ACME”) • Get. Facility. Contacts. By. Facility. ID(“ 1234”) 4

Flow Architecture Design Phase • Design feedback mechanisms and fault condition behavior – Many teams only considered the “happy flow” – Rejected files and partial rejection behavior should be considered and documented – “Get. Status” responses should be explicitly defined for each stage of processing 5

Flow Architecture Design Phase • Determine how to handle Transactional Processing – Ideal scenario – Avoid it! • Data provider should need no prior knowledge of receiver’s data • Receiver should know what data to request and how to consume the returned data • MUCH easier for data providers to implement – If it can’t be avoided… • Use Header “Payload” element to track “Insert/Update” and “Delete” • Last resort: Use transaction codes in schema 6

Determine Data Elements • DON’T design the schema by mirroring an existing database model – Hierarchal structure of schema is best for maximizing interoperability • Relationships expressed through nesting, not keys • Key/Keyref requires data provider to do more elaborate data staging – Goal is to make the data abstract • That said, evaluate each data source for candidate elements 7

Schema Design and Draft Review • Schema design: How rigid should it be? Flexible Rigid Flexible: • no explicit field lengths, fewer required fields, no Key/Keyref, embedded code lists • Use the Shared Schema Components!!! Rigid: • Shorter shelf life (i. e. code lists change) • Usually done to be target system-centric • Limits likelihood that schema will be reused by other flows or matches data standards 8

Schema Design and Draft Review • Compromise! – Keep schema loose (flexible) – Use schematron to enforce target-specific rules • • EPA CDX hosts a schematron validation Few resources to help with Schematron More powerful than good old schema validation Schematron rules can be modified if business rules change without revising the schema! 9

Schema Design and Draft Review • Using the Shared Schema Components 10

Schema Design and Draft Review IMPORTANT THING #1 – Follow the design guidance! • Review the guidance on the EN web site! • If you have questions, get answers! IMPORTANT THING #2 – Submit for Initial review! • Exchange Network offers “early engagement” services to help uncover issues with your schema and exchange design before sinking significant resources in flow processing code! – Major issues have been identified every time! We are here to help! 11

Develop Flow Components • Build the data staging tables – Good practice: stage the data in a data model similar to the schema format – Create SQL scripts for staging tables and publish with the exchange • Build the flow composition/decomposition components – Use OO design to build parsing routines • Maximizes the likelihood for reuse! • All flow deliverables (including code) are part of the public domain 12

Implementation and Testing • Testing will expose problems – Document issues as they arise • Other implementers will encounter the same issues – Performance issues with large files – Test iteratively – Be prepared to revise schema and/or data services if needed – Test between multiple partners – Test end-to-end, including feedback 13

Submit Flow Package for Review • Prepare flow documentation package – Complete and clean up the FCD, DET, and schemas – Create sample XML instance documents – Finalize Schema Conformance Report – Review EN guidance before submitting • Required/Optional materials • Proper schema packaging – Index. xsd – Folder structure – Submit for review – (2 -6 week turn around)14

Schema Design Rules and Guidance ***Use other published flows as a reference 15

Pop Quiz Question: A developer is not sure how to design some aspect of a schema or flow. What should he/she do? Answer: A. Ignore the problem. It will go away. B. Just guess and move on. C. Obfuscate the issue with confusing language in the flow documentation. D. Review the guidance documents. If no answer if found, ask the NTG for assistance. 16

Schema Design Do’s DO: • Read and follow the Design Rules and Conventions (DRC) v 1. 0 and v 1. 1 • Follow Naming Rules (Schema elements and types, Namespace, and Data Services) • Follow versioning rules • Use the Shared Schema Components in your schema • Follow schema file segmentation guidelines • Ask questions • Submit your draft schema for review 17

Flow Configuration Document Do’s DO: • Thoroughly document all data services (parameters, order, required/optional, return schema) • Thoroughly document payload structure (use of Header, zip files, payload operations, arrays of Node. Documents) • Assume the reader is not familiar with your flow • Use the FCD template and other FCDs for reference • If the Header is used, thoroughly document how and when it is used 18

Flow Configuration Document Don’ts DON’T: • Include non-flow specific stuff (i. e. how to ping, what the header is) • Forget to remove comments, track changes or other editing artifacts 19

The Moral of the Story 1. Take advantage of flow development assistance services offered by the NTG! 2. Read, understand, and live the guidance! 3. Sometimes there are no easy answers. . . Ask questions! Buffer your decisions! 20