Ashish Jain Mike Bizub Vignesh Microsoft CSEO How
Ashish Jain, Mike Bizub, & Vignesh Microsoft CSE&O How Microsoft ITSukumar does Integration
Agenda • Recap of who we are and what we do • Composable architecture and how we’ve implemented the Routing Slip Pattern • Hot-Path Telemetry • ALM, Dev-ops, and Governance • Business Continuity and Enterprise Disaster Recovery • Migration Tools update • Appendix
Microsoft Integration Landscape 150 M+ 3, 400+ 200+ Integration Streams Centrally Managed Microsoft Systems $1. 6 B 100+ Partners – Internal and External Daily bank treasury transaction Biz. Talk Servers in use today Message Standards Multi-Platform Messages Per Month 1, 000 + X 12 – 30% XML-12% Other – 32% EDIFACT-14% SWIFT-12% BTS 2016, MABS, Logic Apps Enabling Business Processes Supply Chain Contract Management Licensing Sales Benefits Legal Payroll Finance Travel Suppliers Distributors … etc ü ü ü
Recap – The People (giving credit where it’s due)
Recap – B 2 B Approach
BTS On-P Source File Trigger File Processor Source SFTP Trigger Source HTTP Request Trigger Destination BTS On-P Destination File Processor Destination SFTP Processor Destination HTTP POST File Deleter SFTP Processor Get Context Transform Business Prop Promotion Adapter Router Msg Pub Adapter Get Context FFDecode Business Prop Promotion Adapter Router Msg Pub Adapter Msg Publisher APIM Policy, FFDecode, Business Prop Promotion, Adapter Router Pub / Sub APIM Policy Msg Subscriber Adapter Router FFEncode Transform Get Context Determines physical endpoint adapter APIM Source APIM Composable Architecture Msg Subscriber Processor
Telemetry • Hot-path • Run Scope for exception handling within logic apps (similar to a try/catch) • All exceptions are not created equal; may need to enumerate through exceptions before posting • Single logic app concept to move exceptions into an Event Hub (not the same used by Logic Apps) • Azure Functions consume exceptions from Event Hub for both App Insights and internal Ticket management system • Warm-path • OOB Logic App infrastructure plus Integration Account • Event Hub and OMS • OMS drives non-flow business exceptions
ALM and Dev-ops IST PST Enterprise Commerce / Volume Licensing Supply Chain / Core Integration Service CFE Common Backlog (shared and triaged across regions) Feature Branches sprint Feature Branches Sprint teams live here: iterating through tasks and user stories until features are ready or sprint ends. What’s passing tests makes it to release Pull request into develop ‘develop’ CI / CD DRI’s work here: Executing PR’s into ‘release’ and managing the deployment across primary/secondary regions. Issues found here can be dealt with by DRI following Git best practices for this scenario Pull request into release ‘release’ Pre-production (PPE) DRI’s work here: Executing PR’s into ‘master’ and managing the deployment across primary/secondary regions. Issues found here can be dealt with by DRI following Git best practices for this scenario Pull request into master ‘master’ Production
ALM and Dev-ops • Git for source control. • Each code check-in goes thru pull request (code review), at least 2 approvals required and enforced thru branch policies • Code is compiled – gated check in build • Policies for security or governance enforcement (e. g. scan for potential security risks or ensure the person checking in code is supposed to be) • Azure Dev. Ops (formerly known as Visual Studio Online (VSO)) for build and release management. • Two builds • • Gated check in – code check in and performs compilation of the code executes unit tests Continuous Integration build – used to deploy/release, compiles the code, runs the unit tests and creates a build structure to be consumed by release/deployment. • Release • • One release pipeline for each environment Each release pipeline consists of steps to perform various tasks to complete tasks in a given environment and run functional tests • Branching Strategy • Following a basic Git practice of using a ‘develop’, ‘release’, and ‘master’ branch • Deployments into CI, PPE (Pre-Production), and Production • Deploy core services components on a sprint cadence • Unit test / functional test • Unit test what can be unit tested • Functional test for things like Logic Apps, APIM Policies, and any other non-unit testable capability that makes up the integration service • In general • Agile != ‘No Process’ • Dev-ops != ‘No Governance’
Security and Governance • IP Whitelist to further secure logic apps (Migration to ISE) • Managed Identities for AAD auth between logic apps (HTTP Actions/Triggers). Remove SAS token from URL • Things to consider in a hybrid environment • Access Management – who has access to my system when considering Paa. S and on-premises resources (e. g. Biz. Talk) • Release Management – Separation of duties between developers and release management in production • Privacy – How to handle highly confidential data or personal data (e. g. cosmosdb, table/blob storage, Azure SQL
AIS Migration Story Metadata driven Architecture Migration Accelerators • • • Schema Provisioning Partner – Agreement json creation Maps to XSLT conversion Metadata creation AB Testing Metadata Creation • Schemas Json • Maps Json • End Point • Flow Context • Partner Flow Context Milestones Achieved
Enterprise Integration Disaster Recovery • Scheduled Sync Solution’s • • • Runtime Sync Ø Logic. Apps scheduled to run every 3 mins to update secondary region Integration account to make them sync and vice versa post DR. q AS 2 Synchronization (MIC Algorithm) q X 12 Synchronization (Control Numbers Sync) q EDIFACT Synchronization (Control Numbers Sync) Artifacts Sync Ø Integration Account Artifacts sync scheduled to run every 1 hour to secondary region and vice versa if needed. q Schemas q Maps q Certificates q Partners q Agreements q Assembly q Batch Configurations q Rosetta. Net PIP Pre/Post DR Scripts Execution • Bump Interchange Control Numbers in secondary region and vice versa post DR.
Appendix • Open-source Accelerator Tools, Demos and scripts available at Github https: //aka. ms/Integrate 2019 • Write to us for any integration queries: btsmigrationtool@microsoft. com
Recap – B 2 B Approach • Use one Integration Account and set of logic apps across multiple Businesses and Trading Partners (Partner info, Agreements, Document Types, Schemas, Maps, Batch Configurations) • Flow concept static but type can be overridden based on message (e. g. different logic apps that process X 12 or EDIFACT messages) • Metadata used to populate runtime properties specific to a TP or a LOB (currently stored in a Cosmos. DB and accessed via Azure Function with caching)
Recap – B 2 B Approach Trading Partner Gateway • MTPB/HTTPS • SFTP Trading Partner Gateway • APIM/HTTPS • SFTP Exchange Protocol • AS 2 • RNIF • BTF Document Protocol • • • Exchange Protocol • AS 2 • RNIF • BTF EDI X 12 EDIFACT OAGIS Rosetta. Net BTF Document Protocol • • • EDI X 12 EDIFACT OAGIS Rosetta. Net BTF Business Process / Orchestration • Custom message processing • Transforms • Business Property Promotion Line of Business Integration • Flat file encoding • LOB specific enhancement (custom processing) Line of Business Integration • Flat file decoding • LOB specific enhancement (custom processing) On-premises Connectivity LOB Application • EHC • EIS Adapter • Biz. Talk On-premises Connectivity • EIS Adapter • Biz. Talk LOB Application
Recap – B 2 B Approach EHC Corpnet BTS 2016 Service Lines (Business Units) Replication for Busn. Continuity APIM Gateway TP Primary Logic Apps EHC Corpnet BTS 2016 APIM Gateway Secondary Logic Apps • Gateway provides an Active/Active store and forward mechanism for B 2 B type traffic. Trading Partners post messages to endpoint which round robins between the two regions. • There are two deployed installations of B 2 B Service. However, only one is active and any time. • At deployment time, artifacts are deployed to both regions. This includes all static data including Logic Apps, Functions, Metadata • During runtime, there are properties that are updated in the Enterprise connectors that must be replicated across both regions for business continuity. In the passive region, there is a set of logic apps responsible for listening to events in primary region and updating the appropriate Integration Account Artifacts • EHC provides secure connectivity to on-premises resources. B 2 B Service uses corpnet BTS 2016 servers • BTS 2016 provides pass-thru connectivity to existing LOB Applications over a wide range of technologies • BTS 2016 provides the necessary capabilities to send and receive messages from LOB Applications
PATPC Message. Info. V 2 PADocument Protocol. X 12 V 2 PADebatch Good. Message s. V 2 Integratio n Account Azure Function PADocument Protocol. Edifa ct. V 2 PADebatch Bad. Messages V 2 Integratio n Account PADebatch Generated. Ack s V 2 Cosmos. DB PABusiness Process Orchestration V 2 PALob Integration. V 2 Integratio n Account PALob. Adapter BTSV 2 EHC API M PAExchange Protocol. AS 2 V 2 Adapte r BTS Onpremises LOB Application Integratio n Account APDocument Edifact. V 2 APExchange Protocol. AS 2 V 2 APDocument Protocol. X 12 V 2 Integratio n Account APBatch Messages. X 12 V 2 APAsync. MDN Processor. V 2 APMessage Sender. X 12 V 2 Integratio n Account APBusiness. Proce ss Orchestration. V 2 APLob Integration. V 2 Azure Function Cosmos. DB APIM PADebatch Received. Acks V 2 APIM Recap – B 2 B Approach
Composable Architecture • Patterns and Composable Services • Composable Flows for integrations • Leverage existing patterns or compose a new one • Microservices should be autonomous • Refactor to net-new, don’t change an existing pattern or service
Composable Architecture • Routing Slip Pattern • APIM, API, and Policies to expose and describe pattern • Context to describe the flow specific parameters that allows for multiple scenarios to flow through one pattern • Generic ARM Templates to describe concepts that may need implementation specific properties at deployment time (e. g. File. Adapter, SFTP, Message publishing and subscribing patterns) • Adapter. Router concept for endpoint implementation details
- Slides: 19