Rayis Imayev June 20 th 2019 Continuous Integration

Rayis Imayev June 20 th, 2019 Continuous Integration and Delivery (CI/CD) in Azure Data Factory

"Coming together is a beginning, staying together is progress, and working together is success. “ Henry Ford

Continuous Integration and Delivery (CI/CD) in Azure Data Factory Session plan: o o o o o Introduction to the Azure Data Factory (ADF) Continuous Integration and Delivery (CI/CD) overview ADF pipeline case study Merging ADF with CI/CD ADF integration with a Git. Hub repository Azure Dev. Ops review Creating integration of Git. Hub based ADF code and Dev. Ops pipelines ADF pipeline amendments and affect on Dev. Ops pipeline Conclusion

Introduction to the Azure Data Factory (ADF) Cloud platform to compose data-driven workflows with steps to move and transform data. § Control Flow § Pipeline § Activity § Datasets § Linked Services § Triggers § Mapping Data Flows § Source/Sink § Transformation Tasks Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev

Continuous Integration and Delivery overview CI/CD process helps to establish a good software development practice and aims to build a healthy relationship between development, quality assurance, and other supporting teams. 1. Develop code 2. Unit test 3. Integrate 4. Acceptance test 5. Deploy to production Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev

ADF pipeline case study (Demo) Sample ADF pipeline Copy all files across multiple containers from one file-based storage store to another. Activities: - Get Metadata - For Each Loop container - Copy Data - … more items Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev

Merging ADF with CI/CD Questions: 1. 2. 3. 4. Can multiple people work within the same ADF? Can I integrate my ADF changes into other people's ADF code? Can I have a process to frequently ship the ADF solution code to various environments? Can a continuous deployment be automated to a production environment? Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev

ADF integration with a Git. Hub repository (Demo) 1. 2. 3. 4. Setting a Git. Hub Repository for your Azure Data Factory Saving your ADF code in a Git. Hub repository Adding changes to the existing ADF code Publishing changes to a live Data Factory and creating ARM templates http: //datanrg. blogspot. com/2019/02/azure-data-factory-integration-with. html Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev

Azure Dev. Ops review Plan, track, and discuss work across your teams. Collaborate on code development using free Git public and private repositories, pull requests, and code review. Azure Pipelines helps you implement a build, test, and deployment pipeline for any app. Using testing services for your apps. Code once and share packages across your organization Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev

Creating Dev. Ops project to deploy ADF code (Demo) 1. 2. 3. 4. 5. 6. 7. New Dev. Ops project New Release Pipeline Variable groups for my deployment solution Adding Git artifact for the deployment Deployment stages and deployment steps within each stage Release pipeline variables Testing a deployment release http: //datanrg. blogspot. com/2019/02/continuous-integration-and-delivery. html Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev

ADF code change and its affect in Dev. Ops pipeline (Demo) 1. 2. 3. 4. 5. Changing ADF code Merging the ADF code to your Git. Hub repository Publishing / Creaing ARM templates to deploy ADF code Monitoring ADF code deployment Validating code changes in other environments http: //datanrg. blogspot. com/2019/02/continuous-integration-and-delivery. html Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev

Continuous Integration and Delivery (CI/CD) in Azure Data Factory Session plan: o o o o o Introduction to the Azure Data Factory (ADF) Continuous Integration and Delivery (CI/CD) overview ADF pipeline case study Merging ADF with CI/CD ADF integration with a Git. Hub repository Azure Dev. Ops review Creating integration of Git. Hub based ADF code and Dev. Ops pipelines ADF pipeline amendments and affect on Dev. Ops pipeline Conclusion Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev

Conclusion: 1) 2) 3) ADF code deployment could be automated with the help of Azure Dev. Ops. Don’t forget to include Disable/Enable deployment steps if your ADF has triggers. Make sure to use Azure Key Vault to store your secrets and reuse them during deployment. Links: http: //datanrg. blogspot. com/2019/02/continuous-integration-and-delivery. html https: //docs. microsoft. com/en-us/azure/data-factory/continuous-integration-deployment https: //docs. microsoft. com/en-us/azure/devops/learn/what-is-devops Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev

Thank you for attending my session! Rayis Imayev Toronto, Canada Consultant / Big Data Integration & Analytics Senior BI & ETL Developer at Mosaic North America http: //datanrg. blogspot. com/ https: //twitter. com/Rayis. Imayev Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev