China Mobile CICD Approach for NFV Cloud Chen

China Mobile CI/CD Approach for NFV Cloud Chen Liang China Mobile chenliangyjy@chinamobile. com

Contents • • Challenges for NFV implementation and testing CI/CD approach to solve NFV implementation problem Data description file used in China Mobile HDV toolsets

Challenges for NFV implementation and testing • Multi-vendor hardware and software • Automation tools for different vendor can’t work seamlessly • Huge workload and errors happen for cross-vendor integration

CI/CD approach to solve NFV implementation problem end to end CI-CT-CD pipeline between vendor and operator. Vendor Delivery CI Vendor A 5 3 1 HW Preparation Vendor B CMCC Local Vendor C Online product Automatic integration + acceptance test CD Integration Test Lab: Crossing Vendor Test CT Image Repos • New version Deploy Test CMCC Integration Test Lab CI 2 Deploy Acceptan ce Test Verified Image • Certificated version Production 4 Online

集成��室CI��流程 CI Pipeline within Integration Lab Jenkins Pipeline Hardware Preparation Software checking HDV SW Deploy Test Preparation Test Report Test Execution Monitor CI GW Use DF Test part Create HW df Use DF HW part Update nfv image to df Use DF SW part Vendor Installer HDV tool Hardware: device name, BMC credentials; BIOS Cabling Connection VIM Deployment Image VIM Settings Version device name, BMC HA, AZ credentials; Node role, ip NIC Setting VIM Verification VIM Access Info Openstack ENV Settings Node role, ip Jump Server HA, AZ Description file(HW&SW&Test) Description file defines entire information of all stage end to end, HW, SW, Test Same data btw Hardware and VIM deployment stage Same data btw deployment and verification

NFVi Implementation Automation • 1. A new version released in vendor lab for public verification • 2. CMCC periodically monitor the remote lab to synchronize the new version if found, and upload to NFVi installer directory • 3. create/update installer Description file (LLD json data) with new version information. • 4. CMCC AUTO platform trigger to install the new version software • 5. CMCC AUTO platform trigger to create pre-condition for running IAAS test • 6. Generate the test case report which would feedback to vendor • CMCC Private Test Lab CICD process implemented in CMCC 2 1. A new version released for vendor’s Lab Vendor remote Lab for public access NFVi Jenkins Server 2 3 4 3 5 LLD 5 CMCC Local Repos 6 6 4 2 Test Report AUTO 4 Vendor’s Installer CICD Zone Test Zone

Value for DF • DF is a single point of truth for the whole process of implementation, from hardware integration to software deployment • Standardize resource description file across different tools for hardware and software, implementation and testing, which is a necessary step for cicd automation. • Reduce time and errors for NFVi implementation , which is common interest both in vendors and operators.

DF we use in China Mobile • CMCC DF(LLD in excel) sample (HW&SW) – offline file demo – Case 1: hw_zte_vim_Ericssion demo – Case 2: hw_spur_vim_huawei • Description file contribute into community – Installer DF PR https: //github. com/cntt-n/CNTT/blob/master/doc/ref_impl/cntt-ri/chapters/chapter 06. md – HDV DF PR https: //github. com/cntt-n/CNTT/blob/master/doc/ref_impl/cntt-ri/chapters/chapter 05. md • Proposed in yaml format

HDV toolset Framework Hardware validation by Redfish interface in centralized checking -define the common check point template per vendor in redfish interface - checking of asset /component/sensor/power/remote access etc. - provide the hardware resource input -HDV engine create the real check point case set based on input above to validate hardware case by case, and generate a report Execution Server Check Point Template in json, yaml etc. HDV Engine Vendor Redfish interface Hardware DF <BMC IP, Host. Name, …> Resource Pool Report

Backup

CICD Pipe line against software stack X App vendor …… VNF vendor X App: Accept ance Test …… …… VNF Vendor commit new Prepar e NFVi Lab NFVi Vendor commit new Prepare Hardwa re Server Build up VNF ENV Lab Build up NFVi Lab VNF: Accept ance Test NFVi: Accepta nce Test NFVi vendor CICD Pipeline: from Vendor to CMCC Software stack extensible

What we are working • Participate standardizing of the input description file hardware validation which – will reduce the hardware validation effort across vendors • Participate standardizing of the input description file for software deployment which – will reduce the software validation effort to production across vendors – will decouple the binding with specific installers, – make efficient software validation across software vendors even on same hardware resource

Description file(software) • Current implementation – Replace image information with new version based on vendor’s installer configure file to perform CICD process • Working on: – Adapt the common description file to the vendor’s installer required configure file in the way show below chart. installers CNTT: descriptor file installers' own configuration + installer specific attribute

CMCC CI-CT-CD Vision AUTO Remote operation 8 Region TIC MEC Integration Test Lab

More integration jobs to add in pipeline • Process about automatic hardware validation • Process about the VNF Application validation

Operator’s plain requirement of NFVi setup • GIVEN: – A cluster of hardware resources with same or limited model – A cluster of routing device – Software option in different vendors. • THEN: • Expected NFVi can set up. • BY: • Efficient hardware validation to ensure the configuration meet requirement. • NFVi software deployment with a common format resource description file (START point) • Necessary verification test in function and performance. (END point) Top view: Operator can obtain the NFVi to setup much efficiently via standard DF Planning and standard Test Suite Validation.