Resilience Improvements Existing Script Implementation reference tables to

Resilience Improvements: Existing Script Implementation • reference tables to manage test data, for dynamic test execution (retrieve config, store results) • use tables via HTTP (REST API, http: //10. 50. 12: 8000/); GET and POST; populated using Django; Data Table: Assessments Assessment. Id: Unique ID for assessment Record. Label: Descriptive text to help group and compare similar assessment tests, e. g. v. CPE Assessment with ONAP closed loop control, v. CPE Assessment with non-ONAP MANO such as Tacker, Cloudify, Openstack VIM, etc. Status: Assessment Status, e. g. Init, Started, Passed, Partially. Failed, Error, etc. Start. Time: Assessment Start Timestamp. Completion. Time: Assessment Completion Timestamp. Vnf. Details. Id: Refers to an entry in table Vnf. Details Stress. Or. Fault. Id: Refers to an entry in table Stress. Or. Fault. Data Recovery. Time: Time it took for VNF to recover after fault was injected vnf. Orchestrator: NFV MANO component for VNF management and orchestration, e. g. Tacker, Cloudify, ONAP, Openstack VIM, etc. orchestrator. Info: Credentials to use APIs of the NFV MANO (e. g. Tacker, ONAP) for VNF data collection Resiliency. Measure. Param: e. g. failed. Transaction, service. Degradation, packet. Loss, etc. Resiliency. Measure. Value: Requestor. Id: User Id or Application Id, initiating the Assessment Comments: Text Comments added for the assessment record Data Table: Stress. Or. Fault. Data stress. Fault. Id: Unique ID stress. Fault. Type: compute. Host. Service. Failure, sdn. Servicefailure, ovs. Bridge. Failure, compute. Host. Failure, disk. Failure, link. Failure, nic. Failure, host. Tampering, host. Intrusion, network. Intrusion Start. Time: Timestamp when the fault is injected End. Time: Timestamp when the fault is removed/recovered stress. Or. Fault. Status: Started, Failed, Error, Completed/Removed/Restored host. Ip: Compute Host where the fault is being injected command: Stress or Fault command to execute, e. g. service nova-compute restart/stop command. Outcome: Outcome from the stress/fault command execution Data Table: VNFDetails VNFId: Target VNF Id VNFName: Target VNF Name Hosts. List: Compute Hosts where the VNF VMs are launched Networks. List: Networks being used by VNF Vnf. Info: e. g. flavor, image, etc. VMs. List: List of all VMs of the target VNF impacted. VMs. List: List of VMs impacted due to the fault injection, e. g. all VMs hosted by a failing compute host.

Resilience Improvements: Existing Script Implementation Data Table: Assessments Assessment ID Label (tag) VNF Orchestrator Credentials VNF Orchestrator (ONAP, Tacker, Cloudify, ETSI OSM, . . . ) VNF Orchestrator Info VNF Orchestrator Status Resilience Metric (Name, Definition, Formula) Resilience Metric Value (Quantitative measurement) Start Time Completion Time VNF Details ID Stress. Or. Fault ID Recovery Time Resiliency Measure Param Resiliency Measure Value VNF ID VNF Name VNF Info VIM Credentials VIM (Open. Stack, K 8 S, AWS, Azure, GCP, . . . ) App-C SDN-C VM List Network List Host List VNF impacted VM List Requestor ID Requestor (User, Application, Service) Comments Data Table: VNFDetails Container VM (compute) Virtual Storage (Block, Object, FS, DB, . . . ) Network (subnet, VPN, . . . ) Physical Host (with CPU, RAM, OS) stress. Fault ID stress. Fault Type Start Time End Time stress. Or. Fault Status host IP command Outcome impacted VM List ? Data Table: Stress. Or. Fault. Data Physical Disk (HD, SSD, . . . ) Physical NIC (Eth, Wi. Fi, Cellular, . . . ) Physical Link (wire, radio) Logical Link (tunnel)

Resilience Improvements: Existing Table Samples (no sample for VNF Details) id assessment. Label 2 test. Label 1 3 test. Label 1 5 v. CPE Resilience with Tacke 6 v. CPE Resilience with Tacke 7 v. CPE Resilience with Tacke 8 v. CPE Resilience with Tacke 79 v. CPE Resilience with Tacker 80 v. CPE Resilience with Tacker 81 v. CPE Resilience with Tacker 82 v. CPE Resilience with Tacker Data Table: Assessments Data Table: Stress. Or. Fault. Data id status start. Time completion. Time recovery. Time requestor. Id test. Status 26: 31. 9 10 test. Status 28: 15. 7 10 Init 45: 55. 7 0 autouser Init 47: 56. 3 0 autouser Init 49: 18. 0 0 autouser Init 54: 18. 8 0 autouser Init 43: 19. 1 0 autouser Init 52: 07. 6 0 autouser Init 55: 34. 2 0 autouser Init 57: 35. 1 0 autouser comments Tests during development Tests during development stress. Fault. Type start. Time end. Time stress. Or. Fault. Status 1 service restart nova-compute 52: 28. 2 Completed 2 09: 45. 0 Init 3 14: 07. 2 Init 83 07: 58. 3 Init 84 09: 52. 1 Init 85 compute. Host. Service. Failure 21: 00. 9 Init 86 compute. Host. Service. Failure 23: 21. 3 23: 22. 1 Complete 90 compute. Host. Service. Failure 44: 02. 1 44: 11. 7 Complete 91 compute. Host. Service. Failure 47: 57. 3 48: 06. 8 Complete stress. Or. Fault. Id_id vnf. Orchestrator 1 1 19 20 21 22 93 Tacker 94 Tacker 95 Tacker 96 Tacker command host. Ip service nova-compute status service nova-compute restart service nova-compute restart 172. 16. 10. 102 172. 16. 10. 101 command. Outcome [1; 31 m==== AUTHENTICATION COMPLETE === 92 compute. Host. Service. Failure 49: 46. 0 50: 01. 7 Complete service nova-compute restart 172. 16. 101 [0 m [1; 31 m==== AUTHENTICATION COMPLETE === 93 compute. Host. Service. Failure 43: 19. 1 43: 37. 8 Complete service nova-compute restart 172. 16. 101 [0 m [1; 31 m==== AUTHENTICATION COMPLETE === 94 compute. Host. Service. Failure 95 compute. Host. Service. Failure 96 compute. Host. Service. Failure 52: 07. 6 55: 34. 2 57: 35. 1 52: 17. 2 Complete 55: 34. 2 Init 57: 35. 1 Init service nova-compute restart 172. 16. 101 [0 m service nova-compute status 172. 16. 10. 101

Resilience Improvements: Existing Script Implementation • Python, using Django Framework (web development, URLs for DB), IDE Py. Charm • Scripts are installed and executed on the ARM jump server 10. 50. 12: • Directory: ~/auto-env • executables in ~/auto-env/bin • scripts could also run from a VM or a Docker container • virtual environment managed from ~/auto-env/bin/activate • scripts interact by HTTP APIs with: • configuration and result (data records) storage tables (see next page) • test data records can be used for analysis • VIM: to simulate VI failures, etc. (e. g. Open. Stack) • VNF manager/orchestrator (MANO) (e. g. ONAP) • SQLite is used to browse the tables from the server CLI • e. g. to consult the test results • curl is used for HTTP or FTP transfers of these tables

ONAP-Auto Test Framework (link) 1 Test Overall Objectives Depends. On 2 Test Strategy 6 Depends. On 3 Enables Test Documentation Implements Test Analysis, Conclusions 4 Test Script Implementation Script language (Python, bash, …); Comments in script capture the test documentation Test Specific Objectives Depends. On Use Case Description “section” or “chapter” of a Test; could be just 1 UC; Assertion (Pass/Fail Criteria) Feature/Capability being tested, success/fail determination (binary pass/fail condition, quantitative thresholds/ranges, …) Executes 5 Test Runner (Calling script) Environment Description Physical Resources (unless Test aa. S) Physical servers (CPU, RAM), disks (HD, SSD), OS; pod organization; Is. Variant. Of Cloud & Virtual Resources Manager: Openstack, K 8 S, AWS, … Resources: compute (OS), storage, network, …; VMs or Containers; * Environment Variant VNF Manager VNFs VNF Lifecycle Events ONAP, Cloudify, Tacker, … v. XYZ (CPE, FW, Switch, HSS, I/S/P-CSCF, …); Source (OPNFV, Clear. Water, ETSI, …); Format (JSON, TOSCA, …) Onboard, Remove, Deploy/Activate, Terminate, Scale-In/Out, Monitor, … Follows (sequence) Pre-test State Description of required configuration, environment variables, … Action (Test Step) e. g. : setup, start, use, stop, (functions in a script language); execute planned VNF LC events Assertion Step Evaluation of current state, application of assertion criteria, determination of Pass/Fail result Post-test State Capture/Log of significant state data (environment variables, …) Cleanup Step Function to return system to its Pre -test State Follows (sequence)

Resilience Improvements Use Case: Overall Objectives • https: //wiki. opnfv. org/display/AUTO/Auto+Use+Cases#Auto. Use. Cases-Use. Case 2 • As an NFV edge service provider, I need to assess what degree of added VIM+NFVI platform resilience I obtain by leveraging ONAP closed-loop control, vs VIM+NFVI self-managed resilience, so I can determine the ROI for integrating ONAP with my VIM+NFVI platforms, both locally on the VIM+NFVI platform, or remotely. • assess VIM+NFVI resilience under specific stresses without ONAP support • assess VIM+NFVI resilience under specific stresses with ONAP support, i. e. VES+DCAE+CLAMP+specific policies to address the type of stress • stress the system via: • traffic: different types as needed per the resilience measurements • Instability: • physical infra failure (host, NIC, disk, . . . ) • virtual infra failure (cloud control plane components, SDNC components, NFVI components) • security threats • measure (and compare) resilience in terms of • • link failure, e. g. loss ping failures transaction failure, e. g. failed web/API requests service degradation/failure duration or recovery time scope of disruption GD: failure detection duration (time to notice/notify failure)

Resilience Improvements Use Case: Strategy Resilience Improvements Through ONAP [ID: Auto-UC-02] Physical Infrastructure Failure Server Failure, Migration [ID: auto-resiliency-pif-001] [JIRA: AUTO-9] Virtual Infrastructure Failure Compute Service Failure, (Auto)-Restoration or Migration [ID: auto-resiliency-vif-001] In progress [JIRA: AUTO-13] Security Failure Host Tampering, Fencing, Migration [ID: auto-resiliency-sec-001] [JIRA: AUTO-16] Disk Failure, Migration [ID: auto-resiliency-pif-002] [JIRA: AUTO-10] SDN-C Service Failure, (Auto)-Restoration or Migration [ID: auto-resiliency-vif-002] [JIRA: AUTO-14] Host Intrusion, Fencing, Migration [ID: auto-resiliency-sec-002] [JIRA: AUTO-17] Link Failure, Migration [ID: auto-resiliency-pif-003] [JIRA: AUTO-11] OVS Failure, (Auto)-Restoration or Migration [ID: auto-resiliency-vif-003] [JIRA: AUTO-15] Network Intrusion, Fencing [ID: auto-resiliency-sec-003] [JIRA: AUTO-18] NIC Failure, Migration [ID: auto-resiliency-pif-004] [JIRA: AUTO-12] Storage Service Failure, (Auto)-Restoration or Migration [ID: auto-resiliency-vif-xyz] [JIRA: xyz] Networking Service Failure, (Auto)-Restoration [ID: auto-resiliency-vif-xyz] [JIRA: xyz]

Resilience Improvements: Documentation (readiness) e 2 e Service Test Script VNF Virtual Resource (compute & storage) VNF Physical Server VNF Network: OPNFV OVS, neutron, . . . Cloud/Container management: K 8 S, Open. Stack, AWS, . . . [OPNFV] VES onboard and deploy test VNF(s) in environment, using ONAP SDC/VID and MSO (implied inventory updates in AAI) ONAP onboard & deploy: SDC, VID, AAI ONAP orchestration: MSO ONAP policies: Policy Framework ONAP control loops: CLAMP ONAP data correlation: DCAE onboard VNF as VSP, design e 2 e Service, approve, distribute notify orchestrate VNF (VSP, Service) deployment commands virtual switch is deployed VNF network is deployed e 2 e Service is deployed VNF is deployed (e 2 e) Service configuration of ONAP monitoring and automated actions VSP (Vendor Software Product) VNF manifest (text) VNF zip file (binary) VLM (Vendor License Model) setup VES to collect and push VNF data DCAE gets data from VES (through VES collector) setup VNF governing policies setup VNF control loops DCAE enforces policies and control loops setup : VNF monitoring (alerts, analytics), DCAE VES Collector ONAP controllers (m. VIM, SDNC, APPC, VFC)

Resilience Improvements: Documentation (auto-resiliency-vif-001) e 2 e Service Test Script VNF Physical Server VNF Virtual Resource (compute & storage) VNF Network: OPNFV OVS, neutron, . . . Cloud/Container management: K 8 S, Open. Stack, AWS, . . . [OPNFV] VES ONAP onboard & deploy: SDC, VID, AAI ONAP orchestration: MSO ONAP policies: Policy Framework ONAP control loops: CLAMP ONAP data correlation: DCAE ONAP controllers (m. VIM, SDNC, APPC, VFC) Compute [vif-001] [AUTO-13] simulate VM/Container failure get time T 1 start monitoring VNF, until recovery VNF raw operational data stream (direct collection from VI) VNF processed operational data stream from VES (data from controllers) (direct collection from VIM) determine VNF failure (alert, correlation); RCA: Compute Failure . . . Measured ONAPenabled automatic repair (Recovery Time estimator) (apply policies) notify MSO prepare and trigger Compute repair (restoration, migration) apply CLAMP control loops restore (migration) commands (SDN-C, APP-C) . . . restore (migration) commands (multi-VIM) detect VNF recovery post-repair verifications get time T 2: recovery ~ (T 2 -T 1) stop simulating VM/Container failure get VNF/Service back to its pre-test state (via ONAP) use MSO API (Create/Delete Service/VNF, . . . ) reset (migration) commands determine VNF is restored

Resilience Improvements: Test Data Design User Use Case Test Instance Test Definition Associated Metric Definition CLI Response API Response Metric Value Challenge Instance Recipient Instance CLI Command API Command VNF/Service Instance Challenge Definition Cloud Virtual Resource Instance Challenge Type Physical Resource Instance

Resilience Improvements: Test Data Design Table: Test Cases ID name (vif-001, etc. ) JIRA URL Table: Test Instances ID test. Def. ID user. ID # user or application who launched the test instance challenge. ID start. Time finish. Time ONAPBased. Restoration. Detected. Time # time when the VNF/service restoration was detected measured. Recovery. Time # (ONAPBased. Restoration. Detected. Time challenge. ID. start. Time) associated. Metrics. Values # append measured values log #append new lines for significant events CLIResponses APIResponses Data type: CLI Response CLIResponse #string Data type: API Response APIResponse #object Table: Test Definitions ID challenge. Def. ID test. Case. ID VNFIDs # most of the time, only 1 VNF associated. Metrics. IDs recipient. IDs #if interact with any mgt system, to monitor VNF test. CLICommand. Sent test. APICommand. Sent Table: Metric Definitions ID name info #free-form, description, formula method calculation(parameters) Data type: CLI Command CLICommand #string Data type: API Command APICommand #object Table: Metric Value measured. Value time. Stamp Table: Challenge Instances ID challenge. Def. ID start. Time stop. Time log #append new lines for significant events CLIResponses APIResponses Enum: Challenge. Type compute. Host. Failure disk. Failure link. Failure nic. Failure host. Tampering. . . Table: Recipient Instances ID name #ONAP, Op. Stck, K 8 S, . . . info #free-form version #optional access. IPAddress access. URL user. Name. Creds password. Creds key. Creds network. Info (VPN, VCN, . . . ). . . Table: Challenge Definitions ID challenge. Type recipient. ID impacted. Resources. Info # free-form impacted. Resource. ID start. Challenge. CLICommand. Sent # if CLI; can include hard-coded references to resources stop. Challenge. CLICommand. Sent # if CLI; to restore to normal start. Challenge. APIData. Structure. Sent # if API stop. Challenge. APIData. Structure. Sent # if API. . . VNF Orchestrator (ONAP, Tacker, Cloudify, ETSI OSM, . . . ) Table: VNF/Service Instances ID name #v. FW 1, v. LB 1, v. CPE 4, . . . info #free-form IP@ URL list associated. Resources #VMs, servers, . . . VNF/Service Container Table: Cloud Resource Virtual Instances ID name #VM 2, Vstorg 1, . . . info #free-form IP@ URL list associated. Resources #servers, . . . Table: Physical Resource Instances ID name #server 4, . . . info #free-form IP@ MAC@ VIM (Open. Stack, K 8 S, AWS, Azure, GCP, . . . ) VM (compute) Virtual Storage (Block, Object, FS, DB, . . . ) Network (subnet, VPN, VPC, . . . ) Physical Host (with CPU, RAM, OS) Physical Disk (HD, SSD, . . . ) Physical NIC (Eth, Wi. Fi, Cellular, . . . ) Physical Link (wire, radio) Logical Link (tunnel)

Resilience Improvements: Test Data Implementation (Python classes) Auto. Base. Object ID name File: Test. Cases. bin (pickle, with one List of Objects) Metric. Definition info printout_all() Metric. Value value timestamp, metric_def_ID Challenge. Execution challenge_def_ID start_time stop_time log CLI_responses API_responses write_to_csv() Test. Execution test_def_ID challenge_exec_ID user_ID start_time finish_time challenge_start_time restoration_detection_time recovery_time associated_metric_values log CLI_responses API_responses write_to_csv() Recovery. Time. Def compute (self, time_challenge_started, time_restoration_detected) Uptime. Percentage. Def compute (self, measured_uptime, reference_time, planned_downtime) Test. Definition challenge_def_ID test_case_ID VNF_ID_list associated_metrics_ID_list recipient_ID_list test_CLI_command_sent_list test_code_ID test_code_list run_test_code() test_code 001() test_code 002(). . . printout_all() . . . Def compute (self, . . . ) Time. Stamped. String. List __string_list __timestamp_list append_to_list(string) get_timestamped_strings() length() Time. Stamped. Metric. Value. List __metric_value_list append_to_list(Metric. Value) get_timestamped_metric_values_as_strings() length() Recipient info version_info access_IP_address access_URL username_creds password_creds key_creds network_info printout_all() Challenge. Definition challenge_type recipient_ID impacted_cloud_resources_info impacted_cloud_resource_ID_list impacted_phys_resources_info impacted_phys_resource_ID_list start_challenge_CLI_command_sent stop_challenge_CLI_command_sent start_challenge_API_command_sent stop_challenge_API_command_sent challenge_code_ID (start|stop)_challenge_code_list run_(start|stop)_challenge_code() (start|stop)_challenge_code 001() (start|stop)_challenge_code 002(). . . printout_all() output CSV file name format: chall. Def. Exec. AAA-yyyy-mm-dd-hh-mm-ss. csv test. Def. Exec. BBB-yyyy-mm-dd-hh-mm-ss. csv AAA: challenge definition ID BBB: test definition ID examples: chall. Def. Exec 001 -2018 -03 -05 -14 -35 -53. csv test. Def. Exec 001 -2018 -03 -05 -14 -35 -53. csv Physical. Resource info IP_address MAC_address printout_all() Cloud. Virtual. Resource info IP_address URL related_phys_rsrc_ID_list printout_all() VNFService info IP_address URL related_phys_rsrc_ID_list related_cloud_virt_rsrc_ID_list printout_all() Challenge. Type(Enum) COMPUTE_HOST_FAILURE = 100 DISK_FAILURE = 101 LINK_FAILURE = 102 NIC_FAILURE = 103 OVS_BRIDGE_FAILURE = 200 HOST_TAMPERING = 300 HOST_INTRUSION = 301 NETWORK_INTRUSION = 302 Test. Case JIRA_URL printout_all() List: [Test. Case 1, Test. Case 2, . . . ] Test. Case 1 ID: 1 name: auto-resiliency-pif-001 JIRA_URL: https: //jira. opnfv. org/browse/AUTO-9 Test. Case 2 ID: 2 name: auto-resiliency-pif-002 JIRA_URL: https: //jira. opnfv. org/browse/AUTO-10. . . List of all definition file names: Resources. Physical. bin Resources. Cloud. bin Resources. VNFServices. bin Recipients. bin Test. Cases. bin Definitions. Metrics. bin Definitions. Challenges. bin Definitions. Tests. bin Global variables: test_case_list test_definition_list recipient_list challenge_definition_list metric_definition_list physical_resource_list cloud_virtual_resource_list VNF_Service_list

Resilience Improvements: Test execution Logic Test. Definition challenge_def_ID test_case_ID VNF_ID_list associated_metrics_ID_list recipient_ID_list test_CLI_command_sent_list test_code_ID test_code_list run_test_code() test_code 001() test_code 002(). . . printout_all() Challenge. Definition challenge_type recipient_ID impacted_cloud_resources_info impacted_cloud_resource_ID_list impacted_phys_resources_info impacted_phys_resource_ID_list start_challenge_CLI_command_sent stop_challenge_CLI_command_sent start_challenge_API_command_sent stop_challenge_API_command_sent challenge_code_ID (start|stop)_challenge_code_list run_(start|stop)_challenge_code() (start|stop)_challenge_code 001() (start|stop)_challenge_code 002(). . . printout_all() run_test_code() logic create Tst. Exc: Test. Execution instance set Tst. Exc. start_time get Ch. Def: Challenge. Definition instance create Ch. Exc: Challenge. Execution instance call Ch. Def. run_start_challenge_code set Ch. Exc. start_time set Tst. Exc. challenge_start_time <specific code to wait for VNF recovery> <collect specific other metrics> set Tst. Exc. restoration_detection_time compute Tst. Exc. recovery_time call Ch. Def. run_stop_challenge_code set Ch. Exc. stop_time call Ch. Exc. write_to_csv() set Tst. Exc. finish_time call Tst. Exc. write_to_csv() (after each significant step, add entries to Tst. Exc log and to Ch. Exc log accordingly) Test. Execution test_def_ID challenge_exec_ID user_ID start_time finish_time challenge_start_time restoration_detection_time recovery_time associated_metric_values log CLI_responses API_responses write_to_csv() Challenge. Execution challenge_def_ID start_time stop_time log CLI_responses API_responses write_to_csv()

Resilience Improvements: Module design Module: Manage Test Definitions View/Edit/Delete: physical resources cloud resources VNFs recipients (OS, cloud/VNF managers) challenge definitions optional metrics test definitions Initialize Data (initial storage population) Module: Main Interactive Menu: View Test Definitions Edit Test Definitions Run A Test View Test Results Module: Global variables Persistent Test Data (Files, CSV, binary, DB) Module: Run Test Receive/retrieve chosen test def info pre-test (pings, etc. ) launch test: create execution instances of Test and Challenge simulate challenge get time T 1 loop: wait for VNF recovery optional other metrics store data and logs get time T 2 stop challenge reset (with ONAP MSO) store data and logs post-tests logs Module: Interface With Servers/OS CLI send/receive API send/receive Module: Interface With Cloud Managers CLI send/receive API send/receive Module: Interface With VNF Managers and NFV Orchestrators CLI send/receive API send/receive

Resilience Improvements: Test Runner (1) Ask for Test Definition ID Main Menu (2) Display Test Definition details (pull details from each related object) (3) Run currently selected Test Definition ID (4) Exit

Resilience Improvements: activities Arm VNFs find test VNF K 8 S/Arm installations Install (K 8 S/Docker)/Arm find test VNF OS/Arm Install Open. Stack/Arm Onboard VNFs Deploy VNFs Configure Policies Install ONAP/K 8 S/Arm Configure closed-loops Verify DCAE operations Install ONAP/K 8 S/x 86 Install (K 8 S/Docker)/x 86 find test VNF K 8 S/x 86 find test VNF OS/x 86 Install Open. Stack/x 86 installations x 86 VNFs Dependencies: - ONAP software * DCAE on K 8 S * Arm binary compatibility - VNF software * K 8 S containerized versions * Arm binary compatibility - Laa. S lab availability, capacity Analysis (top-down methodology): define Use Cases and Test Cases; define MSC (Message Sequence Charts) Analysis; definitions in pickle, results in CSV, inmemory classes and lists Gather Lab data: ONAP, OS, K 8 S, NFVI (Servers), VNFs, VM, Containers, . . . Analysis; metrics; list of CLI/API commands; list of Python functions Openstack: Evaluate options (CLI, HTTP, Python clients); validate openstacksdk K 8 S: Evaluate options (Python client) ONAP: Evaluate options (HTTP API; 1 per component) Servers (NFVI): Evaluate options (CLI) Test Case Data Management (classes and files) Test Case Data Initialization Test Case Logic (scripts) Test Case Software (Python 3) Push UC 2 code to Auto repo (Gerrit) Auto Use Cases (Test Case Scripts) UC 1: Edge Cloud UC 2: Resiliency Improvements UC 3: Enterprise v. CPE (. . . future U. C. , like v. IMS) System Interface (ONAP, VIMs, VNFI) User Interface (choose test case, view data, start execution, . . . ) OPNFV integration Auto library OPNFV Scenarios (os|k 8 s)-(nosdn). . . Auto Installation Scripts K 8 S, Docker ONAP by OOM Openstack VNFs by ONAP API (SDC) OPNFV Installers (apex, fuel, compass 4 nfv, . . . ) OPNFV Test frameworks (Functest, Yardstick) Traffic Generators ONAP Configuration Test Case Execution T. C. Results Collection OPNFV Laa. S (Pharos) Push Auto code to Auto repo (Gerrit) OPNFV Jenkins Daily CI/CD OPNFV Docs Document Design (ppt) Document User Guide (RST) Push Design data to Auto Wiki Document Release Notes (RST) Push RST to OPNFV docs (Gerrit) Document Installation (RST) Researching Installing Configuring Coding Documenting Posting

APIs used by Test Case code Strategy: use interface with highest abstraction level available, avoid CLI parsing whenever possible • openstack SDK for Open. Stack • Requests for ONAP • Python client for Kubernetes • Subprocess for physical servers (unless there is an API or a client) OS Test Case Python Code os. system subprocess. call (HTTP Session object: (“cmd params”) ([‘cmd’, ’params’]) GET, POST, . . . ) os lib (openrc) permissions subprocess lib system calls unified OSC (Open. Stack Client) CLI (used to be per-service CLIs) unified OSC Python Client “python-openstackclient” (used to be per-service clients) HTTPS Physical server Requests lib (clouds. yaml) permissions (Connection object/interface) openstack SDK lib (used to be per-service modules) Service Abstraction layer (proxies: Compute, Object. Store, Network, . . . ) Keystone Nova Cinder Neutron Glance . . . Services Open. Stack Controller Node K 8 S CLI (kubectl, kubeadm) HTTPS Service REST APIs Keystone Nova Cinder API API Neutron Glance. . . API AMQP (Advanced Message Queueing Protocol) HTTPS ONAP CLI HTTPS JSON, . . . (client. <method()>) kubernetes python lib Service (~REST) APIs AAI MSO DCAE API API Policy CLAMP. . . API Open. Stack Compute Node. . . Service REST APIs Common Services (MSB, AAF, DMaa. P, . . . ) AAI MSO DCAE Policy CLAMP . . . Services ONAP platform Kubernetes platform

Resilience Improvements: (M)SO API Create VNF Create service instance URI: {server. Root}/service. Instances/v 2/{service. Instance. Id} /vnfs Operation Type: POST URI: {server. Root}/service. Instances/v 2 Operation Type: POST request. Details model. Info subscriber. Info request. Parameters Details at: http: //onap. readthedocs. io/en/latest/submodules/so. git/docs/SO_R 1_Interface. html#so-apis Model. Type Model. Invariant. Id Model. Name. Version. Id Model. Name Model. Version Model. Customization. Name request. Details model. Info cloud. Configuration request. Info related. Instance. List request. Parameters Global. Subscriber. Id Subscriber. Name Instance. Name Product. Family. Id Source Suppress. Rollback Subscription. Service. Type User. Params Model. Type Model. Invariant. Id Model. Name. Version. Id Model. Name Model. Version Model. Customization. Name URI: {server. Root}/service. Instances/v 2/{service. Instance. Id} /vnfs/{vnf. Instance. Id} /vf. Modules Operation Type: POST Instance. Name Product. Family. Id Source Suppress. Rollback Delete VF Module URI: {server. Root}/service. Instances/v 2/{service. Instance. Id} /vnfs/{vnf. Instance. Id} /vf. Modules/{vfmoduleinstance-id} Operation Type: DELETE instance. Id model. Info User. Params Create Network name value Delete service instance URI: {server. Root}/service. Instances/v 2 /{service. Instance. Id} Operation Type: DELETE Delete Volume Group URI: {server. Root}/service. Instances/v 2/{service. Instance. Id} /vnfs/{vnf. Instance. Id} /volume. Groups/{volume-groupinstance-id} Operation Type: DELETE Create VF Module lcp. Cloud. Region. Id tenant. Id name value Create Volume Group URI: {server. Root}/service. Instances/v 2/{service. Instance. Id} /vnfs/{vnf. Instance. Id} /volume. Groups Operation Type: POST Delete VNF URI: {server. Root}/service. Instances/v 2/{service. Instance. Id} /vnfs/{vnf. Instance. Id} Operation Type: DELETE URI: {server. Root}/service. Instances/v 2 /{service. Instance. Id} /networks Operation Type: POST Delete Network URI: {server. Root}/service. Instances/v 2/{service. Instance. Id} /networks/{network. Instance. Id} Operation Type: DELETE GET Orchestration Request URI: {server. Root}/orchestration. Requests/v 2/{request-id} Operation Type: GET Orchestration Requests URI: {server. Root}/orchestration. Requests/v 2 Operation Type: GET

File organization: source code, documentation OPNFV @ Github (github. com/opnfv) Github auto remote repository link Github opnfvdocs remote repository link OPNFV demo documentation (readthedocs. io) link OPNFV Git opnfvdocs remote repository link OPNFV documentation (docs. opnfv. org) link (mirror) OPNFV Git (git. opnfv. org) OPNFV Git auto remote repository link CI (Jenkins) review process (voting) (git submodule integration) git push OPNFV Gerrit (gerrit. opnfv. org) Gerrit auto remote repository home_link, changes_link git clone/pull/fetch, git review (-d) Dev PC (Windows) Test Script Code (. PY, . SH, . . . ) Gerrit opnfvdocs remote repository home_link git add, git commit ~/auto/docs/(relea se/. . . ) Design, Development Use Case OPNFV Documentation (. RST) Edits (possible separate file backup) ONAP UNH IOL (Linux, servers, VMs) VNF (as ONAP e 2 e Service) Compute, Storage, Network Container VM Kubernetes Open. Stack Docker Physical Server Auto Wiki (Confluence) (wiki. opnfv. org) link OPNFV platform CI (build. opnfv. org) (references to pods) link git clone/pull/fetch, git review (-d) ~/auto/lib/auto/tes tcase/resiliency/. . . local Gerrit repository git push Test Script Code (. PY, . SH, . . . ) Test Definition Data (. bin) Test Execution Results (. CSV) Use Case and Test Case Documentation (PPT, WORD) Analysis (e. g. with Excel) Test Execution Results (. CSV)