ONAP CD using OOM Kubernetes Michael OBrien Amdocs

ONAP CD using OOM Kubernetes Michael O’Brien | Amdocs Dec 12, 2017

Goals • Why E 2 E Automation Pipeline - Triggered by merge or hourly - Reference of immediate regression to source - Historical tracking of healtcheck, perf, defects • ONAP integrity through full deployment integration 2

Agenda • Introduction (architecture, Dev. Ops) • Continuous Deployment Environment • CD use cases, rest calls, v. FW automation? • OOM Deployment automation • Live demo (jenkins, CD spot, Kibana analytics) • Upcoming (EC 2 Cloud. Formation, Rancher automation) • References 3

Devops • Jenkins CI • Jenkins CD • CD server (Q yesterday from Jing Wang of DCAE – short regression • https: //wiki. onap. org/pages/viewpage. action? page. Id=16002279 • test of single component • Kibana • Notifications pending 4

Walkthroughs • Build server • http: //jenkins. onap. info/job/oom-cd/ • Deployment server • http: //amsterdam. onap. info: 8880 • Analytics server • http: //kibana. onap. info: 5601 5

Kibana Deployment Health Tracking • https: //wiki. onap. org/display/DW/Auto+Continuous+Deployment+via+J enkins+and+Kibana • 2 scenarios that came up • SDC, SDNC, APPC robot healhcheck is problematic for a while after startup (likely also involves our readiness check) – verify all team supplied healthchecks reflect state of microservice • AAI Chef repo config change caught us off guard because a of mixed master/amsterdam yaml setting we missed at OOM 6

CD Helping with build issues via deployments cont… • Discuss AAI issue this Friday that exposed a git pull to master from the R 1 branch in OOM – fixed in 4 parts • CD job detects • Investigate – inform onap-discuss community – later trigger/callback • Fix • Test locally • Commit • CD job verifies • 8 hour turnaround (finished 2300) 7

SDC/SDNC/APPC • Waiting until liveness is ready 8

Healthcheck Dashboard – Shane Daniel 9

Friday was a good test of the OOM CD system • https: //gerrit. onap. org/r/#/c/25955/ • https: //lists. onap. org/pipermail/onap-discuss/2017 December/006859. html • Walkthrough this 10

CD detection of broken deployments (purple for 8 h) 11

Live Demo 12

Jenkins Configuration 13

Note about RAM, v. CPUs, HD usage in OOM • Check the K 8 S console for peak, idle requirements • http: //amsterdam. onap. info: 8880/r/projects/1 a 7/kubernetesdashboard: 9090/#!/pod? namespace=onap-aai 14

CLI bootstrap of CD server • In works https: //wiki. onap. org/display/DW/ONAP+on+Amazon+EC 2 15

Migrate CD job/server/kibana to LF and Openlab • In works – we have a jenkins_lf vpn account • Working with Intel and the LF 16

Recorded Demo of CD process • https: //wiki. onap. org/display/DW/Auto+Continuous+Deployment+via+J enkins+and+Kibana#Auto. Continuous. Deploymentvia. Jenkinsand. Kiban a-CDDemo. Videos 17

Questions 18