ISTIO ENVOY CNCF project Actively being done as

ISTIO & ENVOY (CNCF project) Actively being done as part of MSB in R 3/R 4.

Background • ISTIO overview • ISTIO advantages • ISTIO Challenges for ONAP • Mitigation plans • Phases • Backup - Security overview - Choices of using AAF for RBAC. 2

Micro Service Architecture - Background Internet SLB/ADC/API-Gateway ES EM EM IM 1 IM 2 IM 4 Image IM 3 IM 5 IM 1 IM 2 IM 4 Image Monolithic Application IS 1 IM 3 IM 5 ES IS 1 IS 2 IS 4 IS 5 IS 3 Micro Service based In monolithic application, entire application is scaled-out, waste of memory, no rolling upgrades, massive integration time, big maintenance. Micro Service architecture: • Each individual module is a micro service • Each module can scale-out independently • Typically each micro service is implemented as container 3

Micro Service Architecture - Challenges • Need for load balancing among internal hierarchy Internet • • • Special load balancers – Expensive Each consumer balancing its connections across producers. Independent LB decision. May load one instance of producers. Special logic at each service. • Need for discovery of service instances SLB/ADC/API-Gateway • Special logic at the each service. • Need for health checks ES IS 1 • Need for tracing/visibility IS 2 IS 3 • IS 4 IS 5 Special logic at each service • Need for network firewall capability • IS 4 Special logic at each service • Authenticate and Authorize consumers • Special logic at each service level • Need for Mutual TLS among producers and consumers. • Special logic to get certificate enrolled. • Special logic to initiate/accept TLS connections 4

Goal – Eliminate Micro service architecture tax Let application service developers not worry about micro-service tax Internet SLB/ADC/API-Gateway ES IS 1 - ES - IS 1 IS 2 IS 4 IS 5 IS 3 Not worry about Service discovery/registration Not worry about security (Secure communication & Certificate enrolment) 5

Micro Service Architecture – Service Mesh technology Internet Istio-ingress envoy IS 1 ES ES ES SC SC ISTIO Control Plane Pilot/Mixer/ CA SC SC SC IS 4 IS 5 SC IS 2 SC IS 3 SC (Side Car) – One for each Service instance (A container in each POD) - Envoy proxy - Service discovery, Load balancing, Failure handling, Circuit breaking (Limits), Fault injection (for troubleshooting), Health checks - Mutual TLS - Ingress control for reaching any service from external end. ISTIO Control plane: - Pilot: - Service discovery glue between Envoy and K 8 S - Traffic rule configuration - Security - Role based Access control (Pluggable RBAC engine, support for local RBAC, K 8 S RBAC adapters, but facility to add new adapters – Example AAF RBAC adapter) - Certificate Authority - Auto injection of side car (No changes required in Helm charts) 6

ONAP - Non-ISTIO to ISTIO (Advantages) ONAP SDC SO OOF SDNC APPC MUSIC AAF • • Certificate Enrolment client Authentication & Authorization Service Discovery/registration TLS, Visibility etc… Non ISTIO DCAE MC VFC POLICY DB ONAP SDC SO OOF SDNC APPC MUSIC ISTIO with Envoy (CNCF) DCAE MC VFC POLICY DB AAF • High productivity • No additional development by projects • No certificate enrollment by apps • No RBAC by apps • No SSL/TLS by apps. 7

ISTIO Challenges Huge Memory overhead: - Each application container will have associated side car container. - Each Side car can occupy 100 to 300 Mbytes of memory. - If ONAP has 200 containers, then ISTIO overhead is 20 Gbytes at the minimum. Higher latency of inter-service traffic - HTTP(S) packets traverse through two additional proxies (Envoy). - Typically around 100 msec of latency get introduced (Since there are no services are data plane related, this may be okay? ) Security of keys • ISTIO CA private keys are not secured (clear in disk and clear in memory) • Envoy user certificate keys are also not secured 8

Mitigations – Reduce memory overhead K 8 S POD App container SC (Envoy) container Proposed activities • Make envoy work in the existing container Reduce container overhead (from hundreds of Mbytes to few Mbytes). • Enable KSM (Kernel same page merging) to reduce the memory overhead resulting from duplicate code pages. K 8 S POD Container APP Envoy 9

Mitigations – Reduce latency or eliminate latency K 8 S POD App container SC (Envoy) container Proposed activities • Use Envoy for all other opeations, except for TLS • Avoid proxying • Let application terminate • Critical micro services can use this option. K 8 S POD Container Low latency APP Envoy TLS 10

HW Security ISTIO CA Go Crypto 11 (PKCS 11 Glue) • Secure CA private key • Distribute CA private key securely to the Hardware (TPM/SGX) – Started conversations with ISITO security group. Envoy proxy Open. SSL Lib. P 11 Go PKCS 11 library SSHSM TPM /SGX/ plugins TPM /SGX plugins Hardware • Secure user cert private key in Envoy Activities - Enhance ISTIO CA to use PKCS 11 based key instead of local key - Leverage key distribution work done in R 3 - Use Open. SSL instead of boring. SSL.

Proposal • Phase 1 - Prove ISTIO in ONAP context - Status: MSB is successful in proving this in R 3. • Phase 2 (Proposal for R 4) - Reduce the memory footprint by making envoy as part of application container - Secure CA private keys - Increase performance of proxy by leveraging crypto accelerators (such as QAT) • Phase 3 - Reduce latency overhead by giving choice to applications to originate/terminate the SSL/TLS connections. - Secure user certificate private keys 12

Inter working with AAF – Proposals

Proposals • Proposal 1: - Projects using AAF directly - Projects using ISTIO - Interoperation among those two buckets. - Proposal 2: - All projects using ISTIO - Provide an option a) Usage ISTIO CA and ISTIO RBAC, b) Integrate ISTIO with AAF RBAC & AAF CA Recommendation after Beijing developer conference: Proposal 2. 14

Proposal 1 Admin or external systems such as OSS/BSS API Router ONAP Services that don’t use ISTIO S 2 (TLS + CADI) S 4 (TLS + CADI) AAF ONAP Services that use ISTIO Ingress S 1 Side car S 3 ISTIO (Control) • Two buckets • Bucket 1 : that use CADI to get the certificates and authorize using AAF. • Bucket 2: that use ISTIO for certificate enrollment and authorization • ISTIO ingress is used to redirect the inbound connections from bucket 1 to bucket 2 AND outbound connections from bucket 2 to bucket 1 15

Proposal 1 - flow 1 Root/Intermediate CA Key Generation Station (Secure system) 2 2 ISTIO CA AAF CA ISTIO Mixer 5 S 1 AAF RBAC 3 3 Side car AAF RBAC Plugin 6 5 CADI 4 S 2 7 No implication (based on current understanding) on existing services that are using CADI for authentication & authorization Additional RBAC plugin development at the ISTIO is required 1. ONAP Administrator creates CA key pair and selfsigned CA certificate (or it could be CA key pair and signed by root/intermediate CA). Create two CA key pairs and certificate (via root/ICA) 2. ONAP Administrator using CA CLI/GUI, upload the CA private key/certificate and any chain in both ISTIO CA instances and AAF CA instances. 3. When S 1 or S 2 is brought up by OOM, corresponding CA will issue service certificates. Services get its own key/certificate & CA chain. 4. TLS establishment : Based on peer ID, service (or its side car) authenticates using CA certificate it has. 5. Authorization: When there is HTTP operation, its method, URL and other operation specific information, requester ID is sent to the RBAC engine to figure out the roles and permissions. 6. RBAC plugin talks to AAF RBAC to get the permissions (in case of ISTIO) 7. If permissions are allowed for that operation, operation proceeds. Otherwise, operation is denied. 16

Proposal 1 – Challenges • Communication between Istio and AAF buckets - Requires special routing rules in Istio for it to work - Services outside Istio will need to use context sensitive service URLs • For services within Istio, they will use the istio-ingress IP instead of service name • For services outside Istio, they will just use the service IP as before • Each service outside the Istio mesh needs to have access to this global context - Services outside Istio need to handle their own m. TLS termination if they are connecting to https services - Services within Istio will need to be configured with appropriate route rules to talk to external services 17

Proposal 2 (a) – All ISTIO Admin or external systems such as OSS/BSS ISTIO ingress S 1 Side car S 2 Side car S 3 S 4 Side car S 4 ISTIO (Control) • All ONAP services moving to ISTIO at once. • No ONAP service need to implement Mutual TLS and certificate enrolment. • Any ONAP service that enabled AAF CA enrolment and CADI integration for authorization needs to disable that code as side car will take care of certificate enrollment, mutual TLS and authorization • Works for any language based containers. 18

Proposal 2 (b) – ISTIO & AAF Admin or external systems such as OSS/BSS ISTIO ingress S 1 Side car S 2 Side car ISTIO (Control) Mixer AAF RBAC plugin S 3 Side car S 3 S 4 Side car S 4 AAF Citadel • All ONAP services moving to ISTIO at once. • No ONAP service need to implement Mutual TLS and certificate enrolment. • Any ONAP service that enabled AAF CA enrolment and CADI integration for authorization needs to disable that code as side car will take care of certificate enrollment, mutual TLS and authorization • Works for any language based containers. • AAF RBAC and AAF CA can be leveraged AAF CA plugin 19

Proposal 2 - Challenges • Istio needs to be proven to ensure that all ONAP protocols work. • Istio adds a sidecar for each POD which adds to the memory and CPU utilization requirements for ONAP • To use AAF RBAC, an additional plugin will need to be developed and integrated into Istio. • AAF CA will have to be integrated with Istio CA (https: //istio. io/docs/tasks/security/plugin-ca-cert/) 20

Recommendation from Beijing developer conference • Proposal 2(a) to start with (in R 4) • Proposal 2(b) in future releases (in R 5) 21

Service to Envoy proxy – Security Concerns & Mitigation K 8 S POD 1 K 8 S POD 2 Name space 2 SVC 1 Instance 1 Envoy Name space 1 IP Tables Tproxy TLS SVC 2 Instance 1 Name space 1 IP tables tproxy K 8 S POD 2 Name space 2 Linux Kernel ISTIO Pilot configures IP Tables to redirect the traffic coming from services to Envoy and Envoy to services. No configuration required at the service level to use Envoy as proxy Envoy acts as transparent proxy. Ensure that Source IP of the connection is maintained on both sides. Even though traffic is clear between instance and side car, it is not considered a security issue as both side car and service belong to same POD, hence same network name space. Clear traffic is never seen on the wire.

ISTIO CA - Features • Can generate self signed CA root certificate • Can take externally generated CA certificate and private key • Support for certificate chain (CA certificate signed by Intermediate CA) • CA redundancy • Two types of user certificate issue methods • Native K 8 S based for K 8 S orchestrated services • Non-K 8 S orchestrated services (VMs/Baremetal) • Certificate renewal • SPIFFE naming convention 1. Create Service account and Deploy service K 8 S Master 2. POD deploy ment event ISTIO CA 4. Send key/cert using secret mount SVC ISTIO CA 3. Generate Cert, Sign using CA key Envoy: 2. Send CSR - Auto reload of renewed certificates/keys 4. Send Cert - Integration with PKCS 11 based security Node agent - Authentication – Mutual TLS 1. Key pair generation and VM/Baremetal - Authorization - ISTIO RBAC or adapters to CSR remote RBAC SC 3. Create user key/certificate (signed by CA cert)

ONAP – ISTIO work • Rancher to instantiate ISTIO software components • Status : Currently manual installation of ISTIO components is done. • Migration path from NON-ISTIO to ISTIO • ISTIO Side car injection related changes • Securing the private keys of ISTIO CA and Envoy private keys. • Provide option to use either ISTIO RBAC and AAF CA RBAC – A way to choose the option (AAF CA RBAC is Stretch goal) • Any glue/tools to simplify the ISTIO deployment Any glue logic and scripts – Put it as part of MSB project. Goal – Entire ONAP using ISTIO all the time (No option to choose). Fix any gaps to make that happen.

HW Security ISTIO CA Go Crypto 11 (PKCS 11 Glue) • Secure CA private key • Distribute CA private key securely to the Hardware (TPM/SGX) – Started conversations with ISITO security group. Envoy proxy Open. SSL Lib. P 11 Go PKCS 11 library SSHSM TPM /SGX/ plugins TPM /SGX plugins Hardware • Secure user cert private key in Envoy Activities - Enhance ISTIO CA to use PKCS 11 based key instead of local key - Leverage key distribution work done in R 3 - Use Open. SSL instead of boring. SSL.

s