Decisions Action Items Wrap up 512020 edgexfoundry org

Decisions, Action Items, Wrap up 5/1/2020 edgexfoundry. org | @edgexfoundry

Business Topics • Edge. X Organization 2. 0 • Proposal generally accepted with following changes: • Outreach Committee chair should be voting TSC member • Vertical Solutions project should be more broad / general at first and seek vertical specialty as the organization grows • Action: next steps • Jim to write up the complete proposal for review ü This has been made available on the wiki and on Slack • TSC to review and provide comments ahead of May 6 TSC meeting • Hold TSC vote to adopt in total unless there are comments / issues on part(s) • Endorsement (aka Certification) Process Proposal • As presented by Kristen Call, Mike, Lenny, Camilo and Rodney • Action: to continue to be defined/refined via certification working group (Rodney) edgexfoundry. org | @edgexfoundry

Architecture Topics • Secrets for message bus connection (expanded to secrets for any 3 rd party software) • Issue: • We don’t want a separate docker image for creating database, message bus, etc. secrets. We would prefer to have all secrets generation as part of security secret store setup (adding trigger for new secret generation there). • Decision: • How do we generate/inject the random passwords into vault? • We need an abstraction and pluggable mechanism to provide generation of secrets (current Go Key mod is not fit for production) –security WG to take this as first high • How do we consume them in the service? • It depends on 3 rd party software - needs a shim piece of software to provide • How the “shim” works in order of preference • 1 - Shim talks to Vault directly to consume secrets (process to process via Vault) • Slight alternate: piece of code to talk to Vault directly and then injects 3 rd party software with secrets • 2 - Use injected env var (which are per process) as a way to pass the secret to service consuming it • Shim talks to Vault, sets and env var and then launches service • 3 - Secrets file on RAM disk (or other protected location other than Docker volumes; this could be Swarm or Kubernetes secrets – temporary file system that priority task – what is the ref impl going to be. • doesn’t write to disk; have to consider reboot circumstances and how to handle re-injecting these) • 4 - Command line argument as a way to pass the secret to service How do we consume them from a client perspective? – here we are good; we get them from Vault • Approach is encoded in go-mod-secrets • Other languages (C for DS) needs an equivalent of go-mod-secrets • Additional action item: • Colin Hutchinson (Hashicorp) to explore Consul Templates to see if we might be able to take better advantage of Vault and Consul Templates for service consuming of secrets, but not sure how that would work with 3 rd party systems (like Redis). edgexfoundry. org | @edgexfoundry

Distribution of DS (security between DS and Core) • How do we secure the communications between device services and core services when DS is on different host? • Decision: Security WG to provide “HOW-TO” guides for following in priority order as part of Hanoi release: 1. SSH Tunnel (needs document and docker compose file snippets and may need some special containers for ssh/sshd or instructions for their setup) 2. Overlay network (needs document and docker compose file examples/snippets) 3. Service mesh (needs document and script on how to automate) • Number of guides provided for Hanoi TBD based on other work • Additional action item: Colin and Tingyu to visit (revisit) Raw IPTables + existing Kong implementation that Tingyu documented last year. edgexfoundry. org | @edgexfoundry

Supporting Metrics & Handling Control Plane Events • How to collect Edge. X specific metrics (like how many Event messages are being processed) and handle control plane events (like a new device just got onboarded)? • This is not part of support notifications (which is about sending a notification out of Edge. X to some 3 rd party about something) • Decision: • Metrics collection • All services will use/integrate go-mod-messaging to facilitate sending metrics object messages to a configurable topic • A message structure will have to be created for this metric data (different from event/reading) • All services will have configuration (which goes in Consul) to say which metrics they provide and the option to turn those metric collections on/off as well as define how often they get collected • Application services will be constructed to optionally receive these messages and act on them (the reference implementation receiver in Edge. X) but users can attach their own receiver to the topic • Control plane event handling • All services will use/integrate go-mod-messaging to facilitate sending control plane events to a configurable topic • A structure will have to be created for this control data event (different from event/reading) • All services will have configuration (which goes in Consul) to say which metrics they provide and the option to turn those metric collections on/off as well as define how often they get collected • Application services will be constructed to optionally receive these messages and act on them (the reference implementation receiver in Edge. X) but users can attach their own receiver to the topic • This is a feature for Ireland or later (not Hanoi), but ADR 0006 to lay it out is the goal for Hanoi (under system management) Both collections/handling are off by default. 0 MQ is not an |option@edgexfoundry since you have multiple publishers edgexfoundry. org

Automatic migration of devices between device services • How would a mobile device migrate from one DS to another? • Options outlined here • Decision: • Deferred for more discussion after Hanoi. • We should come up with real use cases – what sensor, what type of application, etc. • Ancillary discussion items for future discussion: • Profile definition – how is it shared across device services • Right now profiles are attached to a device service and you want profiles to be shared by all DS? ? **JIM to check • How do we name multiple device services that are the same? edgexfoundry. org | @edgexfoundry

Should Edge. X consider offering more deployment / orchestration options? • Currently offer Docker Compose files & Ubuntu Snaps • Decision: • We should (continue to) facilitate as much as possible but not depend on or choose any one solution. • Go for low hanging fruit – what can we do for the least effort and get the most adoption • What we do for K 8 s would help with Swarm or other alternatives • Leverage what OH is doing with Operators (the “new and improved” Helm). • Action: Jim to work with Joe to get Edge. X involved when OH explores Operators edgexfoundry. org | @edgexfoundry

Should Edge. X have more/closer ties to Akraino Blueprints • Upon success with ELIOT blueprint and successful testing of Edge. X in Akraino’s lab with ELIOT stack, should we look for more similar opportunities? • General: should Edge. X look for closer ties to LF Edge projects • There is a general feeling that little benefits are coming from LF Edge association • There is also a sentiment that in order to get benefits, we must lead in the efforts to create the successes • Decision: • See if Akraino works to meet us at least half way to expand our work with them • See if Akraino/LF Edge picks up on our idea to host “click to get” Edge. X on ELIOT stack in the Akraino/UNH lab environment for developers/users to experiment with • This could help/support our certification/endorsement work • Look for mutually beneficial project-to-project opportunities, but only if they are balanced and demonstrate opportunity for true ROI to Edge. X edgexfoundry. org | @edgexfoundry

How does the broader community participate in test automation with the new TAF framework? • While there is a desire not to write the same test twice, the TAF tools/framework don’t lend themselves to developer participation and blackbox/API tests should be done from an independent vantage point. • Counter arguments include an issue with scale/visibility if developers are not involved • Decision: • Unit tests are developer responsibility • Blackbox (API Tests), Integration, Backward comp, Perf tests are the responsibility of Test QA • The community recognizes that with V 2 API & TAF testing work done by IOTech Taiwan resources and without additional help, the efforts will spread to Ireland beyond • There is an opportunity to put some testing into the PR process (periodically if not after each PR) • Action: Lisa R to explore edgexfoundry. org | @edgexfoundry

Edge. X Awards • Innovation • Contribution • Michael Estrin – Dell • James Gregg - Intel edgexfoundry. org | @edgexfoundry • Bryon Nevis – Intel • Lisa Ranjbar - Intel

Project Cadence • April & Oct remain target release months • F 2 F about 1 month in advance of release to allow for some community coding opportunities • Still planning a F 2 F in Oct/Nov time frame. Intel China has extended an invitation • • Hanoi release – Oct 2020 Ireland release – April 2021 Jakarta release – Oct 2021 ‘Kamakura’ release – April 2022 (as named by Tony Espy) • Conferences goals • At least 2 x large marketing/promotional events (Hannover Messe, Io. T SWC) • At least 1 x developer focused event • Still to be determined; no changes or suggested event changes yet. • Current global situation requires more time to wait and see. • Action: Marketing group to explore: • How do we channel funds for shows to new endeavors? • What should virtual venues/events be? edgexfoundry. org | @edgexfoundry

Geneva Release • Outstanding PR’s for Geneva • • • Snaps – has bulk of changes; 1 PR to update Kong Mongo Deprecation; edgex-go to use go-mod-messaging latest Dead code from config seed removal Swagger file edgex-go PR – 2532 Andre’s PRs? ? (JIM) • Known Bugs Still Being Worked • Issue #2527 – endpoint on event (core data); list with unlimited readings can be big and crash core data • • Exploring setting max response size Was there an older issue in Fuji with Mongo on this? • Issue# 190 - Restriction of ports on localhost in compose file • Go-mod-bootstrap PR pending merge • Needs to be worked through using services • Couple of failed test cases • Issue #2521 - edgex-go – core data (unstable situation; doesn’t fail always) • Issue #2524 – logs when support notification not running • Support scheduler – crashes when remove schedule • #442 (blackbox test repository) – security enablement test on ARM system (time out error? ? ) • Could Redis run out of memory in long running situations? Jim action to find out • Upcoming core work group discussion edgexfoundry. org | @edgexfoundry

General • Version 1. 3 vs 2. 0 release • This release shall have no non-backward compatible changes • Including changes that qualify under “Lenny rule” (changed config, but add config ok) • No LTS target defined at this time • Most important features we are currently working on (in priority order) • • V 2 API (partially deliver in Hanoi) Allow device services to be distributed to alternate hosts Implement a message bus between device services and application services Agreed that we should try to have one cycle where V 2 API is beta edgexfoundry. org | @edgexfoundry

General Scope • Move to Go 1. 15 when available in August • Race conditions would be fixed in 1. 15; appears not to be big lift/impact • Backup plan is to go with 1. 14 should 1. 15 slip much • Design a message bus implementation between device services and application services • Stretch goal to provide implementation • Uses go-mod-messaging (which uses which ever bus setup under it) • Concerns to work out through design: • What about check sum for binary readings? Which is done by core today • 0 mq doesn’t work for this model • Crawl implementation may be to just have DS send to core and appl service via pipe • We wish to come out of any crawl step implementation with no tech debt • Allow device services to be distributed to alternate hosts • Dependent on Security options in scope edgexfoundry. org | @edgexfoundry

General Scope • V 2 API - “experimental” • Security not ready for V 2 API by Hanoi (Option B security) • Core, device services, application services, and SDKs V 2 API minus security get done • Supporting services may remain at V 1 (indefinitely? ? ) • North and south bound services are released separately so can go longer • Once clients are done a majority of the work is done for north/south • Stretch: explore possible collapse of core command metadata • Stretch: deprecate Mongo • Resource determination mid-stream • Removal of all code, references to Mongo (ex: compose), and pull Mongo artifact images • Critical - CVE issues are mostly Mongo based so it would be good to get rid of it • Add something into release notes about CVE issues with Mongo (Geneva) edgexfoundry. org | @edgexfoundry

Core and Supporting Services • V 2 API for core services • Stretch: dropping the log service • Low priority • Any UI concerns (Jim to follow up with Gavin) • Have UI team work to remove any logging service reference edgexfoundry. org | @edgexfoundry

Edge. X UI – Core Subproject • Establish a roadmap for the UI • Determine if UI should be in separate repo • Does the UI need secret management of any sort? ? ? • Need privilege to set secrets • Authentication/authorization from UI to services • We have an endpoint to create secrets specific to app services (each has its secrets) • Incorporation of new rules engine • We should have use cases • Creation of rules, edit of rules, etc. • Work requirements through Appl Services edgexfoundry. org | @edgexfoundry

Test. QA • Edge. X V 2 API Blackbox tests (TAF) • Stretch: User guidance on platform needs (want it badly but not under current V 2 API and TAF work) • More performance statistics • # of devices/per recommendations • Providing Edge. X users guidance on platform needs, sensor data throughput and deployment based on performance metrics. Specifically, with the Geneva performance testing apparatus, the Edge. X community will be able to answer these questions for the user: • Will Edge. X fit on my system? - size of Edge. X services, infrastructure, etc. and hardware/platform requirements • What is the speed of data through the system? - from device service sensor data ingestion to the rules engine and back down through command to another device service to trigger a put command, how long does this take? • How many “things” can be processed at a time? – with caveats on the type of thing, type of data, etc. • These questions need to be answered on real hardware (both Intel and ARM) edgexfoundry. org | @edgexfoundry

Device Service & SDKs • Add V 2 API • Message bus (DS to appl service; design first and stretch impl) • Protect the device from harmful commands, there should be the possibility to set a Min and Max limit (or other profile checks to protect the device). • There is a preexisting issue for this • Data filter design between DS and Core Data • Provide a design about how to implement this before implementing. • If possible, can the filter functions be shared across App Services and D. S. (w/ App WG) • Stretch: design bound checking (lead an ADR proposal) • • • Number of operations that can be done Max request size (that lends to Do. S, etc. ) Could be more globally applied – a REST Qo. S Each WG should explore implications and design Articulate the problem better; limit scope a bit; design target for Hanoi • Action: what is our use case for array of types? Jim to get with Steve O and Rodney (geo cords? ) edgexfoundry. org | @edgexfoundry

Application Services and SDK • Add V 2 API • Message bus (DS to appl service; design first and stretch impl) • Deprecate Drools rules engine • Compose file/archive repos/remove artifacts • Design metadata about the “gateway” or host platform (identity, location, …) • ADR with typical use case definitions for Hanoi/labelling or tagging edgexfoundry. org | @edgexfoundry

Kuiper Rules Engine – Appl Serv subproject • Kuiper road map review; explore additional requests • Documentation additions • Need 3 -4 use cases (ex: how to send commands with Kuiper or how to check for values coming in) • Kuiper engine should not barf on binary data and should log info / warning edgexfoundry. org | @edgexfoundry

System Management • Service list fix • SMA errors when supporting service is not running • Both the SMA and the security-proxy-setup service enumerate a set of hardcoded "default services", services which are expected to be up and running in any Edge. X installation. Currently that list of services is hardcoded in edgex-go List. Default. Services. • Static list when not using Consul; Consul list when using Consul • Do not pay attention to default services list from Consul; only use the default services when Consul is not being. • Review SMA polling of services • CLI • Jim to develop list of top X (number TBD) features required for developers/users to start using • Diana/Malini fixing Mike identified issues (and then some) • Make determination of value and move from holding at a later date edgexfoundry. org | @edgexfoundry

Security • See https: //docs. google. com/document/d/1 FS 2 C_r 1 xm. AMj 9 Ae. Pt. RCv. Ukl. YRd. OKL_Tjr. O-CX 466 MZU/edit • Security WG to provide “HOW-TO” guides for following in priority order as part of Hanoi release: • • • SSH Tunnel (needs document and docker compose file snippets and may need some special containers for ssh/sshd or instructions for their setup) Overlay network (needs document and docker compose file examples/snippets) Service mesh (needs document and script on how to automate) • An abstraction and pluggable mechanism to provide generation of secrets • Current Go Key mod is not fit for production • Address containers security issues • • • running as non-root user no new privileges read-only Container images • Design secure subsystem launch/bootstrap dependencies • Review design of Hardware Root of Trust - API with pluggable implementation (ARM to help) • Kong: enable CORS for API Gateway • Kong: Secure admin port with TLS • Design enablement of Vault PKI secret engine • Develop process for security vetting of 3 rd party components • Stretch: improve go-mod-secrets abstraction for insecure and secure secrets ADR. • Already in App Services (secrets in vault versus memory or config) edgexfoundry. org | @edgexfoundry

Dev. Ops • Performance Optimizations • Jenkins Pipeline optimizations for edgex-go • Explore options from LF for supporting Jenkins on K 8 s • Develop process for security and license vetting of 3 rd party components • License compatibility with project also to be considered part of this review • Parallel activity with code dev • Synk could be helpful here • James/Dev. OPs with Security agreed to take the lead • CI check flagging? • Stretch: Restructuring our compose files to take advantage of compose file overrides, which removed the duplication in all our compose files. • Address ARM images • Explore • • • extend mechanism; compose 3. 7 include “middleware” Variable interpolation (for ARM) Impacts black box testing Impacts to Dev/Ops • Stretch: code Coverage for Jenkins Global Libraries (codecov. io) • Stretch: Snap improvements (WIP) • Efficiencies / use nexus proxies / etc. • Support for upgrade to Go Lang 1. 15 • Race flag resolution edgexfoundry. org | @edgexfoundry

Others • Certification • Endorsement (name TBD) process – WIP • Independent of release schedule • Part of new Outreach and getting user input • Vertical Solutions • Next steps in Ideation/topcoder challenge • Open Horizons • Look K 8 s/Jim to lead with Joe edgexfoundry. org | @edgexfoundry

Geneva Release Lessons Learned • What do we continue that we did really well this time? • Release planning going much more smoothly • Project tracker boards being utilized by all and used effectively • Commit linting and bots – helped with discipline with PR • Templates on PRs • Revised pipeline really helped – improved performance • Not stepping on each other as much • No branching at code freeze • Security did better job scoping so it finished on time • No tiger-team mode during the release meeting • Seeds of integration and backward compatibility tests are going to help • Documentation improvements (& search) were great • And release automation around it; don’t have to ask LF • ADR was good start edgexfoundry. org | @edgexfoundry • Where we need to improve? • V 2 API issue – feature creep that needed to be reviewed earlier or at least we needed more eyes on earlier • Need more integration testing in early phases of the project • To include integration testing with security • New features need to engage security team earlier (example: Redis work pushed Secty to the end) • Documentation improvements • Architecture/design decisions • • Can we look at tagging architecture and design issues and items in the project boards? To be taken up in architects meeting • Customer or end user feedback is much needed; we are more mature and need real user input • Separate requirements of users vs requirements of developer/contributor • Need feedback from customers on how better QA would provide better product

Virtual Meeting Lessons Learned • Virtual vs Face-to-face • • • Virtual effective, but a lot of importance in F 2 F (relationship building) Virtual works; F 2 F preferred Not everyone can attend F 2 F dial up is terrible experience Zoom while at F 2 F with everyone online? ? Missed the ability to have the local users / customers at the meeting • Times of virtual meeting (4 hrs vs full day) worked out well • China friendly time good, but switch to different topic mix (mini conf) edgexfoundry. org | @edgexfoundry