Continuous Integration Continuous Delivery For FPGA projects of

Continuous Integration / Continuous Delivery For FPGA projects of Power Converter Controls S. Uznanski, P. Wozny, A. Byszuk, et al. slawosz. uznanski@cern. ch Oct 22, 2020

1. Introduction 2. What is Git and CICD? 3. Infrastructure 4. Design, simulate, build, validate, release… 5. Operational support 6. Conclusions slawosz. uznanski@cern. ch CERN 2016 2

Introduction We are just starting with Continuous Integration and Continuous Delivery Automated in CICD Total PCB boards 23 33 FPGA projects 73 77 Remote reprogramming capability 53 FPGA prj 14 PCB prj Automatic builds >1. 2 k pipelines >65 k jobs What do we already do automatically? • • slawosz. uznanski@cern. ch Builds of any programming file delivered to accelerators Unit testing Remote programming of HW Remote hardware testing CERN 2016 3

Introduction What was our motivation? 1. 2. 3. 4. In Jan 2020, Expert asked for a bit stream delivery… Main designer on holidays Tools how to generate project were undocumented For a week converter was stopped & no one could re-generate a working project from the source code Previous bit stream? Source code? Files? C: / DFS Gitlab CAD version? No simulation? Project properties? Merging DSP/VHDL CICD allows for automation so you never face these problems again! slawosz. uznanski@cern. ch CERN 2016 4

What is Git and CICD Git. Lab is a version control system: https: //gitlab. cern. ch Version control was adopted in software development as early as in the 1950’s (manual) & became an industry standard in the 1980’s Master Version control allows for managing sets of data over time. Tracking of changes: checking out/committing Each commit: timestamp/author/comment Collaborative work between multiple designers and mechanisms to branch/merge data. Our implementation has a central repository hosted at CERN. slawosz. uznanski@cern. ch CERN 2016 5

What is Git and CICD Continuous Integration https: //docs. gitlab. com/ee/ci/ CI is a practice of periodically merging all developers’ work. In software development ideally several times per day. The longer the development is not merged the greater risk of integration conflicts and regressions. CI is intended to be used with automatic: - testing (unit/integrations tests): in simulation or hardware - builds that assure quality control: create programming file All developers can see the results of builds. Continuous Delivery CD aims for producing software/hardware in short cycles assuring we can release it at any time slawosz. uznanski@cern. ch CERN 2016 6

What is Git and CICD Continuous Delivery https: //docs. gitlab. com/ee/ci/ CD aims for producing software/hardware in short cycles assuring we can release it at any time slawosz. uznanski@cern. ch CERN 2016 7

Infrastructure FPGA Prj 2 FPGA Prj 1 Gitlab CICD Automation Commits: https: //gitlab. cern. ch/cce/-/network/master Environment Layer Hardware Layer slawosz. uznanski@cern. ch Images: https: //gitlab. cern. ch/cce/docker_build 160 CPU 320 GB RAM 20 Vol 10 TB CERN 2016 Runners: https: //openstack. cern. ch/project/instances/ 8

Design Flow slawosz. uznanski@cern. ch CERN 2016 9

Design, simulate, build, validate, release… We design RTL in VHDL with VHDL Test Benches for simulation Designer’s computer CCE-DEV-1|2 Gitlab CICD Using local tools Using Docker & Images of Gitlab environments Runs Images of Gitlab environments Uses Gitlab runners & Images of Env Local copy of the repository & non standard tools. Designer uses Gitlab CICD Environments Designer uses dev machines that run Gitlab CICD Environments Designer runs Gitlab CICD from the browser Telework possible All code must be committed to Git on a development branch • • slawosz. uznanski@cern. ch Builds of any bit stream delivered to accelerators Unit testing Remote programming of HW Remote hardware testing CERN 2016 10

Design, simulate, build, validate, release… All simulation Test Benches are run automatically, all projects can be simulated in parallel Gitlab CICD Uses Gitlab runners & Images of Env Designer runs Gitlab CICD from the browser All code is automatically simulated slawosz. uznanski@cern. ch CERN 2016 11

Design, simulate, build, validate, release… All builds are run automatically when they are changed or periodically to check for regressions Gitlab CICD Uses Gitlab runners & Images of Env Designer runs Gitlab CICD from the browser All code is automatically built upon request or periodically slawosz. uznanski@cern. ch CERN 2016 12

Design, simulate, build, validate, release… The validation might be done in the lab using CICD: the project is built, the bit stream is used to reprogram a board and a Python framework allows for automatic HW test Gitlab CICD Uses Gitlab runners & Images of Env Designer runs Gitlab CICD from the browser All code is automatically deployed in the development rack slawosz. uznanski@cern. ch CERN 2016 13

Design, simulate, build, validate, release… All designs are released from the pipelines and sent to OMS team that deploys the programming files either locally in the lab or remotely using the program manager Gitlab CICD Uses Gitlab runners & Images of Env Designer runs Gitlab CICD from the browser The build link is sent to the deployment team slawosz. uznanski@cern. ch CERN 2016 14

Operational Support slawosz. uznanski@cern. ch CERN 2016 15

Conclusions Just some advantages of CICD for us: Code is always ready to be deployed for each system All developers get more time for development avoiding generation of projects Everyone contributes and can find all projects at any time CICD boosted simulation in the team and avoids big part of debugging in HW CICD includes as well much better simulation with code coverage If anyone breaks any project it is detected within hours instead of years! Regression testing comes with no development time • • • What we expect to improve • • • CICD will boost code re-use and code sharing We will build automatic testing platforms avoiding a big part of manual tests CICD will allow to consolidate Costs of CICD • • • Requires a rigorous Dev. Ops team that will define infrastructure and will maintain the framework (at least 0. 5 FTE) Learning curve for all designers and less flexibility as you need to comply with the framework Some occasional limitations (ex. simulation/build times) slawosz. uznanski@cern. ch CERN 2016 16