CI/CD

Continuous Integration

• Automatically build components as changes are committed to the repository
• Confirm that the build is not broken
• Notification of results (pass or fail)
  • Emails to team (or team leads), and to committer of most recent changes if the build fails

Continuous Testing

• Run automated tests on components when the build passes as part of the continuous integration process
• Testing could be set up to precede the automated unrelease
• Levels of Testing
  • Component specific
  • Integration (regress, interaction can be without OpenArbor)
  • OpenArbor integrated (configurations, user-like interactions)

Automation Needed

• Process a list of active components to build and test
• Detect commits made to the component since the last release
  • Maintain a database of releases (component, distro, version number, svn version of build files)
  • Scan server directories/files and update the database continuously
• Prepare a clean docker image for the build and test process and use for the remaining steps
• Checkout the component repository files (HEAD)
• Install all component dependencies for the component build
• Modify configure.ac (build version) and release notes
• Build the component
  • If the build fails, send emails/notifications to the stakeholders and authors of commits since the last release (or at least author of most recent commit)
  • If the build succeeds:
    • Install any additional component dependencies for testing
    • Log the versions of all components involved for test (examples, kernel, BSP, drivers, ...)
    • Run the test suite
      • If the tests fail, send emails/notifications to the stakeholders and authors of commits since the last release (or at least author of most recent commit)
      • If the tests pass, check in the version updated files, and unrelease the component? Then send notification(s) to announce the new unrelease version?

Continuous Deployment

• Automatically unrelease updated components (for testing and release)
• Ideally, release updated components when automated build and testing pass without issues

Possible Tools and Approaches

• In-house scripts (python, bash, other)
• Jenkins
• GitLab CI/CD

Future Concerns

• Ensuring that it is easily maintainable. Keeping in mind that maintainers are not expected/will not be asked to maintain any CI/CD things for their components. It is very important that whatever CI/CD turns into, it is easily maintainable. CI/CD should help decrease the workload of individual maintainers, not increase it.
  • Avoiding tech debt, ensuring there is a solid architecture before getting into the implementation details.
• Need clear expectations and requirements from management on their goals (nightly builds, OA testing, regress, and further?).
  • For the first release (June Tsunami), only nightly builds and an email system are expected. Only expecting to try building the components that are going into the DDS -- no need to test more than necessary.
  • For future releases, we need to continue thinking about OA Testing Integration and Regress Suite Testing Integration.
    • Lots of technical details to flush out here eventually. Some ideas are: gitlab vs jenkins vs our own scripts, cdproc(-makefile?) integration, unrelease vs svn HEAD testing, being able to use real hardware (and what about replacing the boot images if we get unexpected failures? not standardized across the boards, some use tftpboot, some use USB sticks, etc...), programmatically determining the list of components to test, testing different variants (release, diagnostic, etc), knowing what architecture to test on, a through logging system, keeping track of testing history (database?).

Prototypes and Experiments

Python prototype

Runs in a docker container. Processes a list of components, building a clean checkout of HEAD, and building HEAD with updated externals.
CI/CD scripts and support files

• Checkout directory and resulting text files are saved in the directory in which the script is run.
• If the build succeeds, the output text file ends with _cicd_results.log. A build error output file ends with _cicd_error.log.
• Builds checking with updated externals include _extern_up in the directory and output file names.
• Contacts (users/emails) extracted from svn log entries (last 5 for HEAD build, all in updated externals logs for updated externals build)
• smtp library in script, with function to send emails to contacts found if build fails. This has not been tested with an smtp server access set up yet.

Example outputs (test directory, build summaries):

Build directory view	Build Report Summary html	Build Report Summary html (with updated externals)