CI/CD

CI/CD Meeting Details

Wed, 1:30pm MDT
Teams Meeting Link
Meeting ID: 244 603 111 788 3
Dial in by phone
+1 323-618-1547,,440892883# United States, Los Angeles
Phone conference ID: 440 892 883#

Important Links

• CI/CD scripts and support files
• Latest Build Summary
• Latest Build Artifacts Note: Sometimes give an error page, refresh usually clears this error
• Open Tooling PCRs These open PCRs may relate to tooling updates required for CI/CD.
• Local Jenkins
  • To Login: Username: $USER@ddci.com Password is the Active Directory password.

Meeting Notes

Note: The CI/CD topic has been part of the AI Strategy Team meetings and the meeting notes are on that page.
Link to Meeting Notes

2025

Next Meeting Topics

• "Building on the built-in node can be a security issue. You should set up distributed builds. See the documentation." warning

10-Dec-2025

1. Andre: Working with CI/CD team to get Coverity integrated into Jenkins' component builds.

12-Nov-2025

1. CP: Demo'd the WIP updates being made to vfile to support Jenkins CI/CD integration.
   • Updates to cygwin2deos.py and new inclusion of jenkins.yml file in each component to support these updates.
2. GeekFest Presentation
   • Focus on "what was done", "what is being done", and "what is going to be done"
     • Shayne: CI/CD Overview & Python Scripting
     • Peter: Python
     • Evren: Regress
     • Matthew: Jenkins
     • Kenny: OpenArbor
     • TBD: Looking forward, future features, etc

5-Nov-2025

1. PZ: We need a way to invoke instrumented/instrumented_optimized builds of components for automated build and testing (as needed and/or regularly recurring)
   • Not all components have these variants enabled in the configure.ac, is there a build-utils way to bypass this? How does OpenArbor do it?
   • This would be useful for abc-tool testing, maintenance, and enhancement; and for proactively checking the verification readiness (structural coverage) of any component or set of components.
   • Change local checkout of configure.ac to enable these variants?
2. SS/EG: Intern task to make a web interface for the CSV? Can enable engineers to enable/disable their builds, add dependencies, etc...
3. MC: Add a component build using the python script to the Jenkins server.
4. Set up Jenkins so team leads/engineers themselves can go into Jenkins and add/remove/modify jobs. Jenkins jobs will have the component name, SVN URL, etc.

15-Oct-2025

1. Evren and Shayne: working on script to run under Jenkins user

08-Oct-2025

1. Andre: joined to discuss adding Coverity to the CI/CD script
2. Evren and Shayne: working on script to run qemus
   • Issue starting qemu from the script due to qemu start script requiring a graphics console; Aaron to help out

01-Oct-2025

1. Evren and Shayne: working on script to run qemus
2. Shayne and Aaron: build-utils generates a file during the build that captures the component's subversion location on the ftp server(see notes from 24-Sept)
   • configure.ac file provides location of the component on the ftp server (useful when looking for stable or unreleased components)
   • Starting point: BDU script identifies unreleased components; these should be built from Head
3. Aaron: create the CI/CD infrastructure without policies; policies will be added as necessary; eg. custom BDU script(s)

24-Sept-2025

1. Richard: Database for querying information on each component:
   • Shayne: Need to make the build consistent across components; eg. common location for current version../branches/mainline
   • Adina: Which customers are using the component
   • Database options:
     1. sqlite: Can it handle concurrent access?
     2. Other options?
2. Shayne: Open Tooling PCR List related to Planning Docs (game launching checklist based on desired versions)
   • Planning CCB is needed (Peter!)
3. GeekFest Presentation
   • From Richard: "At GeekFest I would like a presentation on what we've done in terms of CI/CD and what's on the short/long term roadmap for extending it. We'll have to decide based on roadmap needs if we need extra time in that session or a breakout to get ideas for implementing the extended features."
4. Nightly build script: runs if components are updated; Peter to add abc-tool
5. Evren: prototype of script is running on qemus
6. Kenny: looking into automating the test harness runs

27-Aug-2025

• CI/CD Script
  1. For Sept, the plan is for a mini tsunami with a very limited set of components: uart-zus (zcu102-aarch64), xilinx-gem (nai-ultrascale)
     • bdu64sales - in progress to build the Docker image on the Jenkins machine; will require OA team to manually install the DDS and kick off the platform integration testing
  2. Evren: working on kicking off regress testing

06-Aug-2025

• CI/CD Script
  1. Peter: cicd_build_components.csv file was not committed to svn last night; Shayne will research the issue - likely caused by file not being up-to-date
  2. Evren: can start working on kicking off regress testing

30-Jul-2025

• CI/CD Script
  1. Shayne: working out script issues for environment variables not being defined
  2. Peter: add check for updates, and only build components that have been modified; this includes externals
  3. Evren: can start working on regress testing
• Addressing the abc failures - check for existence of an instrument script on components locked down on older build utils
  1. short-term (complete): Shayne: add dummy instrument file to the CI/CD script
  2. long-term: update externals on each component for build utils
• Shayne: move the script/files to the permanent location: https://deos.ddci.com/svn/DDCI/maintainer-tools/ci-cd/branches/mainline
  1. Create a component in public bugzilla

23-Jul-2025

• CI/CD failure messages related to abc instrumented
  1. Failure messages include a zipped build log and a link to the build log on nx3000
• CI/CD Script
  1. Done for build objective
  2. Peter: add check for updates, and only build components that have been modified; this includes externals
  3. Evren: can start working on regress testing
• Addressing the abc failures - check for existence of an instrument script on components locked down on older build utils
  1. short-term: Shayne: add dummy instrument file to the CI/CD script
  2. long-term: update externals on each component for build utils
• Shayne: move the script/files to the permanent location: https://deos.ddci.com/svn/DDCI/maintainer-tools/ci-cd/branches/mainline
  1. Create a component in public bugzilla

Continuous Integration

• Automatically build components as changes are committed to the repository
• Confirm that the build is not broken
• Notification of results (pass or fail)
  • Emails to team (or team leads), and to committer of most recent changes if the build fails

Continuous Testing

• Run automated tests on components when the build passes as part of the continuous integration process
• Testing could be set up to precede the automated unrelease
• Levels of Testing
  • Component specific
  • Integration (regress, interaction can be without OpenArbor)
  • OpenArbor integrated (configurations, user-like interactions)

Automation Needed

• Process a list of active components to build and test
• Detect commits made to the component since the last release
  • Maintain a database of releases (component, distro, version number, svn version of build files)
  • Scan server directories/files and update the database continuously
• Prepare a clean docker image for the build and test process and use for the remaining steps
• Checkout the component repository files (HEAD)
• Install all component dependencies for the component build
• Modify configure.ac (build version) and release notes
• Build the component
  • If the build fails, send emails/notifications to the stakeholders and authors of commits since the last release (or at least author of most recent commit)
  • If the build succeeds:
    • Install any additional component dependencies for testing
    • Log the versions of all components involved for test (examples, kernel, BSP, drivers, ...)
    • Run the test suite
      • If the tests fail, send emails/notifications to the stakeholders and authors of commits since the last release (or at least author of most recent commit)
      • If the tests pass, check in the version updated files, and unrelease the component? Then send notification(s) to announce the new unrelease version?

Continuous Deployment

• Automatically unrelease updated components (for testing and release)
• Ideally, release updated components when automated build and testing pass without issues

Possible Tools and Approaches

• In-house scripts (python, bash, other)
• Jenkins
• GitLab CI/CD

Future Concerns

• Ensuring that it is easily maintainable. Keeping in mind that maintainers are not expected/will not be asked to maintain any CI/CD things for their components. It is very important that whatever CI/CD turns into, it is easily maintainable. CI/CD should help decrease the workload of individual maintainers, not increase it.
  • Avoiding tech debt, ensuring there is a solid architecture before getting into the implementation details.
• Need clear expectations and requirements from management on their goals (nightly builds, OA testing, regress, and further?).
  • For the first release (June Tsunami), only nightly builds and an email system are expected. Only expecting to try building the components that are going into the DDS -- no need to test more than necessary.
  • For future releases, we need to continue thinking about OA Testing Integration and Regress Suite Testing Integration.
    • Lots of technical details to flush out here eventually. Some ideas are: gitlab vs jenkins vs our own scripts, cdproc(-makefile?) integration, unrelease vs svn HEAD testing, being able to use real hardware (and what about replacing the boot images if we get unexpected failures? not standardized across the boards, some use tftpboot, some use USB sticks, etc...), programmatically determining the list of components to test, testing different variants (release, diagnostic, etc), knowing what architecture to test on, a through logging system, keeping track of testing history (database?).

Issues, Concerns, and Feedback

Add user feedback and issues here.

• Build components on the build list only when a new change has been committed to the component
  • Possible Solution
    • Check "svn info -r HEAD <component-path>" for "Last Changed" data (Rev, Date, and maybe Author), save data and compare against saved database/file to trigger build if new.
• Automated component build/test dependency identification and addition to build process.
• Build logs too long to include in emails (can crash Outlook)
  • Possible Solution(s)
    • Attach log to email rather than including in message body
    • Provide link to build log file on the build host server

Prototypes and Experiments

Python prototype

Runs in a docker container. Processes a list of components, building a clean checkout of HEAD, and building HEAD with updated externals.
CI/CD scripts and support files

• Checkout directory and resulting text files are saved in the directory in which the script is run.
• If the build succeeds, the output text file ends with _cicd_results.log. A build error output file ends with _cicd_error.log.
• Builds checking with updated externals include _extern_up in the directory and output file names.
• Contacts (users/emails) extracted from svn log entries (last 5 for HEAD build, all in updated externals logs for updated externals build)
• smtp library in script, with function to send emails to contacts found if build fails. This has not been tested with an smtp server access set up yet.

Example outputs (test directory, build summaries):

Build directory view	Build Report Summary html	Build Report Summary html (with updated externals)

Jenkins Builds

The Jenkins Server is running on the remote machine: jenkins.ddci.com

• Jenkins UI: http://jenkins.ddci.com:8080/
• Jenkins Built ChatBot from /desk/help of maintainer builds: http://jenkins.ddci.com:5000/DeosChatBot