AI Strategy Team

From DDCIDeos
Jump to navigationJump to search

AI Strategy Team

  • Initial meeting 3/10/2025
  • Weekly meeting on Wed at 1:30 CDT: | Teams Meeting

Goals

  • Explore possible applications and implications of using AI at DDCI
  • Create a plan and schedule for DDCI’s adoption of AI to increase performance and reduce cost
  • Focus on AI as it applies to the Engineering Team.  No preconceived notions.  No preset cost limitations. Let your mind go, and see what you find!  Consider pros and cons.  Benefits and costs.  Risks and rewards.
  • Consider non-AI solutions to reduce cost and development time (when AI does not apply or make sense)

Possibilities

  • Code, test, requirements generation, document information retrieval
  • Learn from existing BSPs, drivers, etc to make components more consistent/efficient

Use Cases

  • Reduce test development time
  • Improve integration testing to identify issues sooner in the development cycle
    • Identify potential integration conflicts, bugs, and missed coverage/tests early
  • Reduce component/platform development and verification time
  • Reduce component/DDS release testing time while increasing test coverage
    • Analyze and report test failures, compare to history, and submit problem reports if needed

Considerations

  • AI vs (in-house) automation vs current process
    • Cost (learning, setup, maintenance, security)
  • Concerns
    • IP - Can use open source for AI learning/reference as long as any generated artifacts have accurate licensing documented
    • Skill atrophy - the more we rely on AI the more likely our skills in development and reviews will atrophy
  • How much data is sufficient/required to train a model? Too much for DDCI to train it's own model
    • Need to quantify the "limit" of DDCI's data, and what's possible

Modeling and ChatBot Tool

  • Create a tool that can take different type of files (htm, pdf, source code) to generate embeddings, and accept search queries.
  • The tool could be used for user guides (customer facing chatbot), technical documentation, wikis, etc.
  • Tool Location: https://deos.ddci.com/svn/DDCI/contrib/AIChatBot

Software Verification (Kernel and Library components)

  • Test case generation
    • Test case generation is possible using AI tools, BUT test case documents often have to be separated due to unlisted restrictions/obstacles from supporting configuration files.
    • Instead of AI, we could use test case templates and scripting tools to create a starting point for a new test case document?
  • Test procedure generation
    • We need better and more consistent test cases to help with test procedure creation
    • Should we start using TOFU more often?
    • Instead of AI, can we use existing testing tools (which exist aplenty)?
    • Many test procedures require unique configuration file settings and other supporting files
    • Will software static analysis tools (such as Coverity) help with existing test procedure development?
  • Updates to existing test cases and test procedures
    • This scenario might lend itself more appropriately to AI tool since we have something to start with and only need to make modifications/additions

Resources

General References

https://www.techtarget.com/searchenterpriseai/Ultimate-guide-to-artificial-intelligence-in-the-enterprise
https://aws.amazon.com/what-is/large-language-model/
https://www.ibm.com/think/topics/rag-vs-fine-tuning-vs-prompt-engineering (RAG vs Fine-Tuning vs Prompt Engineering)
https://www.ibm.com/think/topics/ai-in-software-development

Automation

https://www.jenkins.io/ (Automation - CI/CD)
https://about.gitlab.com/solutions/continuous-integration/ (GitLab CI/CD)

Tools and Engines

https://ai.google/get-started/gemini-ecosystem/
https://github.com/features/copilot (AI for code development)
https://about.gitlab.com/gitlab-duo/ (GitLab Duo - code and test generation)
https://docs.qase.io/general/get-started-with-the-qase-platform/ai-test-case-generator (Test case generation)
https://www.youtube.com/watch?v=2TJxpyO3ei4 (RAG Python Example for AIChatBot)
https://github.com/pixegami/rag-tutorial-v2/tree/main (RAG Python Example Source Code)
https://ollama.com/ (Local large language models, LLM)

Agenda Suggestions for Next Meeting

CI/CD Topic

Meeting Notes

2025-Jul-16

  • Failures due to dependencies:
  • Make build logs a (compressed) attachment for failed build emails (Peter, suggested by Ryan)
    • Shayne: Provide a link to the build log on the server. Best option.
  • Work on filter/database to trigger component builds only if svn version change since last build run (Peter)
  • Team access/permissions on the jenkins@ddci.com - Kevin has this set up for some of the team
  • Mark: test-infrastructure also needs to be added to the DDS for the test suites to build
  • Email notifications (scripting) - ongoing updates
  • Merging the scripting onto the Jenkins machine - after the scripting is working "perfectly"
  • Next steps:
    1. Evren: figure out how to build regress tests for components identified on the Verification_Team
    2. Kenny: existing test harness requires manual steps to load DDS and kick off OA platform integration test suite. Need to automate the test harness.
      • Matthew: Swtbot plugin - potential to automate command line? Jenkins??
      • Aaron: Can the required parameters needed to kick off the test suite on a particular target be passed to the test harness?
      • Step 1: kick off test suite on qemus

2025-Jul-9

  • Team access/permissions on the jenkins@ddci.com - Kevin is working on this; need to confirm
  • Matthew: added maintainer-tools to the DDS (single DDS); only being built if bdu script is updated
  • Mark: test-infrastructure also needs to be added to the DDS for the test suites to build
  • Email notifications (scripting) - in progress; next step: create a cron job
  • Next steps:
    1. Evren: figure out how to build regress tests for components identified on the Verification_Team
    2. Kenny: existing test harness requires manual steps to load DDS and kick off OA platform integration test suite. Need to automate the test harness.
      • Matthew: Swtbot plugin - potential to automate command line? Jenkins??
      • Aaron: Can the required parameters needed to kick off the test suite on a particular target be passed to the test harness?
      • Step 1: kick off test suite on qemus

2025-July-02

  • Mark proposed bringing Evren onto the team next week since they have some Jenkins experience and may have time to help with adding testing to our CI/CD setup.
  • Matthew:
    • Should Jenkins bdu builds be weekly instead of nightly or only when sales bdu script changes?
    • Considering scripting a check to see if latest bdu build has all the same versions as last published build (no change, don't publish, reduces clutter in the registry)
    • The docker image that is being created does not contain maintainer-tools, as it is based off the customer-sales package. Still needs to be resolved.
    • deos-maintainer-tools is added to the deos-maintainer-kismet-image.
    • Still doing some AI work -- Chatbot progress, Bill indicated more visibility to the team would be good. AIChatBot hosted on site: https://jenkins.ddci.com:5000/DeosChatBot

2025-June-25

  • Team access/permissions on the jenkins@ddci.com - Kevin is working on this; access control will be refined over time
  • The docker image that is being created does not contain maintainer-tools, as it is based off the customer-sales package. Still needs to be resolved.
    1. CI/CD process should include 2 docker images: one for customer-sales and one for (customer-sales + customer-maintainer)
  • Email notifications (scripting) - in progress; next step: create a cron job


2025-June-18

  • Team access/permissions on the jenkins@ddci.com
    1. Kevin is aware that Deos Engineering needs access, just like permissions to svn repositories
    2. ToDo Matthew: create ticket for Kevin to create a nightly backup of jenkins machine
  • The docker image that is being created does not contain maintainer-tools, as it is based off the customer-sales package. Do we want to add this or should engineers be responsible for adding it themselves? This image is only useful for testing (in OA) at the moment.
    1. CI/CD process should include 2 docker images: one for customer-sales and one for (customer-sales + customer-maintainer)

2025-June-4

  • CI/CD testing for the June tsunami
    1. Jenkins (wrapper):
      • Runs the script for each component; requires a separate pipeline for each component
      • Sends email to owners for components that don't build
    2. Component Build Script
      • Performs svn checkout and builds each component (messages in log file used by Jenkins for pass/fail)
      • Make the script configurable to build a specific revision per component
    3. Docker Build Script or Jenkins to ensure the team is testing with a consistent DDS
      • ToDo Matthew: follow up with Jean on the steps to build the docker image using the BDU script
      • ToDo Matthew: look into option of using a Registry http://jenkins.ddci.com:8080/
      • Need an owner for the BDU script
      • Complete: Need a CI/CD server, and have Kevin set up SMTP host application for email notifications
  • Long-term: Jenkins vs python tool (configurable/selectable components)
  • Test team:
    1. short-term: ensure the tests build using regress script
    2. long-term: check the results

2025-May-14

  • CI/CD testing for the June tsunami
    1. Script that builds components (GitLab/GitHub and Jenkins/Subversion not needed in the short-term)
      • Email sent to owners for components that don't build
      • Built components -> regression test runs (limited set)
      • Need a CI/CD server, and have Kevin set up SMTP host application for email notifications
  • Long-term: Jenkins vs python tool (configurable/selectable components)
    1. Jenkins server builds a Docker Image -> ties into OA Platform Integration Testing

2025-Apr-23

  • Focus areas - status
    1. Peter/Kenny/Mark: CI/CD optimization (after CI/CD is already in place)
      • Implemented for Docker use only; high priority is x86_64 platform (since our current verf customers on Kismet use this platform)
      • Looking at GitLab AI augmented test/development (for pay)
      • Created a prototype Python CI/CD script. Notes at CI/CD#Prototypes_and_Experiments
    2. Shayne/Matthew: LLM Search Summarization
      • RAG (Retrival Augmented Generation)
    3. Kelly: PM tasks and resource mgmt
    4. Reach out to consulting companies that specialize in AI adoption

2025-Apr-16

  • Focus areas - status
    1. Peter/Kenny/Mark: CI/CD optimization (after CI/CD is already in place)
      • Looking at GitLab AI augmented test/development (for pay); Richard looking into licenses for Aaron/Ryan
      • Created a prototype Python CI/CD script. Notes at CI/CD#Prototypes_and_Experiments
      • Matthew: look into Jenkins for CI/CD use
    2. Shayne/Matthew: LLM Search Summarization
      • RAG (Retrival Augmented Generation) - plug chatGPT into DDCI RAG database (set up by Shayne), for support cases (for example) or PM database
    3. Kelly: PM tasks and resource mgmt
    4. Reach out to consulting companies that specialize in AI adoption

2025-Apr-09

  • Focus areas
    1. Peter/Kenny/Mark: CI/CD optimization (after CI/CD is already in place)
    2. Shayne/Matthew: LLM Search Summarization
    3. Kelly: PM tasks and resource mgmt
    4. Reach out to consulting companies that specialize in AI adoption

2025-Mar-28

Attendees: Peter, Mark, JD, Kenny, Matthew

  • Matthew: Used a python project to train AI with a DDS. Output was a bit gibberish. Not sure how to train AI better for better results, example for kernel.
  • Kenny: on OA side, Looked into test generators. Seems like the model would need to be retrained each time. Seems like it could be expensive. Will look into Qase and Copilot.
  • Peter: Found potentially promising things like Github Copilot, Qase. Most tools seem more geared to web and mobile apps.
  • Tasks: everyone pick an area to focus on for the next meeting
    • Kelly: resource management
    • Shayne: LLM options
    • Kenny: Looking for ways to generated tests that can be used in OA testing.
    • Peter: Looking for general tools that might apply to embedded coding and tests.
    • Mark:
    • Matthew: Trying out AI engines with real data.

2025-Mar-19

  • Mark: have vendors provide sales pitch and demo license for potential solutions
  • Kenny: on OA side, using AI for creating model to test cases. There's too much data for the model to analyze and be helpful
  • Shayne: Considering LLM being hosted by DDCI. Possible to start with a pre-trained model, eg. llama? This could reduce the amount of horsepower required by the engine.
  • Peter: break down tasks by automation vs analysis/generation
    • Automation: example are the steps to implement CI/CD
    • Analysis/Generation: have an AI tool generate reqs or test cases
  • Tasks: everyone pick an area to focus on for the next meeting
    • Kelly: resource management
    • Shayne: LLM options
    • Kenny:
    • Peter:
    • Mark:
    • Matthew:
Retrieved from "https://ddci.zapto.org/ddciWiki/index.php?title=AI_Strategy_Team&oldid=72587"