OpenArbor Integration Testing

From DDCIDeos
Jump to navigationJump to search


Introduction

The OpenArbor Test Harness eclipse application provides an interface for executing the OpenArbor automated test suites as well as individual OpenArbor automated tests.
This is a high-level overview of how to accomplish different types of testing use the OpenArbor test harness. More detailed documentation can be found at OpenArbor Testing.
The two steps are described below:

  1. Choose your test machine setup
  2. Determine the type of Testing Session you will be performing

Test Machine Setup

The types of available testing sessions are described in the next section. To use the OpenArbor Test Harness, it can be installed and built on your machine or you can connect to one of the available OpenArbor test machines. Both options are described below.

Run the OA test harness on your machine:

  1. Follow the client setup instructions.

Run the OA test harness on an OpenArbor test machine:

  1. Pick one from the list.
  2. Reserve the test machine on X9
  3. Connect to the test machine:
    • For a Windows test machine, Connect to it using vnc
    • For a Linux test machine, connect via Remote Destop Connection (RDC)
    • NOTE: If you are unable to connect, contact one of the OpenArbor team members for help. While unlikely, it is possible the test machine may not be setup. The OpenArbor team member can evaluate it the test harness is not running or if the test machine is not setup properly. If so:
    1. Follow the server instructions.
    2. Follow the client setup instructions and then connect to the server.
  4. If not running, launch the test harness on the test machine.
    1. Windows:cd /cygdrive/c/ddci_eclipse_test_harness_dev/dds-manager-e4 && ./oaTestHarness.sh -n.
    2. Docker:cd ~ddci_eclipse_test_harness_dev/dds-manager-e4 && ./oaTestHarness.sh -n.

Types of Testing Sessions

Follow the instructions below for the type of testing you need to do (Official DDS?), and then:

  1. Add a config.
    1. If you are trying to test a board with no official config, just pick one that is close to what you are trying to test and then make the necessary modifications.
    2. If using an OpenArbor test machine, you will need to lock the x9 target as yourself instead of as scoretest.
    3. To ensure your test machine setup is correct, you can execute the Run test for the hello-world example. To do this, specify com.ddci.openarbortests.compileRunDebug.Run.hello-world in the Tests field.
    4. You probably don't need to test all of OpenArbor, so use one of the subsets instead. To do this, specify the subset's name in the Tests field. The available subsets are:
      1. loadABunchOfTests: Mostly project option tests, and some miscellaneous stuff like the Ada path dialog and the dumpers/disassembler.
      2. loadCompileRunDebug: Ensures that all of the examples can compile, run, and debug successfully. This can be a good BSP smoketest.
      3. loadRtemsTests: Exercises the RTEMS support functionality in the OpenArbor GUI.
      4. loadDeosTests: Exercises a lot of Deos-specific functionality in the OpenArbor GUI.
      5. loadGdb: This is a meta-grouping that exercises some aspect of Gdb's interaction with OpenArbor.
      6. loadMldTests: This is a meta-grouping that exercises every aspect of the MLD's interaction with OpenArbor.
        1. loadDebugCommands: Commands that manipulate the output of the MLD.
        2. loadExpression: How objects and expressions are displayed and formatted.
        3. loadStopPoints: Breakpoint and tracepoint tests.
        4. loadWindows: Tests each view in the debugging perspective.
        5. loadDisassembly: Exercises the disassembly view.
        6. loadSubprograms: Tests running various types of subprograms through the MLD.
        7. loadAda: Ada-specific debugger functionality.
        8. loadMemory: Various methods of exploring and manipulating memory regions.
      7. loadOtherTests: A miscellaneous grouping that verifies minimum OA functionality.
      8. loadPlatformIntegration: A minimal subset specifically targeted at smoketesting a complete DDS.
        1. loadPlatformIntegrationLegacy: Like above, minus tests that don't exist on the elbert branch.
      9. loadTargetOuptutTests: Performs some basic run and debug testing.
      10. com.ddci.openarbortests.deos.BuildBspDevKit: Builds an installed dev kit from it's zip file.
  2. Click the Test button and wait for the results.

Testing Specific Examples

The "Examples to test exclusively" checkboxes on the DDS Manager view contains a checkbox for each category of examples installed in the DDS. Developers can select one (or more) example categories to be tested. When checked and the Test button is pressed, the examples in each category will be imported into the test workspace, and the following tests will be executed for each example: Compile, Run, Debug.

  1. Check one (or more) categories of examples to test media:ExampleCategoryChecked.png
  2. Click the Test button


The Log Viewer tab, keeps a count of the testing progress. The number of Failed, Passed and Unrun tests are displayed as well as the Total number of tests. When 'Unrun' is 0, then all the tests requested have been run!

The Log Viewer also shows the results for each test executed. Clicking on a test name will cause the log for the test to also be displayed. When failures occur, the test will be listed in red (and in the Fail drop-down list). An OpenArbor team member can be helpful in understanding what the failures mean.

Testing an Official DDS

An Official DDS is one that has been (1) QA'd and shipped to a customer or (2) created for formal testing on DDSBUILD. The Add a dds provides location selection choices of official DDS'.

  1. To select an Official DDS, right-click on the machine name (e.g localhost) and select Add a dds.
  2. Click the Install button to install the DDS onto the test machine.
  3. The installation is complete when the DDS Manager View displays: "ddciFlex is Up!" and the hourglass is no longer visible for the test machine.
  4. If you need to modify the dds, you will need follow the instructions below to add unreleased components before testing.

Testing with Unreleased Components

Windows

  1. Click the Run Desk Setup button, select Install From Internet, and click through to select the unreleased components and install them.

Docker

  1. Create a docker image containing the unreleased components using the bdu64 script (or something similar)
    1. ./bdu64 kismet customer-sales
  2. Replace the offical dds' docker image with the one you've created
    1. run docker on the ubuntu-kismet-sales image created above and in another terminal use docker container commit to save the changes to the DDS image you are testing

Testing with Local Builds (Windows only)

  1. Now you can make any local changes that you need, such as installing a test version of a component. The dds installations are located in <workspace>/DDS-list/<dds>/.
    1. For example, if you wanted to test a new bsp, you would replace the matching bsp in <workspace>/DDS-list/<dds>/desk/platform/<bsp> before starting a test run.

Testing Existing Installations

An existing installation is a DDS that was not installed by the OpenArbor Test Harness. Use Add DDS->Add an existing installation..., if you have a DDS installed on your Windows machine, or an image created on your Docker workstation that you'd like to use for running OpenArbor automated tests. Once added, the test workspace can be modified to specify Deos Search Path directories for Deos components that are 'in work' (e.g. not unreleased or stable).

Windows

  1. Right-click on localhost and select Add DDS->Add an existing installation....
  2. Use the File dialog to select the directory containing an installed DDS
    1. Note that test data is stored by the name of the location's directory, e.g. C:/foo/bar will be tested out of C:/workspace/ddses/bar. This means that adding C:/baz/bar would collide.
  3. Right-click on the dds name added (e.g. C:/foo/bar), and select Add Test Config.
  4. Select the config for the target being tested (e.g. deos_arm_qemu)

Docker

  1. Right-click on localhost and select Add DDS->Add an existing installation....
  2. The Docker Image List Dialog is displayed media:docker-image-list-dialog.png
  3. Use the Select docker image drop-down list to select an image to use for testing (e.g. dds-docker-training-deos-kismet-20240524:latest)
    1. Note that test data is stored by the name of the image, e.g. dds-docker-training-deos-kismet-20240524:latest will be tested out of ~/workspace/ddses/dds-docker-training-deos-kismet-20240524_latest.
  4. Right-click on the dds name added (e.g. dds-docker-training-deos-kismet-20240524:latest), and select Add Test Config.
  5. Select the config for the target being tested (e.g. deos_arm_qemu)

(Optional) Define Deos Search Paths in the testing workspace

For Deos developers that want to test a component that is not yet unreleased (or stable), use the instructions below to include the component in the test workspace. The keepSearchPaths setting is used by the test harness to preserve any Deos Search Path entries in the test workspace when tests are run. Note: The keepSearchPaths config setting is only supported in OpenArbor 12.3.0+

  1. Right-click on the config, and select Modify Config....
  2. In the Input Dialog add "keepSearchPaths=true", and click OK.
  3. On the DDS Manager tab view, click the Run for Debugging button
  4. Once OA has launched, use the Windows | Preferences | DDCI | Deos Search Path dialog to add paths for components 'in work' to be tested.
  5. Apply and close the Preferences dialog
  6. Close OA

Known Testing Issues

Test Machines

Freezing

If a machine freezes or hangs, as they happen to do, they can be restarted remotely from a terminal with:

ssh <machine-name>
sudo reboot

OA Test Harness

Use this space to document issues with the test harness and any work-arounds.

Image/Container 'already exists'

If the following message appears in the test harness log when trying to launch a test:

The DDCI flexnet server docker image 'ddci-flexnet-server-linux-20230119' already exists.
The DDCI flexnet server docker container 'ddci-flexnet-server-linux-20230119-7181' already exists.
To continue you should:
docker container stop ddci-flexnet-server-linux-20230119-7181
docker container rm ddci-flexnet-server-linux-20230119-7181
Then re-run this script.

Delete the license.lic in the ~/oaTestharness/workspace/ddses/<dds-name> folder, and launch a test. This will fix the issue.

java.lang.Exception: Could not find plugin "com.ddci.openarbortests"

This problem has happened a few times when starting an OA test run on Linux or WSL.

Workaround #1:
  1. cd to oaTestharness/branches/<branch>/workspace/build/oatest-workspace
  2. delete the "com.ddci.openarbortest_feature" and "com.ddci.openarbortests" folders in that directory
Workaround #2:

The ~/.eclipse/1418632101_linux_gtk_x86_64 contains a features and plugins folder. These folders should be empty, or contain the folder and jar file for the version of the test plugin for the DDS under test (e.g. com.ddci.openarbortest_12.5.0.r108769, com.ddci.openarbortests_12.5.0.r108769.jar) 

+ /OpenArbor/eclipse -nosplash -application org.eclipse.equinox.p2.director -uninstallIU com.ddci.openarbortest.feature.group -repository file:/tmp/oatest-workspace/com.ddci.openarbortest_feature/target/site
SLF4J: No SLF4J providers were found.
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See https://www.slf4j.org/codes.html#noProviders for further details.
Uninstalling com.ddci.openarbortest.feature.group 12.4.0.r108580.
Installation failed.
Cannot complete the install because of a conflicting dependency.
    Software currently installed: DDC-I Common Feature 12.4.0.r107898 (com.ddci.common.feature.group 12.4.0.r107898)
    Software currently installed: OpenArbor 12.0.0 (com.ddci.openarbor.application.product 12.0.0)
    Only one of the following can be installed at once: 
        DDC-I Help Plug-in 12.4.0.r107898 (com.ddci.common.help 12.4.0.r107898)
        DDC-I Help Plug-in 12.5.0.r108644 (com.ddci.common.help 12.5.0.r108644)
Cannot satisfy dependency:
    From: DDC-I Common Feature 12.4.0.r107898 (com.ddci.common.feature.group 12.4.0.r107898)
    To: org.eclipse.equinox.p2.iu; com.ddci.common.help [12.4.0.r107898,12.4.0.r107898]
Cannot satisfy dependency:
    From: OpenArbor 12.0.0 (com.ddci.openarbor.application.product 12.0.0)
    To: org.eclipse.equinox.p2.iu; com.ddci.common.help [12.5.0.r108644,12.5.0.r108644]
  1. cd to ~/.eclipse/1418632101_linux_gtk_x86_64/features
  2. delete com.ddci.openarbortest_<version>.r<revision> folder and it's content for any version that does not match the version under test
  3. cd to ~/.eclipse/1418632101_linux_gtk_x86_64/plugins folder
  4. delete com.ddci.openarbortests_<version>.r<revision>.jar file for any version that does not match the version under test


Workaround #3:

I ran into this problem while testing an image with an unreleased OA. The first 2 workarounds did not work. What did work was making a new image using the bdu64 script.

Could not determine GDB version using command: <path to gdb>\arm-eabi-gdb --version

media:gdb-error-screenshot.png
This problem occurs when the OA Test Harness or OA Development environment is not launched from the proper cygwin environment.
For Kismet, cygwin2024 and for Jupiter cygwin 2018.

Target Boards

Use this space to document issues with setting up target boards for OA testing

Retrieved from "https://ddci.zapto.org/ddciWiki/index.php?title=OpenArbor_Integration_Testing&oldid=72834"