OpenArbor Integration Testing

Introduction

The OpenArbor Test Harness eclipse application provides an interface for executing the OpenArbor automated test suites as well as individual OpenArbor automated tests.
This is a high-level overview of how to accomplish different types of testing use the OpenArbor test harness. More detailed documentation can be found at OpenArbor Testing. The two steps are described below:

1. Choose your test machine setup
2. Determine the type of Testing Session you will be performing

Test Machine Setup

The types of available testing sessions are described in the next section. To use the OpenArbor Test Harness, it can be installed and built on your machine or you can connect to one of the available OpenArbor test machines. Both options are described below.

Run the OA test harness on your machine:

1. 1. Follow the client setup instructions.

Use an OpenArbor test machine:

1. 1. Pick one from the list.
   2. Reserve the test machine on X9
   3. For a Windows test machine, Connect to it using vnc. For a Linux test machine, connect via Remote Destop Connection (RDC). If you are unable to connect, contact one of the OpenArbor team members for help. While unlikely, it is possible the test machine may not be setup. The OpenArbor team member can evaluate it the test harness is not running or if the test machine is not setup properly. If so:
      1. Follow the server instructions.
      2. Follow the client setup instructions and then connect to the server.
   4. If not running, launch the test harness on the test machine.
      1. Windows:cd /cygdrive/c/ddci_eclipse_test_harness_dev/dds-manager-e4 && ./oaTestHarness.sh -n.
      2. Docker:cd ~ddci_eclipse_test_harness_dev/dds-manager-e4 && ./oaTestHarness.sh -n.

Types of Testing Sessions

1. Follow the instructions below for the type of testing you need to do (Official DDS?), and then:
   1. Add a config.
      1. If you are trying to test a board with no official config, just pick one that is close to what you are trying to test and then make the necessary modifications.
      2. If using an OpenArbor test machine, you will need to lock the x9 target as yourself instead of as scoretest.
         
      3. To ensure your test machine setup is correct, you can execute the Run test for the hello-world example. To do this, specify com.ddci.openarbortests.compileRunDebug.Run.hello-world in the Tests field.
      4. You probably don't need to test all of OpenArbor, so use one of the subsets instead. To do this, specify the subset's name in the Tests field. The available subsets are:
         
         1. loadABunchOfTests: Mostly project option tests, and some miscellaneous stuff like the Ada path dialog and the dumpers/disassembler.
         2. loadCompileRunDebug: Ensures that all of the examples can compile, run, and debug successfully. This can be a good BSP smoketest.
         3. loadRtemsTests: Exercises the RTEMS support functionality in the OpenArbor GUI.
         4. loadDeosTests: Exercises a lot of Deos-specific functionality in the OpenArbor GUI.
         5. loadGdb: This is a meta-grouping that exercises some aspect of Gdb's interaction with OpenArbor.
         6. loadMldTests: This is a meta-grouping that exercises every aspect of the MLD's interaction with OpenArbor.
            1. loadDebugCommands: Commands that manipulate the output of the MLD.
            2. loadExpression: How objects and expressions are displayed and formatted.
            3. loadStopPoints: Breakpoint and tracepoint tests.
            4. loadWindows: Tests each view in the debugging perspective.
            5. loadDisassembly: Exercises the disassembly view.
            6. loadSubprograms: Tests running various types of subprograms through the MLD.
            7. loadAda: Ada-specific debugger functionality.
            8. loadMemory: Various methods of exploring and manipulating memory regions.
         7. loadOtherTests: A miscellaneous grouping that verifies minimum OA functionality.
         8. loadPlatformIntegration: A minimal subset specifically targeted at smoketesting a complete DDS.
            1. loadPlatformIntegrationLegacy: Like above, minus tests that don't exist on the elbert branch.
         9. loadTargetOuptutTests: Performs some basic run and debug testing.
         10. com.ddci.openarbortests.deos.BuildBspDevKit: Builds an installed dev kit from it's zip file.
   2. Click the Test button and wait for the results.

Testing Specific Examples

The "Examples to test exclusively" checkboxes on the DDS Manager view contains a checkbox for each category of examples installed in the DDS. Developers can select one (or more) example categories to be tested. When checked and the Test button is pressed, the examples in each category will be imported into the test workspace, and the following tests will be executed for each example: Compile, Run, Debug.

1. Check one (or more) categories of examples to test media:ExampleCategoryChecked.png
2. Click the Test button

The Log Viewer tab, keeps a count of the testing progress. The number of Failed, Passed and Unrun tests are displayed as well as the Total number of tests. When 'Unrun' is 0, then all the tests requested have been run!

The Log Viewer also shows the results for each test executed. Clicking on a test name will cause the log for the test to also be displayed. When failures occur, the test will be listed in red (and in the Fail drop-down list). An OpenArbor team member can be helpful in understanding what the failures mean.

Testing an Official DDS

An Official DDS is one that has been (1) QA'd and shipped to a customer or (2) created for formal testing on DDSBUILD. The Add a dds provides location selection choices of official DDS'.

1. To select an Official DDS, right-click on the machine name (e.g localhost) and select Add a dds.
2. Click the Install button to install the DDS onto the test machine.
3. The installation is complete when the DDS Manager View displays: "ddciFlex is Up!" and the hourglass is no longer visible for the test machine.
4. If you need to modify the dds, you will need follow the instructions below to add unreleased components before testing.

Testing with Unreleased Components

Windows

1. Click the Run Desk Setup button, select Install From Internet, and click through to select the unreleased components and install them.

Docker

1. Create a docker image containing the unreleased components using the bdu64 script (or something similar)
2. Replace the offical dds' docker image with the one you've created

Testing with Local Builds (Windows only)

1. Now you can make any local changes that you need, such as installing a test version of a component. The dds installations are located in <workspace>/DDS-list/<dds>/.
   1. For example, if you wanted to test a new bsp, you would replace the matching bsp in <workspace>/DDS-list/<dds>/desk/platform/<bsp> before starting a test run.

Testing Existing Installations

An existing installation is a DDS that was not installed by the OpenArbor Test Harness. Use Add DDS->Add an existing installation..., if you have a DDS installed on your Windows machine, or an image created on your Docker workstation that you'd like to use for running OpenArbor automated tests. Once added, the test workspace can be modified to specify Deos Search Path directories for Deos components that are 'in work' (e.g. not unreleased or stable).

Windows

1. Right-click on localhost and select Add DDS->Add an existing installation....
2. Use the File dialog to select the directory containing an installed DDS
   1. Note that test data is stored by the name of the location's directory, e.g. C:/foo/bar will be tested out of C:/workspace/ddses/bar. This means that adding C:/baz/bar would collide.
3. Right-click on the dds name added (e.g. C:/foo/bar), and select Add Test Config.
4. Select the config for the target being tested (e.g. deos_arm_qemu)

Docker

1. Right-click on localhost and select Add DDS->Add an existing installation....
2. The Docker Image List Dialog is displayed media:docker-image-list-dialog.png
3. Use the Select docker image drop-down list to select an image to use for testing (e.g. dds-docker-training-deos-kismet-20240524:latest)
   1. Note that test data is stored by the name of the image, e.g. dds-docker-training-deos-kismet-20240524:latest will be tested out of ~/workspace/ddses/dds-docker-training-deos-kismet-20240524_latest.
4. Right-click on the dds name added (e.g. dds-docker-training-deos-kismet-20240524:latest), and select Add Test Config.
5. Select the config for the target being tested (e.g. deos_arm_qemu)

(Optional) Define Deos Search Paths in the testing workspace

The keepSearchPaths config setting is only supported in OpenArbor 12.3.0+

1. 1. Right-click on the config, and select Modify Config....
   2. In the Input Dialog add "keepSearchPaths=true", and click OK.
   3. On the DDS Manager tab view, click the Run for Debugging button
   4. Once OA has launched, use the Windows | Preferences | DDCI | Deos Search Path dialog to add paths for components 'in work' to be tested.
   5. Apply and close the Preferences dialog
   6. Close OA