OpenArbor Integration Testing

This is a high-level overview of how to accomplish different types of testing use the OpenArbor test harness. More detailed documentation can be found at OpenArbor Testing.

Getting Started

1. Use a test machine: (recommended)
   1. Pick one from the list.
   2. Reserve the test machine on X9
   3. Connect to it using vnc. If you are unable to connect, contact one of the OpenArbor team members for help. While unlikely, it is possible the test machine may not be setup. The OpenArbor team member can evaluate it the test harness is not running or if the test machine is not setup properly. If so:
      1. Follow the server instructions.
      2. Follow the client setup instructions and then connect to the server.
   4. If not running, launch the test harness on the test machine. cd /cygdrive/c/ddci_eclipse_test_harness_dev/dds-manager-e4 && ./oaTestHarness.sh -n.
2. Follow the instructions below for the type of testing you need to do (Official DDS?), and then:
   1. Add a config.
      1. If you are trying to test a board with no official config, just pick one that is close to what you are trying to test and then make the necessary modifications.
      2. You may also wish to lock the x9 target as yourself instead of as scoretest.
      3. You probably don't need to test all of OpenArbor, so use one of the subsets instead. To do this, specify the subset's name in the Tests field. The available subsets are:
         1. loadABunchOfTests: Mostly project option tests, and some miscellaneous stuff like the Ada path dialog and the dumpers/disassembler.
         2. loadCompileRunDebug: Ensures that all of the examples can compile, run, and debug successfully. This can be a good BSP smoketest.
         3. loadDeosTests: Exercises a lot of Deos-specific functionality in the OpenArbor GUI.
         4. loadMldTests: This is a meta-grouping that exercises every aspect of the MLD's interaction with OpenArbor.
            1. loadDebugCommands: Commands that manipulate the output of the MLD.
            2. loadExpression: How objects and expressions are displayed and formatted.
            3. loadStopPoints: Breakpoint and tracepoint tests.
            4. loadWindows: Tests each view in the debugging perspective.
            5. loadDisassembly: Exercises the disassembly view.
            6. loadSubprograms: Tests running various types of subprograms through the MLD.
            7. loadAda: Ada-specific debugger functionality.
            8. loadMemory: Various methods of exploring and manipulating memory regions.
         5. loadOtherTests: A miscellaneous grouping that verifies minimum OA functionality.
         6. loadPlatformIntegration: A minimal subset specifically targeted at smoketesting a complete DDS.
            1. loadPlatformIntegrationLegacy: Like above, minus tests that don't exist on the elbert branch.
         7. loadTargetOuptutTests: Performs some basic run and debug testing.
         8. com.ddci.openarbortests.deos.BuildBspDevKit: Builds an installed dev kit from it's zip file.
   2. Click the Test button and wait for the results.
3. If you prefer to run tests locally, you will need to follow the client setup instructions.

Testing Specific Examples

The "Examples to test exclusively" checkboxes on the DDS Manager view contains a checkbox for each category of examples installed in the DDS. Developers can select one (or more) example categories to be tested. When checked and the Test button is pressed, the examples in each category will be imported into the test workspace, and the following tests will be executed for each example: Compile, Run, Debug.

1. Check one (or more) categories of examples to test media:ExampleCategoryChecked.png
2. Click the Test button

The Log Viewer tab, keeps a count of the testing progress. The number of Failed, Passed and Unrun tests are displayed as well as the Total number of tests. When 'Unrun' is 0, then all the tests requested have been run!

The Log Viewer also shows the results for each test executed. Clicking on a test name will cause the log for the test to also be displayed. When failures occur, the test will be listed in red (and in the Fail drop-down list). An OpenArbor team member can be helpful in understanding what the failures mean.

Testing an Official DDS

1. Add a dds.
2. Click the Install button to install the DDS onto the test machine.
3. The installation is complete when the DDS Manager View displays: "ddciFlex is Up!" and the hourglass is no longer visible for the test machine.
4. If you need to modify the dds, you will need follow the instructions below to add unreleased components before testing.

Testing with Unreleased Components

1. Click the Run Desk Setup button, select Install From Internet, and click through to select the unreleased components and install them.

Testing with Local Builds

1. Now you can make any local changes that you need, such as installing a test version of a component. The dds installations are located in <workspace>/DDS-list/<dds>/.
   1. For example, if you wanted to test a new bsp, you would replace the matching bsp in <workspace>/DDS-list/<dds>/desk/platform/<bsp> before starting a test run.

Testing Existing Installations

Windows

1. Right-click on localhost and select Add DDS->Add an existing installation....
2. Use the File dialog to select the directory containing an installed DDS
   1. Note that test data is stored by the name of the location's directory, e.g. C:/foo/bar will be tested out of C:/workspace/ddses/bar. This means that adding C:/baz/bar would collide.

Linux

1. Right-click on localhost and select Add DDS->Add an existing installation....
2. The Docker Image List Dialog is displayed media:docker-image-list-dialog.png
3. Use the Select docker image drop-down list to select an image to use for testing