OpenArbor 12.2.1 Test Plan and Informal Report
About
OpenArbor Windows/Linux 12.2.1
SVN Revision: rTBD
SVN Branch Name: 12.2.1
Test Schedule
| Scheduled Test Runs | Start Date | Due Date | Delivered | Percentage Complete |
|---|---|---|---|---|
| Formal | 04-24-2024 | 04-24-2024 | TBD | 0% |
Introduction
The automated test suites for Eclipse, supports Eclipse 2023-06-R, which OpenArbor is built upon. The tests were run the on build mentioned above.
A list of PCRs corrected for this release can be found here: OpenArbor_12.2.1
See below for a list of machines that will be used to reproduce/fix test failures.
The test logs can be viewed at:
- \\nx3000\openarbor\TestResults\Release12.2.1
Legend
| Pass | The test passed. |
|---|---|
| Expected Deviation | The test fails and a fix is not going to happen due to technical restrictions or restraints beyond our control. |
| Known Failure | The test fails, but we have plans to fix the problem in a future release. |
| Fail | The test failed. |
| Not scheduled to be tested for this release. | |
| X | Scheduled to be tested for this release. |
| Not Supported. |
Test Machines
| Machine | OS | Comments | |
|---|---|---|---|
| OATESTLAP | Windows 10 (64bit) | ||
| OATEST | Windows 11 (64bit) | ||
| TESTW10E1 | Windows 10 (64bit) | ||
| TESTW10E2 | Windows 10 (64bit) | ||
| testlinux01 | Linux (64bit) | ||
| testlinux02 | Linux (64bit) | ||
| OALinux01 | Linux (64bit) | LCJ Linux development | |
| OALinux02 | Linux (64bit) | NH Linux development backup test machine |
|
| OALinux03 | Linux (64bit) | KT Linux development backup test machine |
|
| OATEST-W11VM | Windows 11 (64bit) | Unusable media:oatest-w11vm.txt |
|
| TESTW10E3 | Windows 10 (64bit) | VM, slow | |
| TESTW10E4 | Windows 10 (64bit) | Unusable media:testw10e4.txt |
|
| TESTW11E1 | Windows 11 (64bit) | Unusable media:testw11e1.txt |
|
| OATEST | Windows 7 (64bit) | Retired | |
| TESTW7P2 | Windows 7 (64bit) | Retired |
Executed Formal Test Runs
The automated OpenArbor Test Suites are executed for each entry in the table below. The "Test Suites" column will indicate what tests were run. The full set of OpenArbor tests are indicated by the name Launcher. A second test suite, CompileRunDebug is used to just test compilation, linking, running and debugging the example projects. The PlatformIntegration test suite is a small set of tests that exercise the more common operations customers use in interacting with the platform.
Unreleased Packages
The environment for testing is encoded in the bdu64-shaka script:
https://deos.ddci.com/scm/Deos/maintainer-tools/docker/branches/mainline/bdu64-shaka
I.e.,:
svn up ~/scm/Deos/maintainer-tools/docker/branches/mainline ~/scm/Deos/maintainer-tools/docker/branches/mainline/bdu64-shaka kismet customer-shaka
Testing Notes
| Product | DDS Release | Timestamp | Target Architecture |
Available Boards |
Test Suites | Test Machine | Who | Formal Results |
Comments |
|---|---|---|---|---|---|---|---|---|---|
| Deos | DDS-sales-deos-kismet-20240509 | TBD | QEMU-ARM (qemu-arm) | Arm | Launcher | testw10e2 | lcj | Fail | 64 Failures of 668 tests run See testing notes |
| ID | Priority | OS | PCR | Status | Affected Tests | Target | Screenshot | Comments |
|---|---|---|---|---|---|---|---|---|
| 1 | Windows | PCR:15247 | Known Failure | BuildAll.kernelModeCalculatorPRL | QEMU-ARM (qemu-arm) | media:kernelModeCalculatorPRL-arm-build-errors-02192024.txt | Instrumented builds fail for this example make[1]: *** [/OpenArbor/openarbor-common/compiler-rules.mk:235: inst_kernelModeCalculatorPRL.s_] Error 1 | |
| 2 | Windows | PCR:TBD | Fail | Building of several examples NewProjectValidation.checkNonProjectFolderInWorkspace VarianSpecificFiles.variantItFilesForExeProject |
QEMU-ARM (qemu-arm) | media:examples-arm-build-warning-05122024.txt | Several examples are failing with this warning: 10: make[1]: warning: undefined variable 'GNUMAKEFLAGS' See media link for complete list (110 examples) | |
| 3 | Windows | PCR:TBD | Fail | TRAC docs are installed other.HelpInstalled |
QEMU-ARM (qemu-arm) | media:trac-docs-installed-05122024.png | TRAC is no longer being delivered in kismet DDS', but some TRAC docs are being installed | |
| 4 | Windows | PCR:TBD | Fail | BuildAll.rtems-cpu-usage BuildAll.rtems-hm-partition1 BuildAll.rtems-partition1 BuildAll.rtems-partition2 BuildAll.rtems-partition3 BuildAll.rtems-partition4 Debug.arinc653-partition |
QEMU-ARM (qemu-arm) | media:rtems-cpu-usage-arm-build-error-05132024.txt | Rtems examples fail to build Error: undefined reference to 'getTimeBias' | |
| 5 | Windows | PCR:TBD | Fail | BuildAll.rtems-socket-example Debug.rtems-socket-example Run.rtems-socket-example MakeExport.rtems-socket-example |
QEMU-ARM (qemu-arm) | media:rtems-socket-example-arm-build-error-05132024.txt | rtems-socket-example fails to build Error: ld: cannot find -ldeossocket | |
| 6 | Windows Linux |
DDCI_PCR:4932 | Known Failure | MulticoreIntegration | QEMU-ARM (qemu-arm) | This test requires an update to work properly with BSPs that support pia. | ||
| 7 | Windows | DDCI_PCR:4051 | Known Failure | ProjectConverter | QEMU-ARM (qemu-arm) | Unable to determine code name for DDS This test needs to be updated. | ||
| 8 | Windows | PCR:15344 | Known Failure | Run.zlib-compress | QEMU-ARM (qemu-arm) | Media:zlib-compress-videos-stream-05132024.txt | No Pattern Defined for zlib-compress. Output appears to be overlapping: zlib-compress example: finished!ceLen=4A52E/desk/help/deos-zlib-examples-user-guide.htm does not include the zlib-compress example. | |
| 9 | Windows | PCR:n/a | Known Failure | Debug.SuperMakefile | QEMU-ARM (qemu-arm) | Media:n/a | The SuperMakefile test fails because examples in the workspace fail to build. Test needs to be re-run when items #2, #4 and #5 are resolved. | |
| 10 | Windows | PCR:14080 | Known Failure | CacheTrasher MakeExport.worker |
QEMU-ARM (qemu-arm) | Cache Trashing is not supported on the platform. | ||
| 11 | Windows | PCR:TBD | Fail | deos.ConfigureRapita | QEMU-ARM (qemu-arm) | media:TBD | Timeout after: 500ms.: Problems view does not contain\QNo RVS data is available from Critical Time Data\E | |
| 12 | Windows | PCR:TBD | Known Failure | deos.DeosCrittimeTest | QEMU-ARM (qemu-arm) | Timeout after: 5000 ms.: The widget ToolItem with tooltip text {Clear Crittime} was not enabled. DDS does not contain a crittime kernel /desk/ppc/appbin/crittime/libkernel.so does not exist | ||
| 13 | Windows | PCR:TBD | Known Failure | deos.PingFailure | QEMU-ARM (qemu-arm) | Test fails due to 'Target has not responded ..." dialog is not showing up. Tested manually and it works ok. possible test update is needed. | ||
| 14 | Windows | DDCI_PCR:5115 | known Failure | gdb.GdbVariables.Deos653 gdb.GdbVariables.MultipleThreadsRMA gdb.GdbVariables.OneThreadsRMA |
QEMU-ARM (qemu-arm) | media:TBD | Setting of bp is inconsistent with the test. | |
| 15 | Windows | DDCI_PCR:5014 | Known Failure | other.RenameProjects | QEMU-ARM (qemu-arm) | Media:renameprojects-qemu-arm-12-19-2023.txt | When the qemu platform project is renamed, the launching of the emulator displays an error. |
