OpenArbor 12.1.1 Test Plan and Informal Report
From DDCIDeos
Jump to navigationJump to search
About
OpenArbor Windows/Linux 12.1.1
SVN Revision: r106287
SVN Branch Name: mainline
Test Schedule
| Scheduled Test Runs | Start Date | Due Date | Delivered | Percentage Complete |
|---|---|---|---|---|
| Formal | 04-02-2024 | tbd | tbd | 0% |
Introduction
The automated test suites for Eclipse, supports Eclipse 2023-06-R, which OpenArbor is built upon. The tests were run the on build mentioned above.
A list of PCRs corrected for this release can be found here: OpenArbor_12.1.1
See below for a list of machines that will be used to reproduce/fix test failures.
The test logs can be viewed at:
- \\nx3000\openarbor\TestResults\Release12.1.1
Legend
| Pass | The test passed. |
|---|---|
| Expected Deviation | The test fails and a fix is not going to happen due to technical restrictions or restraints beyond our control. |
| Known Failure | The test fails, but we have plans to fix the problem in a future release. |
| Fail | The test failed. |
| Not scheduled to be tested for this release. | |
| X | Scheduled to be tested for this release. |
| Not Supported. |
Test Machines
| Machine | OS | Comments | |
|---|---|---|---|
| OATESTLAP | Windows 10 (64bit) | ||
| OATEST | Windows 11 (64bit) | ||
| TESTW10E1 | Windows 10 (64bit) | ||
| TESTW10E2 | Windows 10 (64bit) | ||
| testlinux01 | Linux (64bit) | ||
| testlinux02 | Linux (64bit) | ||
| OALinux01 | Linux (64bit) | LCJ Linux development | |
| OALinux02 | Linux (64bit) | NH Linux development backup test machine |
|
| OALinux03 | Linux (64bit) | KT Linux development backup test machine |
|
| OATEST-W11VM | Windows 11 (64bit) | Unusable media:oatest-w11vm.txt |
|
| TESTW10E3 | Windows 10 (64bit) | VM, slow | |
| TESTW10E4 | Windows 10 (64bit) | Unusable media:testw10e4.txt |
|
| TESTW11E1 | Windows 11 (64bit) | Unusable media:testw11e1.txt |
|
| OATEST | Windows 7 (64bit) | Retired | |
| TESTW7P2 | Windows 7 (64bit) | Retired |
Executed Informal Test Runs
The automated OpenArbor Test Suites are executed for each entry in the table below. The "Test Suites" column will indicate what tests were run. The full set of OpenArbor tests are indicated by the name Launcher. A second test suite, CompileRunDebug is used to just test compilation, linking, running and debugging the example projects. The PlatformIntegration test suite is a small set of tests that exercise the more common operations customers use in interacting with the platform.
| Product | DDS Release | Timestamp | Target Architecture |
Available Boards |
Test Suites | Test Machine | Who | Formal Results |
Comments |
|---|---|---|---|---|---|---|---|---|---|
| Deos | Kismet Docker image for customer-shaka plus unreleased components | n/a | ARM | QEMU-ARM (qemu-arm) | Launcher | OALinux01 | LJ | X | See testing notes |
| Deos | DDS-tostones3-deos-jupiter-20240325 + stable since 3/25 + unreleased OA | n/a | ARM | QEMU-PPC (qemu-ppc) | Launcher | testw10e2 | NH | X | See testing notes |
| Deos | DDS-loewen-deos-europa-20230327 + stable since 3/27 + unreleased OA | n/a | ARM | QEMU-ARM (qemu-arm) | Launcher | oatestlap | LJ | X | See testing notes |
Unreleased Packages
The environment for testing is encoded in the bdu64-shaka script:
https://deos.ddci.com/scm/Deos/maintainer-tools/docker/branches/mainline/bdu64-shaka
I.e.,:
svn up ~/scm/Deos/maintainer-tools/docker/branches/mainline ~/scm/Deos/maintainer-tools/docker/branches/mainline/bdu64-shaka kismet customer-shaka
Testing Notes
| ID | Priority | OS | PCR | Status | Affected Tests | Target | Screenshot | Comments |
|---|---|---|---|---|---|---|---|---|
| 1 | Windows | DDCI_PCR:4964 | Known Failure | BuildAll.ada-main BuildAll.ada-shared-library BuildAll.test-call-overloading |
Jupiter QEMU-PPC (qemu-ppc) | Media:tostones3-jupiter-windows-qemu-ppc-adaMain-Instrumented-build-error-04-04-2024.txt | ada-main has build errors when built in Instrumented and InstrumentedABC configuration. Traceback (most recent call last): File "<string>", line 1, in <module> IOError: [Errno 2] No such file or directory: 'index.txt' | |
| 2 | Windows | Fail | BuildAll.cpm-partition1 Compile.cpm-partition1 Debug.cpm-partition1 Debug.cpm-partition2 Run.cpm-partition1 Run.cpm-partition2 MakeExport.cpm-partition1 |
Europa QEMU-ARM (qemu-arm) |
Media:cpm-partition1-build-errors-3-12-24.txt | cpm-partition1 has build errors. example imported from: \desk\examples\deos653\cross_partition_mode_workspace The cpm-653-configuration, cpm-partition1 and cpm-partition2 example projects were removed from the test workspace | ||
| 3 | Windows | Fail | BuildAll.dvmssimple-with-exfat | Europa QEMU-ARM (qemu-arm) Jupiter QEMU-PPC (qemu-ppc) |
Media:qemu-arm_with_dvmssimple-with-exfat_buildErrors_04-01-2024.txt | pi: Error #1: Process Template (dvms-demo1) uses feature set (dvms-exfat), which is not present.<brWith dvmssimple-with-exfat integrated, qemu-arm has build errors. example imported from: \desk\examples\dvms\dvmssimple | ||
| 4 | Windows | Fail | BuildAll.dvmsthoughput-with-jounaled-exfat | Europa QEMU-ARM (qemu-arm) Jupiter QEMU-PPC (qemu-ppc) Kismet QEMU-ARM (qemu-arm) |
Media:qemu-arm_with_dvmsthoughput-with-journaled-exfat_buildErrors_04-01-2024.txt | pi: Error #1: Process Template (dvms-demo1) uses feature set (dvms-exfat), which is not present. pi: Error #2: Process Template (dvms-demo1) uses feature set (dvms-journal), which is not present. | ||
| 5 | Windows | PCR:15247 | Known Failure | BuildAll.kernelModeCalculatorPRL | Europa QEMU-ARM (qemu-arm) | media:kernelModeCalculatorPRL-arm-build-errors-02192024.txt | Instrumented builds fail for this example make[1]: *** [/OpenArbor/openarbor-common/compiler-rules.mk:235: inst_kernelModeCalculatorPRL.s_] Error 1 | |
| 6 | Linux | DDCI_PCR:5097 | Known Failure | BuildAll.mms-partition1 BuildAll.mms-partition2 BuildAll.mms-partition3 BuildAll.mms-partition4 Run.mms-partition1 |
Kismet QEMU-ARM (qemu-arm) | Media:savianos-kismet-D4-windows-nai68int6-mms-WAT-3-13-2024.txt | In the setup for the mms examples, the WATs are configured using the platreg.pi.xml. The setup needs to be updated to support configuration for pia. The test fails because platreg.pi.xml is never found when the WAT configuration is attempted.
https://deos.ddci.com/ddcibugzilla/show_bug.cgi?id=4973 | |
| 8 | Windows | Fail | Debug.ist-partition-1 | Jupiter QEMU-PPC (qemu-ppc) | media:ist-partition-1-qemu-ppc-debug-error-04042024.png | After integrating the ist-653-to-host-configuration project into qemu-ppc and launch a debugging session, pressing the Suspend button times out. | ||
| 9 | Windows | Fail | Debug.ist-partition1 Debug.ist-partition2 Debug.udp-vs-tcp |
Jupiter QEMU-PPC (qemu-ppc) | media:ist-partition1-qemu-ppc-debug-error-04042024.png | After integrating the ist-simple-configuration project into qemu-ppc and launch a debugging session, the test times out waiting for the Confirm Perspective Switch dialog. After integrating the udp-vs-tcp project into qemu-ppc and launch a debugging session, the test times out waiting for the Confirm Perspective Switch dialog. | ||
| 10 | Linux | Fail | Debug.rtems-cpu-usage Debug.rtems-htm-partition1 Debug.rtems-htm-partition2 Debug.rtems-htm-partition3 Debug.rtems-htm-partition4 Debug.rtems-socket-example |
Kismet QEMU-ARM (qemu-arm) | media:rtems-cpu-usage-qemu-arm-debug-error-04042024.png | After integrating the hello-world-ist project into qemu-ppc and launching a debugging session, the test times out waiting for the Confirm Perspective Switch dialog. Failed to execute MI command: -target-select remote 192.168.19.17:49155 | ||
| 11 | Linux | Fail | Deos653GdbTest | Kismet QEMU-ARM (qemu-arm) | media:deos653gdb-qemu-arm-debug-error-04042024.png | Timeout after: 120000 ms.: Some thread(s) never suspended | ||
| 12 | Windows | DDCI_PCR:tbd | Fail | MulticoreIntegration | Europa QEMU-ARM (qemu-arm) | media:MulticoreIntegration-qemu-arm_error-04-01-2024.png | TimeoutException: Timeout after: 120000 ms.: Video Stream looks blank The test is attempting to Run.testNoConnect(trasherProject), but times out. | |
| 13 | Windows | DDCI_PCR:tbd | Faile | MulticoreIntegration | Jupiter QEMU-PPC (qemu-ppc) | media:MulticoreIntegration-qemu-ppc_error-04-04-2024.png | Update Target Load for the multi-threaded-process project times out. | |
| 14 | Linux | DDCI_PCR:4932 | Known Failure | MulticoreIntegration | Kismet QEMU-ARM (qemu-arm) | This test requires an update to work properly with BSPs that support pia. | ||
| 15 | Windows | DDCI_PCR:4051 | Known Failure | ProjectConverter | Europa QEMU-ARM (qemu-arm) | This test needs to be updated. | ||
| 16 | Windows | DDCI_PCR:5110 | Known Failure | Run.ada-main | Jupiter QEMU-PPC (qemu-ppc) | Media:lpb-jupiter-D4-windows-imx8qm-adaMain-videoStream-3-27-2024.png | Video Stream is blank. No exceptions seen in Status Monitor. Update: the ada-main example requires an update to correctly output to the video stream. | |
| 17 | Windows | PCR:15143 | Known Failure | Run.except | Europa QEMU-ARM (qemu-arm) | Media:loewen-d2-except-output.txt | The media link displays the output captured from running the except example. The example doesn't display all necessary output. Work around to increase budget: <threadTemplate name = "mainThreadTemplate" budget = "670"
| |
| 18 | Windows | PCR:15344 | Known Failure | Run.zlib-compress | Europa QEMU-ARM (qemu-arm) Jupiter QEMU-PPC (qemu-ppc) Kismet QEMU-ARM (qemu-arm) |
Media:zlibCompress-video-stream-output-3-12-24.txt | No Pattern Defined for zlib-compress. Output appears to be overlapping: zlib-compress example: finished!ceLen=4A52E/desk/help/deos-zlib-examples-user-guide.htm does not include the zlib-compress example. | |
| 19 | Windows | PCR:15160 | Known Failure | deos.CacheTrasher MakeExport.worker |
Europa QEMU-ARM (qemu-arm) Jupiter QEMU-PPC (qemu-ppc) Kismet QEMU-ARM (qemu-arm) |
media:cacheTrasher-qemu-arm-fail-12-18-2023.txt media:worker-qemu-arm-fail-12-18-2023.txt |
The kernel example cache-trasher's ipal and hpal projects have build errors | |
| 20 | Linux | Known Failure | deos.DeosCrittimeTest | Kismet QEMU-ARM (qemu-arm) | Timeout after: 5000 ms.: The widget ToolItem with tooltip text {Clear Crittime} was not enabled. DDS does not contain a crittime kernel /desk/ppc/appbin/crittime/libkernel.so does not exist | |||
| 21 | Linux | PCR:TBD | Fail | DeosGdbLaunchConfigurations.deosGdbLaunching DeosGdbLaunchConfigurations.deosGdbMultipleRemoteTargets |
Kismet QEMU-ARM (qemu-arm) |
media:DeosGdbLaunchConfigurations-kismet-arm-failure.png | ||
| 22 | Windows | PCR:TBD | Fail | DeosLaunchConfigurations.deosMldLaunching | Europa QEMU-ARM (qemu-arm) Jupiter QEMU-PPC (qemu-ppc) |
media:deosMldLaunching-qemu-arm-fail-04-02-2024.png | *** FAIL Tue Apr 02 08:39:46 EDT 2024: Both the Resume button and Suspend button are disabled! | |
| 23 | Windows | PCR:TBD | Fail | ExternalDebugging.noLoadListMfs ExternalDebugging.partialLoadList |
Jupiter QEMU-PPC (qemu-ppc) | media:externalDebugging-qemu-ppc-fail-04-04-2024.png | Suspend button is not enabled. | |
| 24 | Windows | PCR:TBD | Fail | PingFailure | Jupiter QEMU-PPC (qemu-ppc) | media:pingFailure-qemu-ppc-fail-04-04-2024.png | Test times out waiting for the target to ping. | |
| 25 | Windows | PCR:TBD | Fail | StatusMonitor | Europa QEMU-ARM (qemu-arm) Jupiter QEMU-PPC (qemu-ppc) |
*** FAIL Tue Apr 02 08:39:46 EDT 2024: testRTEMSTabs() cannot be completed The required project rtems-653-configuration is not in the workspace No RTEMs examples are installed in the DDS. | ||
| 25.1 | Linux | PCR:TBD | Fail | StatusMonitor | Kismet QEMU-ARM (qemu-arm) |
media:StatusMonitor-kismet-arm-failure.png | ||
| 26 | Windows | DDCI_PCR:tbd | Fail | GdbVariables.Deos653 | Europa QEMU-ARM (qemu-arm) | media:gdbVariables-deos653-arm-failure-04022024.txt media:gdbVariables-deos653-arm-resume-failure-04022024.png |
Resume after setting bp results the bp not being encountered again. | |
| 26.1 | Windows | DDCI_PCR:tbd | Fail | GdbVariables.Deos653 | Jupiter QEMU-PPC (qemu-ppc) | media:gdbVariables-deos653-ppc-failure-04042024.txt media:gdbVariables-deos653-ppc-resume-failure-04042024.png |
||
| 27 | Windows | DDCI_PCR:tbd | Fail | GdbVariables.MultipleThreadsRMA | Europa QEMU-ARM (qemu-arm) | media:gdbVariables-MultipleThreadsRMA-arm-bp-failure-04022024.txt media:gdbVariables-MultipleThreadsRMA-arm-bp-failure-04022024.png |
A resume after setting a bp results in the debugging session being terminated. Debugger console contains: ../../gdb-8.0.1/gdb/infrun.c:5575: internal-error: int finish_step_over(execution_control_state*): Assertion `ecs->event_thread->control.trap_expected' failed. A problem internal to GDB has been detected,further debugging may prove unreliable. | |
| 27.1 | Windows | DDCI_PCR:tbd | Fail | GdbVariables.MultipleThreadsRMA | Jupiter QEMU-PPC (qemu-ppc) | media:gdbVariables-MultipleThreadsRMA-ppc-bp-failure-04022024.txt media:gdbVariables-MultipleThreadsRMA-ppc-bp-failure-04022024.png |
||
| 27.2 | Links | DDCI_PCR:tbd | Fail | GdbVariables.MultipleThreadsRMA | Kismet QEMU-ARM (qemu-arm) | media:gdbVariables-MultipleThreadsRMA-kismet-arm-bp-failure-04042024.txt media:gdbVariables-MultipleThreadsRMA-kismet-arm-bp-failure-04042024.png |
||
| 28 | Windows | DDCI_PCR:tbd | Fail | GdbVariables.OneThreadRMA | Europa QEMU-ARM (qemu-arm)\ | media:gdbVariables-oneThreadRMA-arm-bp-failure-04022024.txt media:gdbVariables-oneThreadRMA-arm-bp-failure-04022024.png |
The global variables are modifed as expected after a Go and Step/Over. The function local, and thread local variables are not due to a problem with the execution pointer not being placed at the proper location after a Go when the bp is set. The 'info b' command reveals that the previous bp (at line 61) is still listed, even though it has been deleted, and the execution pointer is placed there, instead of at line 66. | |
| 28.1 | Windows | DDCI_PCR:tbd | Fail | GdbVariables.OneThreadRMA | Jupiter QEMU-PPC (qemu-ppc) | media:gdbVariables-oneThreadRMA-ppc-bp-failure-04022024.txt media:gdbVariables-oneThreadRMA-ppc-bp-failure-04022024.png |
||
| 28.2 | Links | DDCI_PCR:tbd | Fail | GdbVariables.OneThreadRMA | Kismet QEMU-ARM (qemu-arm) | media:gdbVariables-OneThreadRMA-kismet-arm-bp-failure-04042024.txt media:gdbVariables-OneThreadRMA-kismet-arm-bp-failure-04042024.png |
||
| 29 | Windows | DDCI_PCR:tbd | Fail | GdbVariables.Rtems | Europa QEMU-ARM (qemu-arm)Jupiter QEMU-PPC (qemu-ppc) | Error: Process Template (dbg-rtems) uses feature set (deos-rtems), which is not present. The /rtems/tools/bin dir is empty in the loewen DDS. RTEMs is not installed. | ||
| 29.2 | Linux | DDCI_PCR:tbd | Fail | GdbVariables.Rtems | Kismet QEMU-ARM (qemu-arm) | media:gdbVariables-Rtems-kismet-arm-bp-failure-04042024.txt media:gdbVariables-Rtems-kismet-arm-bp-failure-04042024.png |
||
| 30 | Windows | PCR:15561 | Known Failure | mld.MultiThread | Europa QEMU-ARM (qemu-arm) Jupiter QEMU-PPC (qemu-ppc) |
Media:MultiThread-qemu-arm-04-02-2024.png Media:MultiThread-qemu-arm-buttons-disabled-04-02-2024.png |
The test expects all threads in the stack frame to contain 'autoTuneWaitUntilNextPeriod'. The screenshot shows that 'autoTuneWaitUntilNextPeriod' only appears in the first 2 threads. To wait longer, the test attempts a Resume followed by a Suspend, but after the resume the both resume and suspend buttons are disabled. | |
| 31 | Windows | DDCI_PCR:tbd | Fail | mld.SharedObjectException | Europa QEMU-ARM (qemu-arm) Jupiter QEMU-PPC (qemu-ppc) |
media:SharedObjectExeption-qemu-arm-04-02-2024.png | Line 105 of the test attempts a Resume and waits for the bp to be encounted, which does not happen within the 60 second wait. | |
| 32 | Windows | PCR:15397 | Known Failure | mld.StepOperations.disassembly mld.Suspend mld.TerminateWhileRunning mld.stopPoints.MultiProcessDebugging.testMultiprocessUI |
Europa QEMU-ARM (qemu-arm) Jupiter QEMU-PPC (qemu-ppc) |
media:mld-suspend-fail-3-14-24.png | While in a debug session, these tests attempt to click the Suspend button to suspend a thread. The thread never suspends, causing a failure. | |
| 33 | Windows | Fail | mld.Ada.AdaBreakpoints mld.Ada.AdaTracepoints |
Jupiter QEMU-PPC (qemu-ppc) | ||||
| 34 | Windows | Fail | mld.commands.DebugCommands mld.commands.EnableDisableLogFile mld.commands.EnableDisableTrace mld.commands.Go mld.commands.Restart mld.commands.SetBreakAtAddressCounter mld.commands.SetSizeSetBase mld.disassembly.ShowMachineCodeAtAddress mld.disassembly.ShowMachineCodeAtSourcePosition mld.expression.ExpressionFormatting mld.memory.AddMemoryMonitor mld.memory.SearchMemory mld.memory.ShowMemory mld.stoppoints.GutterClicking mld.stoppoints.MultiProcessDebugging.testMultiprocessUI mld.stoppoints.MultiprocessDebugging.testSaveRestoreBreakpointsForMultipleProcesses mld.stoppoints.NewTracepoint mld.windows.DebuggerConsoleViewTest mld.windows.DisassemblyView mld.windows.VariablesAndExpressionsViews ExecuteSubprogram.sub13 ExecuteSubprogram.sub17 ExecuteSubprogram.sub19 ExecuteSubprogram.sub23 ExecuteSubprogram.sub27 ExecuteSubprogram.sub3 |
Jupiter QEMU-PPC (qemu-ppc) | DF-51-ERROR Unable to connect to program | |||
| 35 | Linux | DDCI_PCR:4930 | Known Failure | MinimalFileSystem.hello-world-mfs MinimalFileSystem.mms-configuration |
Kismet QEMU-ARM (qemu-arm) | This test requires an update to work properly with BSPs that support pia. | ||
| 36 | Windows | DDCI_PCR:5014 | Known Failure | other.RenameProjects | Europa QEMU-ARM (qemu-arm) Jupiter QEMU-PPC (qemu-ppc) |
Media:renameprojects-qemu-arm-12-19-2023.txt | When the qemu platform project is renamed, the launching of the emulator displays an error. | |
| 37 | Windows | Fail | RtemsLinkOptions RtemsAssemblerOptions Rtems C and Cpp Compile Options |
Jupiter QEMU-PPC (qemu-ppc) | ||||
| 38 | Windows | DDCI_PCR:2608 | Known Failure | mld.windows.VariablesAndExpressionsViews.stringEditing | Europa QEMU-ARM (qemu-arm) | media:tostones-d3-zcu102-VariablesAndExpressionsViews-stringEditing.png media:tostones-d3-zcu102-VariablesAndExpressionsViews-stringEditing-Debugger-Console.txt |
The test changes a variable value to a temporary value, which works correctly. When attempting to assign the original value back to the variable, the assignment fails, and an error is recorded in the debugger console. | |
| 39 | Windows | DDCI_PCR:tbd | Fail | RtemsLinkOptions RtemsAssembler RtemsCCompile RtemsCppCompile ProjectOptions |
Europa QEMU-ARM (qemu-arm) | The dds contains an empty /rtems/tools/bin dir. Because the directory exists, OA thinks rtems is installed and runs all these test which fail. |
