OpenArbor 12.2.1 Test Plan and Report SanTan K1
About
OpenArbor Windows/Linux 12.2.1
SVN Revision: r106478
SVN Branch Name: mainline
Test Schedule
| Scheduled Test Runs | Start Date | Due Date | Delivered | Percentage Complete |
|---|---|---|---|---|
| Formal | 04-26-2024 | 05-6-2024 | 05-6-2024 | 100% |
Introduction
The automated test suites for Eclipse, supports Eclipse 2023-06-R, which OpenArbor is built upon. The tests were run the on build mentioned above.
A list of PCRs corrected for this release can be found here: OpenArbor_12.2.1
See below for a list of machines that will be used to reproduce/fix test failures.
The test logs can be viewed at:
- \\nx3000\openarbor\TestResults\Release12.2.1
Legend
| Pass | The test passed. |
|---|---|
| Expected Deviation | The test fails and a fix is not going to happen due to technical restrictions or restraints beyond our control. |
| Known Failure | The test fails, but we have plans to fix the problem in a future release. |
| Fail | The test failed. |
| Not scheduled to be tested for this release. | |
| X | Scheduled to be tested for this release. |
| Not Supported. |
Test Machines
| Machine | OS | Comments | |
|---|---|---|---|
| OATESTLAP | Windows 10 (64bit) | ||
| OATEST | Windows 11 (64bit) | ||
| TESTW10E1 | Windows 10 (64bit) | ||
| TESTW10E2 | Windows 10 (64bit) | ||
| testlinux01 | Linux (64bit) | ||
| testlinux02 | Linux (64bit) | ||
| OALinux01 | Linux (64bit) | LCJ Linux development | |
| OALinux02 | Linux (64bit) | NH Linux development backup test machine |
|
| OALinux03 | Linux (64bit) | KT Linux development backup test machine |
|
| OATEST-W11VM | Windows 11 (64bit) | Unusable media:oatest-w11vm.txt |
|
| TESTW10E3 | Windows 10 (64bit) | VM, slow | |
| TESTW10E4 | Windows 10 (64bit) | Unusable media:testw10e4.txt |
|
| TESTW11E1 | Windows 11 (64bit) | Unusable media:testw11e1.txt |
|
| OATEST | Windows 7 (64bit) | Retired | |
| TESTW7P2 | Windows 7 (64bit) | Retired |
Executed Formal Test Runs
The automated OpenArbor Test Suites are executed for each entry in the table below. The "Test Suites" column will indicate what tests were run. The full set of OpenArbor tests are indicated by the name Launcher. A second test suite, CompileRunDebug is used to just test compilation, linking, running and debugging the example projects. The PlatformIntegration test suite is a small set of tests that exercise the more common operations customers use in interacting with the platform.
| Product | DDS Release | Timestamp | Target Architecture |
Available Boards |
Test Suites | Test Machine | Who | Formal Results |
Comments |
|---|---|---|---|---|---|---|---|---|---|
| Deos | DDS-santan-deos-kismet-20240426 | 1714163107 | ARM | QEMU-ARM(qemu-arm) | Launcher | testw10e1 | NH,KT | Known Failure | See Testing Notes |
| Deos | DDS-santan-deos-kismet-20240427 | 1714252132 | ARM | DeosMCP1048-1(santan-ls1048a) | PlatformIntegration | testw10e2 | NH | Known Failure | See Testing Notes |
| Deos | DDS-docker-santan-deos-kismet-20240427 | 1714252132 | ARM | DeosMCP1048-1(santan-ls1048a) | PlatformIntegration | oalinux02 | NH | Known Failure | See Testing Notes |
| Deos | DDS-santan-deos-kismet-20240427 | 1714252132 | ARM | DeosMCP1048-1(santan-ls1048a) | Manual test with SATA MAL if not covered above | testw10e2 oalinux02 |
NH | Pass | DvmsMal test updated to include santan-ls1048a. The test passed. |
Testing Notes
| ID | Priority | OS | PCR | Status | Affected Tests | Target | Screenshot | Comments |
|---|---|---|---|---|---|---|---|---|
| 1 | Windows | PCR:15247 | Known Failure | BuildAll.kernelModeCalculatorPRL | QEMU-ARM (qemu-arm) | Media:santan-kismet-k1-windows-qemuarm-kernelModeCalculator-BuildError-instrumentedABC-5-2-2024.txt | project has build errors in Instrumented and ABC build modes. make[1]: *** [/OpenArbor/openarbor-common/compiler-rules.mk:235: inst_kernelModeCalculatorPRL.s_] Error 1 | |
| 2 | Windows Linux |
PCR:15344 | Known Failure | Run.zlib-compress | QEMU-ARM (qemu-arm) DeosMCP1048-1 (santan-ls1048a) |
Media:sales-jupter-24a-qemu-ppc-run-zlib-compress-video-stream.png | No Pattern Defined for zlib-compress. Output appears to be overlapping: zlib-compress example: finished!ceLen=4A52E/desk/help/deos-zlib-examples-user-guide.htm does not include the zlib-compress example.
| |
| 3 | Windows Linux |
DDCI_PCR:4932 | Known Failure | MulticoreIntegration | QEMU-ARM (qemu-arm) DeosMCP1048-1 (santan-ls1048a) |
This test requires an update to work properly with BSPs that support pia. | ||
| 4 | Windows | DDCI_PCR:4051 | Known Failure | ProjectConverter | QEMU-ARM (qemu-arm) | Unable to determine code name for DDS This test needs to be updated. | ||
| 5 | Windows | Expected Deviation | DeosLaunchConfigurations.commonTabs DeosLaunchConfigurations.deosMldLaunching |
QEMU-ARM (qemu-arm) | MLD is not installed in the DDS | |||
| 6 | Windows | DDCI_PCR:4930 | Known Failure | MinimalFileSystem.hello-world-mfs MinimalFileSystem.mms-configuration |
QEMU-ARM (qemu-arm) | This test requires an update to work properly with BSPs that support pia. | ||
| 7 | Windows | DDCI_PCR:5144 | Known Failure | RtemsLinkOptions RtemsAssembler RtemsCCompile RtemsCProcessor RtemsCppCompile ProjectOptions |
QEMU-ARM (qemu-arm) | The dds contains an empty /rtems/tools/bin dir. Because the directory exists, OA thinks rtems is installed and runs all these test which fail.
This entry was taken from the following report, no pcr was listed for this issue: https://deos.ddci.com/ddciWiki/OpenArbor_12.1.1_Test_Plan_and_Informal_Report 5-6-24 Update: The mentioned directory was found to not be installed after a fresh install, so OA needs to determine if testing is causing the dir to be created. | ||
| 8 | Windows | PCR:15131 | Known Failure | deos.DdsInstallerTest | DeosMCP1048-1 (santan-ls1048a) | Media:santan-k1-windows-DdsInstallerTest-fail-5-3-2024.txt | The installer reports: "mount: /tmp: Invalid argument" The "mount" issue is a longstanding problem. I could add a filter rule for it, but some solution is probably warranted, I just don't know what it would be. | |
| 9 | Windows | DDCI_PCR:5014 | Known Failure | other.RenameProjects | QEMU-ARM (qemu-arm) | Media:renameprojects-qemu-arm-12-19-2023.txt | When the qemu platform project is renamed, the launching of the emulator displays an error.
Media is from entry 36 of previous report: https://deos.ddci.com/ddciWiki/OpenArbor_12.2.0_Test_Plan_and_Informal_Report | |
| 10 | Windows | PCR:14080 | Known Failure | CacheTrasher MakeExport.worker |
QEMU-ARM (qemu-arm) | Cache Trashing is not supported on the platform. | ||
| 11 | Windows | Pass | Makehash | QEMU-ARM (qemu-arm) | The test passes manually. In the test's current state, the strings used to verify the 'Perform makehash after building' checkbox in 2 lines of the code need to be changed, as a '*' is appended as a prefix to the checkbox name at the time of the check. Also, 'Authorizing with Eclipse.org' popups occur during the test, disrupting the test flow causing waitUntil()'s to fail. After changing the 2 checks, and clicking through the authorization pop ups, the test flow passes. KT is updating the test for the 2 incorrect checks, need to talk to team about the popups.
Update: Committed checkbox name fix to 12.2.1 release management. Discovered how to prevent dialog popup. | |||
| 12 | Windows | Expected Deviation | DeosCrittimeTest | QEMU-ARM (qemu-arm) | The test is failing when it checks HYPSTART_loadlist.txt to verify that the .txt file contains "appbin/crittime/libkernel.so". Here is the line from HYPSTART_loadlist.txt that is no longer matching: -kernelFile=libkernel.so,/desk/arm/appbin/libkernel.so crittime kernel is not provided in kismet yet | |||
| 13 | Windows | DDCI_PCR:5151 | Known Failure | HelpInstalled | QEMU-ARM (qemu-arm) | The test 'thinks' TRAC is installed, and therefore incorrectly failed when it did not find TRAC documents in the 'Help Contents'. The core issue is that the way it checks if TRAC is installed is by looking for "PC_WNT_TRAC" in the license.lic, which is present, although TRAC was removed from kismet.
Therefore, the test harness code that creates the license.lic needs to be improved to ensure TRAC is not included in the license if it has been removed. | ||
| 14 | Windows | PCR:tbd | Known Failure | GdbVariables | QEMU-ARM (qemu-arm) | The test fails occurring appear to be issues caused by the GdbVariables tests not accounting for some differences in timing between threads, and optimizations by the compiler, and not anything wrong with gdb. The OA team should have a meeting on how to fix these for the future, and write a PCR for said fixes. |
