OpenArbor 12.3.0 Test Plan and Informal Report

From DDCIDeos
Jump to navigationJump to search


About

OpenArbor Windows/Linux 12.3.0
SVN Revision: rTBD
SVN Branch Name: 12.3.0

Test Schedule

Scheduled Test Runs Start Date Due Date Delivered Percentage Complete
Formal 06-12-2024 06-14-2024 6-17-2024 100%

Introduction

The automated test suites for Eclipse, supports Eclipse 2023-06-R, which OpenArbor is built upon. The tests were run the on build mentioned above.

A list of PCRs corrected for this release can be found here: OpenArbor_12.3.0

See below for a list of machines that will be used to reproduce/fix test failures.

The test logs can be viewed at:

\\nx3000\openarbor\TestResults\Release12.3.0

Legend

Pass The test passed.
Expected Deviation The test fails and a fix is not going to happen due to technical restrictions or restraints beyond our control.
Known Failure The test fails, but we have plans to fix the problem in a future release.
Fail The test failed.
Not scheduled to be tested for this release.
X Scheduled to be tested for this release.
Not Supported.

Test Machines

Machine OS Comments
OATESTLAP Windows 10 (64bit)
OATEST Windows 11 (64bit)
TESTW10E1 Windows 10 (64bit)
TESTW10E2 Windows 10 (64bit)
testlinux01 Linux (64bit)
testlinux02 Linux (64bit)
OALinux01 Linux (64bit) LCJ Linux development
OALinux02 Linux (64bit) NH Linux development
backup test machine
OALinux03 Linux (64bit) KT Linux development
backup test machine
OATEST-W11VM Windows 11 (64bit) Unusable
media:oatest-w11vm.txt
TESTW10E3 Windows 10 (64bit) VM, slow
TESTW10E4 Windows 10 (64bit) Unusable
media:testw10e4.txt
TESTW11E1 Windows 11 (64bit) Unusable
media:testw11e1.txt
OATEST Windows 7 (64bit) Retired
TESTW7P2 Windows 7 (64bit) Retired

Executed Formal Test Runs

The automated OpenArbor Test Suites are executed for each entry in the table below. The "Test Suites" column will indicate what tests were run. The full set of OpenArbor tests are indicated by the name Launcher. A second test suite, CompileRunDebug is used to just test compilation, linking, running and debugging the example projects. The PlatformIntegration test suite is a small set of tests that exercise the more common operations customers use in interacting with the platform.

Unreleased Packages

The Sales 24A wiki page lists the unreleased components that need to be included in the testing.

Windows

The environment for testing was accomplished by using the cygwin installer to install the unreleased packages:media:UnreleasedPackages.png

Linux

The environment for testing was encoded in the bdu64 script:media:bdu64_script_content.txt

Testing Notes

Product DDS Release Timestamp Target
Architecture		 Available
Boards		 Test Suites Test Machine Who Formal
Results		 Comments
Deos DDS-docker-sales-deos-kismet-20240611 TBD QEMU-ARM (qemu-arm) Arm Launcher testlinux02 lcj Fail Failed: 32 Passed: 658 Unrun: 0 Total: 690 Not Allowed: 43
See testing notes
Deos DDS-sales-deos-kismet-20240611 TBD QEMU-AARCH64 (qemu-aarch64) AARCH64 Launcher oatest lcj Known Failure See testing notes
Deos DDS-sales-deos-kismet-20240611 TBD QEMU-PPC (qemu-ppc) PowerPC Launcher testw10e2 lcj Known Failure See testing notes
ID Priority OS PCR Status Affected Tests Target Screenshot Comments
1 Aaron Windows PCR:15743 Known Failure deos.DdsInstaller
other.DeosOptionTabs.integrationToolTabTest 		QEMU-AARCH64 (qemu-aarch64) media:setup_log_full.txt Several post install scripts report unxpected errors.
Of particular note these two:

/etc/postinstall/0p_100_update_desk_xml_files.sh: /etc/postinstall/python3-argcomplete.sh: line 9: python3: command not found
/etc/postinstall/desk.sh: grep: /etc/fstab: No such file or directory
sed: can't read /etc/fstab: No such file or directory
/bin/mv: cannot stat '/etc/fstab': No such file or directory
mount: /tmp: Invalid argument
A subdirectory or file C:\ProgramData\Microsoft\Windows\Start Menu\Programs/DDC-I/oaTestHarness/workspace/DDS-list/DDS-sales-deos-kismet-20240611 already exists.

2 Chris Linux (qemu-arm) DDCI_PCR:5177 Known Failure Building project: procfs-example QEMU-ARM (qemu-arm)
QEMU-AARCH64 (qemu-aarch64)
QEMU-PPC (qemu-ppc) 		media:profs-example-build-output-20240612.txt The vfile procfs-example reports a warning when built:warning: undefined variable 'vfileconfig_arch_switch'
With input from CP, wrote and fixed PCR 5177
2.5 Ron TBD BSP team member to resolve 64-bit PSIO address. Windows (qemu-aarch64) PCR:15192 Known Failure Building project: psio-print QEMU-AARCH64 (qemu-aarch64) media:psio-print-example-aarch64-build-output-20240614.txt The following unreleases should resolve this:

bsp-examples installed 1.6.0-2
qemu-aarch64-boot installed 1.0.1-1

../../../code/psio-print.cpp: In function 'int main()':
../../../code/psio-print.cpp:29:53: error: cast from 'const PlatformSpecificInfoTYP*' {aka 'const PlatformSpecificInfo*'} to 'uint32_t' {aka 'unsigned int'} loses precision [-fpermissive]

2.75 Ron TBD BSP team member to address PCR. Windows
Linux 		PCR:15532 Known Failure Building project: one-shot-timer QEMU-ARM (qemu-arm)
QEMU-AARCH64 (qemu-aarch64)
QEMU-PPC (qemu-ppc) 		media:one-shot-timer-example-aarch64-build-output-20240614.txt ../../../code/one-shot-timer.cpp:18:10: fatal error: pal-exported-functions.h: No such file or directory
Per AR, this example should only be tested with the jacinto7evm bsp.
Documenting here that it doesn't build 'out-of-the-box', and it was removed

from the test workspace.
The loewen bsp is not included in the Sales DDS, so it seems strange that this example is installed in the desk.
LJ modified the oatestharness to not import the one-shot-timer example if not testing the loewen bsp.

3 Linux (qemu-arm) PCR:15532 Known Failure BuildAll.ext-clock-sync
Debug.ext-clock-sync
Run.ext-clock-sync 		QEMU-ARM (qemu-arm)
QEMU-AARCH64 (qemu-aarch64)
QEMU-PPC (qemu-ppc) 		Media:qemu-arm-with-ext-clock-sync-integrated-build-output-20240612.txt This example should not be executed on qemus.

pi: Error #1: Process Instance (ext-clock-sync) owns undefined memory mapped resource (extClockShift)
Per AR, qemus do not support clock synch
LJ modified the oatestharness to not import the example when testing qemus.

4 Linux (qemu-arm) PCR:15247 Known Failure BuildAll.kernelModeCalculatorPRL QEMU-ARM (qemu-arm) Media:kernelModeCalculatorPRL-BuildError-instrumentedABC-20240612.txt project has build errors in Instrumented and ABC build modes. make[1]: *** [/OpenArbor/openarbor-common/compiler-rules.mk:235: inst_kernelModeCalculatorPRL.s_] Error 1
4.5 Linux
Windows 		PCR:15221 Known Failure All BuildAll tests
DeosABCTest.ConfigErrorCheck
DeosABCTest.multi
DeosABCTest.one
VariantSpecificItFiles.variantItFilesForExeProject
MultipleBuildConfigs
DeosOptionTabs.abcToolTabTest 		QEMU-AARCH64 (qemu-aarch64) media:a-deos-process-linux-kismet-2-22-2024.txt /desk/bin/instrument --index=index.txt --format=GNU --arch=AARCH64 --assembly=aDeosProcess.s < aDeosProcess.s_ > inst_aDeosProcess.s_

Can't locate GNUAARCH64Postprocessing.pl in @INC (@INC contains: /desk /desk/bin /desk/lib/abcscat /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.34.0 /usr/local/share/perl/5.34.0 /usr/lib/x86_64-linux-gnu/perl5/5.34 /usr/share/perl5 /usr/lib
/x86_64-linux-gnu/perl-base /usr/lib/x86_64-linux-gnu/perl/5.34 /usr/share/perl/5.34 /usr/local/lib/site_perl) at /desk/bin/instrument line 163.
BEGIN failed--compilation aborted at /desk/bin/instrument line 169 (#1)

abc not supporting 64-bit

5 Linux (qemu-arm) DDCI_PCR:5162 Pass Debug.dvmsbasher-raw
 QEMU_ARM (qemu-arm)
QEMU-AARCH64 (qemu-aarch64)
QEMU-PPC (qemu-ppc) 		PCR 5162 was fixed, but debugging the dvmsbasher-raw example fails.
TimeoutException: Timeout after: 200000 ms.: waiting for dialog: Confirm Perspective Switch
The DVMS Examples User Guide doesn't mention this example, but it seems to have the same structure as the others, so the test needs to be modified to setup the example correctly.
Updated the test to setup example correctly
6 Linux (qemu-arm) DDCI_PCR:5162 Pass Debug.dvmsbasher-with-exfat
 QEMU_ARM (qemu-arm)
QEMU-AARCH64 (qemu-aarch64)
QEMU-PPC (qemu-ppc) 		Debugging the dvmsbasher-with-exfat example fails.
TimeoutException: Timeout after: 200000 ms.: waiting for dialog: Confirm Perspective Switch
The DVMS Examples User Guide doesn't mention this example, but it seems to have the same structure as the others, so the test needs to be modified to setup the example correctly.
Updated the test to setup example correctly
7 Linux (qemu-arm) DDCI_PCR:5162 Pass Debug.dvmsbasher-with-journaled-exfat
 QEMU_ARM (qemu-arm)
QEMU-AARCH64 (qemu-aarch64)
QEMU-PPC (qemu-ppc) 		Debugging the dvmsbasher-with-journaled-exfat example fails.
TimeoutException: Timeout after: 200000 ms.: waiting for dialog: Confirm Perspective Switch
The DVMS Examples User Guide doesn't mention this example, but it seems to have the same structure as the others, so the test needs to be modified to setup the example correctly.
Updated the test to setup example correctly
8 John K. Linux (qemu-arm) DDCI_PCR:5137 Pass Debug.hello-world-ist
 QEMU_ARM (qemu-arm)
QEMU-AARCH64 (qemu-aarch64)
QEMU-PPC (qemu-ppc) 		Debugging the hello-world-ist example fails.
TimeoutException: Timeout after: 200000 ms.: waiting for dialog: Confirm Perspective Switch

JJK> I installed dds-docker-sales-deos-kismet-20240611 on my laptop, pulled in the example ist>ioi>hello-world-ist, updated lwip.config per the instructions, fired it up in qemu-arm, and did StatusMonitor>hello-world-ist.exe>Debug with gdb. The debugger started up fine and I can single step. Am I not replicating correctly?
A bug fix via PCR 5137 fixed an issue with launching a debug configuration.
installed new unreleased OA 12.3.0 and re-ran the test and it passed.
9 Linux (qemu-arm) DDCI_PCR:4932 Known Failure MulticoreIntegration QEMU-ARM (qemu-arm)
QEMU-AARCH64 (qemu-aarch64)
QEMU-PPC (qemu-ppc) 		This test requires an update to work properly with BSPs that support pia.
10 Linux (qemu-arm) DDCI_PCR:4051 Known Failure ProjectConverter QEMU-ARM (qemu-arm)
QEMU-AARCH64 (qemu-aarch64)
QEMU-PPC (qemu-ppc) 		Unable to determine code name for DDS
This test needs to be updated.
11 Linux (qemu-arm) DDCI_PCR:5162 Pass Run.dvmsbasher-raw
 QEMU-ARM (qemu-arm)
QEMU-AARCH64 (qemu-aarch64)
QEMU-PPC (qemu-ppc) 		media: sales-kismet-dvmsbasher-raw-output-qemu-arm-20240612.txt This example has not been tested by OA's Run test before.
The test has no Pattern defined to verify the output of the example.
What is the expected output?
Updated test to check for output.
12 Linux (qemu-arm) DDCI_PCR:5162 Pass Run.dvmsbasher-with-exfat
 QEMU-ARM (qemu-arm)
QEMU-AARCH64 (qemu-aarch64)
QEMU-PPC (qemu-ppc) 		media: sales-kismet-dvmsbasher-with-exfat-output-qemu-arm-20240612.txt This example has not been tested by OA's Run test before.
The test has no Pattern defined to verify the output of the example.
Is the output correct?
Updated test to check for output.
13 Linux (qemu-arm) DDCI_PCR:5162 Pass Run.dvmsbasher-with-journaled-exfat
 QEMU-ARM (qemu-arm)
QEMU-AARCH64 (qemu-aarch64)
QEMU-PPC (qemu-ppc) 		media: sales-kismet-dvmsbasher-with-journaled-exfat-output-qemu-arm-20240612.txt This example has not been tested by OA's Run test before.
The test has no Pattern defined to verify the output of the example.
Is the output correct?
Updated test to check for output.
14 Linux (qemu-arm) PCR:15344 Known Failure Run.zlib-compress QEMU-ARM (qemu-arm)
QEMU-AARCH64 (qemu-aarch64)
QEMU-PPC (qemu-ppc) 		Media:sales-kismet-24a-qemu-arm-run-zlib-compress-video-stream-20240613.txt No Pattern Defined for zlib-compress.

Output appears to be incomplete
14.5 Ron TBD BSP team member to address PCR. Windows PCR:15532 Known Failure compileRunDebug.RunPalExtension QEMU-AARCH64 (qemu-aarch64) media:runpalextension-aarch64-output-20240614.png Video stream is blank, missing expected output
The test fails on qemu-aarch64, passes on qemu-ppc and qemu-arm.
15 Linux (qemu-arm) PCR:14080 Known Failure CacheTrasher
MakeExport.worker 		QEMU-ARM (qemu-arm)
QEMU-AARCH64 (qemu-aarch64)
QEMU-PPC (qemu-ppc) 		Cache Trashing is not supported on the platform.
16 Michael & Andre Linux (qemu-arm) PCR:15789 Known Failure deos.ConfigureRapita QEMU_ARM (qemu-arm)
QEMU-AARCH64 (qemu-aarch64)
QEMU-PPC (qemu-ppc) 		media: TBD Timeout after: 5000 ms.: Problems view does not contain \QNo RVS data is available from Critical Time Data\E
17 Linux (qemu-arm) PCR:TBD Known Failure deos.DeosCrittimeTest QEMU-ARM (qemu-arm)
QEMU-AARCH64 (qemu-aarch64)
QEMU-PPC (qemu-ppc) 		Crittime is not installed.
Timeout after: 5000 ms.: The widget ToolItem with tooltip text {Clear Crittime} was not enabled.
DDS does not contain a crittime kernel
/desk/ppc/appbin/crittime/libkernel.so does not exist
18 Linux (qemu-arm) DDCI_PCR:5077 Known Failure deos.DeosGdbLaunchConfigurations.deosGdbMultipleRemoteTargets QEMU-ARM (qemu-arm) media:deosGdbMultipleRemoteTargets-failure-2-23-24.png the DeosGdbLaunchConfigurations.deosGdbMultipleRemoteTargets test is hanging when trying to debug using a second target, leading to a Load List Communication timeout.
18.5 Linux
Windows 		DDCI_PCR:5137 Pass deos.DeosGdbLaunchConfigurations.deosGdbLaunching QEMU-ARM (qemu-arm)
QEMU-AARCH64 (qemu-aarch64)
QEMU-PPC (qemu-ppc) 		media:deosGdbMultipleRemoteTargets-failure-2-23-24.png Debug button is not enabled when expected by the test.
Updated the test for the new gdb command file option and for the auto-update test.
Reran the test and it passes now
19 Linux (qemu-arm) PCR:15231
(see comment #14) 		Known Failure deos.StatusMonitor QEMU-ARM (qemu-arm)
QEMU-AARCH64 (qemu-aarch64)
QEMU-PPC (qemu-ppc) 		media:StatusMonitor-failures-20240612.txt
media:StatusMonitor-missing-names-20240614.png 		The Names column is not populated with the process names mainProcess, firstProcess and secondProcess as expected by the test when the rtems-653-configuration example is integrated with the platform project.
From JK:The API from OAR no longer gives us the name of a thread, so we have it falling back to just display the thread ID. We'll want to see if OAR can start giving us the names again.
20 Linux (qemu-arm) DDCI_PCR:5115 Known Failure gdb.GdbVariables.Deos653
gdb.GdbVariables.MulitpleThreadsRMA
gdb.GdbVariables.Rtems 		QEMU-ARM (qemu-arm)
QEMU-AARCH64 (qemu-aarch64)
QEMU-PPC (qemu-ppc) 		TimeoutException: Timeout after: 5000 ms.: The widget with mnemonic 'Toggle Breakpoint' was not enabled.
21 Aaron Linux (qemu-arm) DDCI_PCR:4363 Pass other.ParallelizedBuilding QEMU-ARM (qemu-arm)
QEMU-AARCH64 (qemu-aarch64)
QEMU-PPC (qemu-ppc) 		media:ParallelizedBuilding-failures-20240612.txt make[1]: Leaving directory '/home/scoretest/oaTestHarness/workspace/ddses/DDS-docker-sales-deos-kismet-20240611/deos_arm_qemu/workspace/qemu-arm/output'

make[1]: *** No rule to make target 'integration-checks/', needed by 'integration-checks/valid-architecture-ftpserv.exe.confirmed'. Stop.
make[1]: *** Waiting for unfinished jobs....
AL updated several canned makefiles to fix the issue.
LJ unreleased OA with changes and re-ran test, passes now.

22 Linux (qemu-arm) DDCI_PCR:5014 Known Failure other.RenameProjects QEMU-ARM (qemu-arm)
QEMU-AARCH64 (qemu-aarch64)
QEMU-PPC (qemu-ppc) 		Media:renameprojects-qemu-arm-12-19-2023.txt When the qemu platform project is renamed, the launching of the emulator displays an error.
23 Linux (qemu-arm) Expected Deviation DebugOptionsAdditionalOptions
DebugOptionsMessages
DebugOptionsStartup 		QEMU-ARM (qemu-arm)
QEMU-AARCH64 (qemu-aarch64)
QEMU-PPC (qemu-ppc) 		MLD is not installed on Linux
MLD does not support 64bit archs
24 Linux
 DDCI_PCR:5182 Known Failure features.txt file
other.HelpInstalled 		QEMU-ARM (qemu-arm)
QEMU-AARCH64 (qemu-aarch64)
QEMU-PPC (qemu-ppc) 		Media:TBD The features.txt file generated by populateDDS.py contains 'TRAC' feature names, which are license feature names for MLD.
The content of the features.txt file is used by OA when borrowing licenses, and it's used by the OA test harness to generate a license to test the DDS.
Since we are no longer going to be shipping MLD, I think the script should be modified to no longer generate those lines.
Retrieved from "https://ddci.zapto.org/ddciWiki/index.php?title=OpenArbor_12.3.0_Test_Plan_and_Informal_Report&oldid=64823"