OpenArbor Test Machine Setup

From DDCIDeos
Jump to navigationJump to search

Setup

The OpenArbor automated test system is a client-server based system. The client, the "DDS Manager GUI" installed on some machine, communicates with the five available test machines which all have a server running on them.

Test Machines

There are currently several machines with dummy plugs installed available for testing. In addition, you may need to turn off notifications from Windows that ask if you want to allow an application to make updates to the machine.

Machine OS
OATEST Windows 11 (64bit)
OATESTLAP Windows 10 (64bit)
TESTW10E1 Windows 10 (64bit)
TESTW10E2 Windows 10 (64bit)
TESTLINUX01 Linux VM
TESTLINUX02 Linux VM

Antivirus Exceptions

Exceptions should be added to the symantec antivirus to allow an exception for the oatestharness directories and any OpenArbor develpment directories.

  1. Open the Trend Micro Security Agent Dialog
  2. Click the Settings button (gear icon)
  3. Select Real-time Scan
  4. Expand Exclusions
  5. Add the following directories:
    1. C:\temp
    2. C:\oaTestHarness
    3. C:\ddci_eclipse
    4. C:\ddci_eclipse_test_harness_dev

Setup Instructions for the Test Monitoring Machine (a.k.a. Client)

The OpenArbor Test Harness GUI is the client program for the dds manager system. It is an Eclipse RCP project that has been stored in SVN.

Windows

  1. Install a desk maintainer environment.
  2. Update the cygwin terminal to be "run as adminstrator"
    1. Start Menu | right click on Cygwin Terminal | Open File Location, from the explorer window, right click Cygwin Terminal | Properties | Advanced | check "Run As Administrator"
  3. Disable screen saver - this interrupts the test system.
  4. Make directory C:\ddci_eclipse_test_harness_dev
  5. cd to the above directory and use svn to checkout a fresh copy of: https://deos.ddci.com/svn/DDCI/products/openarbor/utils/dds-manager/dds-manager-e4/
    1. svn co --username=<username> --password=<password> --no-auth-cache https://deos.ddci.com/svn/DDCI/products/openarbor/utils/dds-manager/dds-manager-e4/ dds-manager
  6. If you just want to start using the client, run the oaTestHarness.sh script. It accepts a few arguments.
    1. -n will skip running make kindaclean.
    2. -w accepts an absolute path to a workspace. By default it will go to Root:/oaTestHarness/workspace.
    3. -d accepts an absolute path to a desk maintainer root. By default the script uses the desk environment it is launched in.
    4. ./oaTestHarness.sh [-n]
  7. If you see an maven error about not being able to get stuff in offline mode try:
    1. make MAVEN_OFFLINE=""
    2. then run the oatestharness script again.
  8. Ensure the directory C:\temp exists before attempting to install a dds.
  9. The OpenArbor Test Harness GUI should now be displayed.
    1. right-click localhost and "Add Machine" for each test machine you want to monitor


Docker

  1. Get a Docker Capable Workstation
  2. Define Environment Variables
  3. Checkout the test harness source code
  4. Setup Mount Points
  5. Build Test Harness in a Baseline Docker Image
  6. Launching Test Harness in a Baseline Docker Image

Connecting to a Test Machine

Windows

Connecting to a Test Machine via VNC Viewer As part of the automation effort, connecting via VNC is now the preferred method of administering the test machines. Dummy plugs have been installed on all five machines in order to allow the tests to run without the need for anyone to be connected. You may now start your test suite, close the window, and go about your day. While RDC still works, it can interfere with the VNC setup, so if you absolutely must use RDC, please ensure that no tests are currently running on the machine.

VNC is very easy to set up and use.

  1. Download TightVNC from http://tightvnc.com/download.php
  2. When installing, if you don't want a server on your machine, choose custom install and only install the viewer
  3. Start the TightVNC viewer
  4. Enter the name of the machine you wish to connect to
  5. The password is still scteorste1
  6. You may need to log into the local machine as well
    1. Username is scoretest for all test machines

VNC has some additional benefits and features as well. In addition to headless testing, multiple people can connect to the same machine at once, allowing easy screen sharing of any of our testing machines. TightVNC also has a viewer-only mode, should you want to observe tests running without interfering with them due to errant mouse input.

If you really need it, you can also use Remote Desktop Connection (RDC) to connect to the reserved test machine.

Username: scoretest
Password: scteorste1

Logging in as user "scoretest" is a strategy to allow anyone on the testing team to login and administer the test run. Most of the time, the test runs span several days and problems need to be addressed by more than one person.

Docker

RDC Settings



Connecting to a Test Machine viaRemote Desktop Connection (RDC)

Username: scoretest
Password: scteorste1

Logging in as user "scoretest" is a strategy to allow anyone on the testing team to login and administer the test run. Most of the time, the test runs span several days and problems need to be addressed by more than one person.

Setup for the Test Machine (a.k.a. Server)

Windows

    1. Make sure that tightvnc is installed, along with the DFMirage mirror driver. If the tightvnc installer asks you for passwords, use the scoretest password for all of them.
    2. Use TightVNC (setting up vnc) to log onto a test machine.
    3. Make sure there is a desk maintainer environment installed to C:\cygwin.
    4. Setup appropriate shortcuts for desk environment. For example, if you have installed the Cygwin2022 desk environment, create the following shortcuts:
      1. Create desktop shortcut for C:\cygwin2022-kismet\bin\mintty.exe named 'Cygwin2022 Terminal'.
      2. Change shortcut's Target to: C:\cygwin2022-kismet\bin\mintty.exe -i /Cygwin-Terminal.ico -
      3. Change Shortcut's Start in to: C:\cygwin2022-kismet\bin
      4. In 'Advanced', check 'Run as Administrator' box.
      5. Create desktop shortcut for C:\cygwin2022-kismet\bin\run.exe named '2023-06-R OA Dev Cygwin2022'.
      6. Change shortcut's Target to: C:\cygwin2022-kismet\bin\run.exe /bin/python /desk/desk-environment.py C:\Eclipse\2023-06-R\eclipse\eclipse.exe
      7. Change Shortcut's Start in to: C:\Eclipse\2023-06-R\eclipse
      8. In 'Advanced', check 'Run as Administrator' box.
    5. If you have installed the Cygwin2022 desk environment, set environment variable: setx DDCI_FLEXNET_TOOLS_LOC C:\ddci_eclipse_test_harness_dev\dds-manager\build\test-harness\64bit\windows\FLEXnet
    6. If you want to setup all the test machines at once:
      1. Make sure you have a checkout of https://deos.ddci.com/svn/DDCI/products/openarbor/utils/dds-manager/dds-manager-e4/ to C:/ddci_eclipse_test_harness_dev/dds-manager-e4.
      2. Run the updateTestMachines.sh script. If this is a new test machine then you will need to update the script by adding the name of the new machine. This script makes sure all the test machines are running the latest version of the test harness.
    7. If you are only trying to get one machine going:
      1. If there isn't a checkout in C:/ddci_eclipse_test_harness_dev/dds-manager-e4, then svn co https://deos.ddci.com/svn/DDCI/products/openarbor/utils/dds-manager/dds-manager-e4/ /cygdrive/c/ddci_eclipse_test_harness_dev/dds-manager-e4.
      2. Otherwise, svn up /cygdrive/c/ddci_eclipse_test_harness_dev/dds-manager-e4.
      3. Now, cd /cygdrive/c/ddci_eclipse_test_harness_dev/dds-manager-e4 && ./oaTestHarness.sh.

Docker

Follow the instructions here to build and launch the OATestHarness(Steps 1-6): OpenArbor_Development_Docker

Debugging an Automated Test

OpenArbor runs in the docker container. The Eclipse development environment must be installed in the docker container in order to debug a test with Eclipse.

  1. Start the docker container using run-docker
    run-docker --mount=type=bind,source=/mnt,target=/mnt -i --tty --rm dds-docker-sales-deos-jupiter-20221018
  2. Define DDCI_FLEXNET_TOOLS_LOC environment variable
    export DDCI_FLEXNET_TOOLS_LOC==/home/scoretest/ddci_eclipse_test_harness_dev/dds-manager-e4/build/test-harness/64bit/linux/FLEXnet
  3. Install the Eclipse development environment
    sudo apt-get install openarbormaintainertools
  4. Save the updated docker container
    open a separate terminal
    docker container ls
    docker container commit <container name> dds-docker-sales-deos-jupiter-20220524-dev (e.g. container name is something like determined_banzai from the ls of containers)
  5. Start Eclipse specifying the test harness workspace
    /Eclipse/2020-12-R/eclipse/eclipse
    Workspace: /home/scoretest/ddci_eclipse/workspace_n.n.n
    Setup Eclipse dev env and begin debugging from within the container
  6. When done, clean up containers and dev images.

Debugging the Test Harness

The test harness runs in the vm. The Eclipse development environment must be installed in the vm in order to debug with Eclipse.

  1. Define DDCI_FLEXNET_TOOLS_LOC environment variable
    export DDCI_FLEXNET_TOOLS_LOC=/home/scoretest/ddci_eclipse_test_harness_dev/dds-manager-e4/build/test-harness/64bit/linux/FLEXnet
  2. Install the Eclipse development environment
    sudo apt-get install openarbormaintainertools
  3. Start Eclipse specifying the test harness workspace
    /Eclipse/2020-12-R/eclipse/eclipse
    Workspace: /home/scoretest/ddci_eclipse_test_harness_dev/dds-manager-e4
  4. Ensure Window | Preferences | Run/Debug | String Substitution values are correct
    username "scoretest"
    deskRootLocation "/"
    dockerRootLocation "/home/scoretest/docker"
  5. Ensure Debug Configuration for DDS Test Harness Launch is correct
    Arguments tab, add --dockerRoot ${dockerRootLocation}
    Plugins tab, uncheck 'Validate Plug-ins automatically prior to launching"

Disabling SVN password storage on Test Machines

If authentication settings in the SVN config or servers files are set to save passwords, users may be unable to switch from a logged in account to log into their own, creating a roadblock if a user that is not logged in wants to run a SVN command. In order to prevent this from happening, disable SVN password storage:

Windows

  1. In cygwin terminal, cd to ~/.subversion
  2. rm -r auth
  3. in ~/.subversion/config ([auth] section), set 'password-stores = '. Comment out any existing 'password-stores' entries.
  4. in ~/.subversion/servers ([global] section), set 'store-passwords = no'. Comment out any existing 'store-passwords' entries.

Docker

  1. cd to ~/.subversion
  2. sudo rm -r auth
  3. in ~/.subversion/config ([auth] section), set 'password-stores = '. Comment out any existing 'password-stores' entries.
  4. in ~/.subversion/servers ([global] section), set 'store-passwords = no'. Comment out any existing 'store-passwords' entries.

Troubleshooting

CreateProcess error=740, The requested operation requires elevation
I got this message once when the test machine was trying to execute setupDDCI.exe, after pressing the "Install" button on the test harness GUI.
The solution is to do the following on the test machine:

  1. Start | Control Panel | User Accounts | Change User Account Control Settings | Move the lever to "Never Notify"
  2. Reboot the test machine


TCP Port 7181 in use error - lmgrd cannot start the license server. Use netstat to determine what pid using the tcp port


  1. scoretest@testlinux01:~/ddci_eclipse_test_harness_dev/dds-manager-e4/build/test-harness/64bit/linux/FLEXnet$ netstat -tulpn | grep LISTEN
    (Not all processes could be identified, non-owned process info
    will not be shown, you would have to be root to see it all.)
    tcp 0 0 0.0.0.0:3142 0.0.0.0:* LISTEN -
    tcp 0 0 0.0.0.0:139 0.0.0.0:* LISTEN -
    tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN -
    tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
    tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN -
    tcp 0 0 0.0.0.0:445 0.0.0.0:* LISTEN -
    tcp 0 0 127.0.0.1:43489 0.0.0.0:* LISTEN -
    tcp6 0 0 :::3142  :::* LISTEN -
    tcp6 0 0 :::139  :::* LISTEN -
    tcp6 0 0 :::7181  :::* LISTEN 88938/./lmgrd
    tcp6 0 0 :::16462  :::* LISTEN 2808/java
    tcp6 0 0 :::22  :::* LISTEN -
    tcp6 0 0 ::1:3350  :::* LISTEN -
    tcp6 0 0 ::1:631  :::* LISTEN -
    tcp6 0 0 :::44827  :::* LISTEN 88940/ddciflex
    tcp6 0 0 :::3389  :::* LISTEN -
    tcp6 0 0 :::445  :::* LISTEN -
    scoretest@testlinux01:~/ddci_eclipse_test_harness_dev/dds-manager-e4/build/test-harness/64bit/linux/FLEXnet$ ps -a | grep lmgrd
    88938 pts/0 00:00:00 lmgrd

Test Machine Failure Log (BSOD, Freeze, etc.)

A log of all known failures of the test machines, recorded with the hopes to find trends that can be fixed going forward.

Machine OS Failure Type Did any action seem to provoke the failure? Details
OATESTLAP Windows 11 (64bit) BSOD Launcher test run 20240726:LJ started a launcher test run for qemu-ppc on DDS-tostones3-doeos-jupiter-20240422 after installing latest stable Deos components and unreleased OA 12.1.3 using the cygwin installer

20240727:LJ connecting to OATESTLAP fails from either VncViewer or RDC.
20240729:Kevin L looked into the issue and found that a BSOD had occurred, but no minidump was created.
He ran the DISM restore health and sfc /scannow commands, which resulted in the machine stating no component store or system file corruption
20240730: LJ restored the test harness to working condition, 231 tests had run, and the last test was other.HelpAboutTest which didn't complete.

Retrieved from "https://ddci.zapto.org/ddciWiki/index.php?title=OpenArbor_Test_Machine_Setup&oldid=71066"