OpenArbor Development Linux

This is how to setup a Linux development environment for developing OpenArbor, debugging OpenArbor, running the automated test system, and other project-management activities.

For a quick guide to building and installing OpenArbor, see OpenArbor Build.

See also OpenArbor Testing.

It is assumed that all command line operations will be done in a bash shell in a desk maintainer environment.

Get a Docker Capable Workstation

1. Follow the instructions here DESK_On_Docker_Project

Download and install Eclipse

1. Request a VMWare DDCI virtual machine on the ESX server. For details, see: VMWare_Server_Ubuntu#Getting_A_Machine
2. Access the virtual machine. For details, see: VMWare_Server_Ubuntu#Accessing_the_Machine

   RDC to the machine

3. At the login to oem-virtual-machine

   Username: your username

   Password: your password

4. If no icon for opening a terminal is available on the desktop, click on 'Activites'

   type 'terminal', then click on the terminal app to open a terminal

5. As part of your first login you should run (from a terminal), the ddci-vm-fixup script. The fixup script enables samba (windows file sharing) and does some user specific config of RDP and the GUI to make things faster.

   ddci-vm-fixup

6. Define access to the ddci repositories. For details, see:OpenArbor_Test_Machine_Setup#Define_access_to_ddci_repositories
7. Install docker. For details, see: OpenArbor_Test_Machine_Setup#Get_docker_files
8. Setup mount points. For details, see: OpenArbor_Test_Machine_Setup#Setup_needed_mount_points
9. Install required desk packages. For details, see: OpenArbor_Test_Machine_Setup#Install_required_Desk_packages
10. Install the OpenArbor development environment for Linux:

    sudo apt-get install openarbormaintainertools

11. Allow OA building /testing to update /OpenArbor dir without asking for a linux password

    sudo bash -c "echo '$USER ALL = (ALL) NOPASSWD: ALL' >> /etc/sudoers"

12. To update/debug the OA Test Harness, launch the Eclipse development environment in the vm

    /Eclipse/2023-06-R/eclipse/eclipse

13. To debug a docker DDS, or a automated test, install and launch the Eclipse development environment from within the DDS docker image:

    Use the OA Test Harness to install a docker DDS (e.g. dds-docker-sales-deos-jupiter-20220524)

    Once the docker DDS is installed, in your linux vm, open a terminal window

14. Run the docker image and allow the mount points defined on the vm to be available in the docker image

    run-docker -it --tty --rm --mount=type=bind,source=/mnt,target=/mnt dds-docker-sales-deos-jupiter-20220524

15. Install the openarbor development environment in the docker image

    sudo apt-get install openarbormaintainertools

16. Save the updated docker container to a new image

    Open a second terminal in the vm

    docker container ls -a

    docker container commit <container name> dds-docker-sales-deos-jupiter-20220524-dev (e.g. container name is something like determined_banzai from the ls of containers)

17. Create the development workspace

    exit all docker containers

    From the linux VM, create the development workspace (e.g. ~/ddci_eclipse/workspace-10.n.n)

    (Note: if you accidentally create the dev workspace while in a docker container, launching OA will result in an error that the .metadata folder is read-only!)

18. Now from any terminal you an run the development environment for the dds:

    ./run-docker -i --rm --tty --mount=type=bind,source=/mnt,target=/mnt dds-docker-sales-deos-jupiter-20220524-dev

19. Setup DDCIFLEX_LICENSE_FILE

    export DDCIFLEX_LICENSE_FILE=/home/username/all_prods.lic

20. Launch Eclipse to debug OA source and/or OA automated tests within the docker image

    /Eclipse/2023-06-R/eclipse/eclipse

1. On an OA Linux development vm, you can setup subversion to store your personal password so Eclipse will not keep asking for your password when your synchronizing plugin dirs, etc.

   In ~/.subversion/config, uncomment the store-passwords=no line, and change no to yes

   In ~/.subversion/config, uncomment the store-auth-creds=no line, and change no to yes

Check out the development workspaces

mkdir -p /home/<username>/ddci_eclipse
svn co https://deos.ddci.com/svn/DDCI/products/openarbor/openarbor/branches /home/<username>/ddci_eclipse/branches

Workspace configuration

1. Shortcuts are not yet supported. Use the following command to launch the OA dev environment: /Eclipse/2023-03-R/eclipse/eclipse
2. Create a new workspace such as: /home/<username>/ddci_eclipse/workspace-<branchName>
   • Rationale: Branches come and go, and sometimes you just hose up your working copy. Keeping the workspace and the working copy separate makes it easier to switch branches or delete and re-download your SVN checkouts without having to worry about losing your workspace configuration.
3. Import Existing projects into workspace
   1. File->Import...->General->Existing Projects Into Workspace
   2. Browse... to /home/<username>/ddci_eclipse/branches/mainline/workspace
   3. OK
   4. Finish
4. Import the launch configurations.
   1. File->Import...->Run/Debug->Launch Configurations
      1. Set From Directory to /Eclipse/2021-12-R/eclipse/launches
      2. Check the launches checkbox.
         • The DDS Dev Launch is for debugging OpenArbor, the DDS Test Launch is for debugging tests.
   2. If this is your first launch, use Window->Preferences->Run/Debug->String Substitution to create the string substitution variables username, ddsBranchName, ddsConfigName and ddsName as per #Dds_Variables.
5. In Window->Preferences->Java->Editor->Save Actions
   1. Check the 'Perform the selected actions on save' box
6. Fix auto-props for both environments.
   1. In your DESK maintainer environment, edit /etc/subversion/config; change the line that says enable-auto-props = no to enable-auto-props = yes. You will also have to uncomment the preceding [miscellany].
   2. In Eclipse, go to Window->Preferences->Team->SVN and change the configuration location at the bottom to point to the /etc/subversion/ location of your maintainer environment, i.e., the result of the shell command cygpath -wa /etc/subversion/

1. To start OpenArbor: Run->Run Configurations...->Eclipse Application->DDS Dev Launch.

DDS Variables

The imported launch configurations use several environment variables to allow developers to easily switch between multiple installed DDS' when both developing and/or usring the automated test system. It is possible to test multiple bsp's for an installed DDS. The developer should only need to specify the ddsBranchName, ddsConfigName and ddsName. When a DDS is installed using the automated test system (OpenArbor_Testing), the DDS is installed at C:\oatestharness\workspace\DDS-list\$(ddsName). A workspace for that DDS is created at C:\oatestharness\workspace \ddses\$(ddsName\$(ddsConfigName)\workspace.

Variable	Example Value	Description
ddsBranchName	mainline	Must match the SVN branch name under development/test. Used by the testSystemInvocation variable to form the --branchRoot option to the test system.
ddsConfigName	deos_arm_qemu	Indicates the bsp under development/test. Used by the launch configurations to location the DDS workspace directory.
ddsName	DDS-docker-sales-jupiter-20230731	Indicates the dds under development/test. Used by the deosInstallDir and testWorkspacePrefix variables to locate the DDS installation directory and DDS workspace directory.
deosInstallDir	/	Indicates the location of the DDS installation directory.
deskRootLocation	/	Indicates the location of the DESK installation directory.
oaTestHarnessWorkspace	/home/${username}/oaTestHarness/workspace	Indicates the location of all test harness data.
testSystemInvocation	--installationRoot ${deosInstallDir} --branchRoot ${oaTestHarnessWorkspace}/branches/${ddsBranchName}	Indicates the the arguments the test system needs to run.
testWorkspacePrefix	${oaTestHarnessWorkspace}/ddses/${ddsName}	Indicates where the workspaces will be located.
username	scoretest	Ubuntu only: indicates user for launching tests

Installing Experimental Packages

Sometimes, you may want to test changes to the cygwin package scripts or package structure without pushing anything to a live server. This can be done by using an existing DDS as a base to modify.

1. Download the DDS you want to use as a base for your changes and put it somewhere.
2. Drill down into {DDS_LOC}/deos-heartos-posix/deos/{PACKAGE_NAME} to find the tar.bz2 you want to modify. Make a backup copy just in case.
3. make a temporary directory somewhere, and extract the contents of the tar.bz2 to the temporary directory, using tar -xjvf.
4. extract the contents of the tar file to this same temporary directory as well
5. You will likely have two directories "etc" and "<some package dir (ie. OpenArbor>". Modify the contents of either director as required.
6. To re-zip the updated directories, issue the following command at the top-most level of the temporary directory containing the updated files: tar jcvf {PACKAGE_NAME} {dir1} {dir2} (ie. tar jcvf trac-docs-258-4.tar.bz2 etc TRAC_)
7. Find the setup.ini file in {DDS_LOC}/deos-heartos-posix/, and find your package name inside this file.
8. Use md5sum to recalculate your new package, and ls -l to get the exact size of the new package. Replace the corresponding values in the setup.ini, and change the location of the archive, if you are using an alternative location.
9. The installer can now be run with the changes in place. Try it and see what happens.

• example line from setup.ini:
  • install: deos/trac-docs/trac-docs-258-4.tar.bz2 3308783 5ad42277e650bdc69a4be5fd48768071
    • size is 3308783 in the line above
    • md5sum is 5ad42277e650bdc69a4be5fd48768071 in the line above

Developing the OpenArbor Test Harness

1. make directory C:\ddci_eclipse_test_harness_dev
2. cd to the above directory and use svn to checkout a fresh copy of: https://deos.ddci.com/svn/DDCI/products/openarbor/utils/dds-manager/dds-manager-e4/
   
   1. svn co --username=<username> --password=<password> --no-auth-cache https://deos.ddci.com/svn/DDCI/products/openarbor/utils/dds-manager/dds-manager-e4/ dds-manager

1. Using the OpenArbor Test Harness, install a DDS, and add a test config.
2. /Eclipse/2023-03-R/eclipse/eclipse

   /home/scoretest/ddci_eclipse_test_harness_dev/dds-manager-e4

3. String substitution settings: Media:linux-debug-testharness-settings.png

Setup test machine on Linux (Deprecated)

Linux/Ubuntu

1. RDC to the machine
2. At the login to oem-virtual-machine

   Username: scoretest

   Password: (scoretest password)

3. If no icon for opening a terminal is available on the desktop, click on 'Activites'

   type 'terminal', then click on the terminal app to open a terminal

Define access to ddci repositories

1. Run the vm setup script

   ddci-vm-fixup

2. Open a terminal

   sudo -i

3. Modify /etc/apt/sources.list to use the DDCI package mirror

   cd /etc/apt

   mv /etc/apt/sources.list sources.list.bak

   nano sources.list

4. Paste the content of the sources_list2.txt file below into empty sources.list file

Media:sources_list2.txt

1. Add the DDCI DDS distribution-specific http server directory as a new package repository. (note, do no cut/paste from here due to windows line endings)

   distribution=jupiter

   echo "deb [trusted=yes arch=amd64] http://redhat5.ddci.com/Workstation/ubuntu-ddci/ ${distribution} ddci" > \

   /etc/apt/sources.list.d/ddci-dist-${distribution}.list \

   && echo "deb [trusted=yes arch=amd64] http://redhat5.ddci.com/Workstation/ubuntu-ddci/ ${distribution} ddci-unreleased" > \

   /etc/apt/sources.list.d/ddci-dist-${distribution}-unreleased.list

2. exit sudo

   exit

Get docker files

1. Checkout docker files from svn

   mkdir ~/scm

   svn co -N --username $USER@ddci.com https://deos.ddci.com/scm/Deos ~/scm/Deos

   svn up --parents ~/scm/Deos/maintainer-tools/docker/branches/mainline

2. login to docker (one time operation for a machine)

   docker login redhat5.ddci.com -u username@ddci.com

3. Allow 'run-docker' to be executed from anywhere, by updating .bashrc, adding the following line:

   PATH=$HOME/scm/Deos/maintainer-tools/docker/branches/mainline:$PATH

4. Add this to end of .bashrc to display the 'dkr' prompt when running a docker image.

  #If executing in docker (ref https://stackoverflow.com/questions/20010199/how-to-determine-if-a-process-runs-inside-lxc-docker)

  if grep -Fqs docker /proc/1/cgroup > /dev/null ; then

     PS1='dkr \W$ '

  fi

1. Start the apt cacher

   cd to docker installation dir

   ./start-apt-cacher

Install required Desk packages

1. From the linux vm, update and upgrade to get the latest packages:

   sudo apt-get update

   sudo apt-get --with-new-pkgs upgrade

2. To get the desk components:

   sudo apt-get install desk-python-tools make zip rename gcc-arm-eabi maven

3. Create a link for python

   cd /bin

   sudo ln -s python2 python

4. Create links for /lib64 (work around for flexnet tools)

   cd /lib64

   sudo ln -s ld-linux-x86-64.so.2 ld-lsb-x86-64.so.2

   sudo ln -s ld-linux-x86-64.so.2 ld-lsb-x86-64.so.3

5. Update .bashrc file

   gedit ~/.bashrc

   DDCIFLEX_LICENSE_FILE=/home/scoretest/all_prods.lic; export DDCIFLEX_LICENSE_FILE

   DDCI_FLEXNET_TOOLS_LOC=~/ddci_eclipse_test_harness_dev/dds-manager-e4/build/test-harness/64bit/linux/FLEXnet; export DDCI_FLEXNET_TOOLS_LOC

Setup needed mount points

1. Create mount points for the required directory locations
2. Install cifs-utils

   cd /home/scoretest

   wget http://archive.ubuntu.com/ubuntu/pool/main/c/cifs-utils/cifs-utils_6.9-1_amd64.deb (? why not sudo apt-get install cifs-utils ?)

   sudo dpkg -i cifs-utils_6.9-1_amd64.deb

3. Create the ~/.cifcreds file to add two lines

   touch ~/.cifcreds

   gedit ~/.cifcreds

   username=scoretest

   password=scteorste1

4. Modify the permissions for ~/.cifcreds

   sudo chmod 400 ~/.cifcreds

5. Add a directory for each mount point in /media. The first 3 must match those defined in Util.java DDS_LOCATIONS_LINUX, /media/sf_temp is used as unix temp dir by test code

   cd /mnt

   sudo mkdir /mnt/sf_DDCI_integration

   sudo chmod 755 /mnt/sf_DDCI_integration

   sudo mkdir /mnt/sf_approved

   sudo chmod 755 /mnt/sf_approved

   sudo mkdir /mnt/sf_other

   sudo chmod 755 /mnt/sf_other

   sudo mkdir /mnt/sf_temp

   sudo chmod 777 /mnt/sf_temp

   sudo mkdir /mnt/sf_TestResults

   sudo chmod 777 /mnt/sf_TestResults

   sudo mkdir /mnt/sf_Downloads

   sudo chmod 755 /mnt/sf_Downloads

   sudo mkdir /mnt/sf_LinuxRCP

   sudo chmod 755 /mnt/sf_LinuxRCP

6. Edit the /etc/fstab file to add a line for each mount point. (Use arrow keys to move cursor to new line, Ctrl-X to save and exit)

   sudo nano /etc/fstab

   For \\ddsbuild\DDCI_integration

   //10.0.0.47/DDCI_integration /mnt/sf_DDCI_integration/ cifs ro,auto,,vers=2.0,credentials=/home/scoretest/.cifcreds 0 0

   For \\nx3000\ship\dds\ubuntu\approved

   //10.0.1.219/ship/dds/ubuntu/approved /mnt/sf_approved/ cifs ro,auto,vers=1.0,credentials=/home/scoretest/.cifcreds 0 0

   For \\nx3000\ship\dds\ubuntu\other

   //10.0.1.219/ship/dds/ubuntu/other /mnt/sf_other/ cifs ro,auto,vers=2.0,credentials=/home/scoretest/.cifcreds 0 2

   For \\nx3000\openarbor\TestResults

   //10.0.1.219/openarbor/TestResults /mnt/sf_TestResults/ cifs guest,uid=845601180,rw,auto,vers=2.0,credentials=/home/scoretest/.cifcreds 0 2

   For \\nx3000\openarbor\Downloads

   //10.0.1.219/openarbor/Downloads /mnt/sf_Downloads/ cifs ro,auto,vers=2.0,credentials=/home/scoretest/.cifcreds 0 2

   For \\nx3000\OAStage (LinuxRCP)

   //10.0.1.219/OAStage /mnt/LinuxRCP cifs ro,auto,vers=2.0,credentials=/home/scoretest/.cifcreds 0 2

   For /mnt/sf_temp

   No definition is needed here. /mnt/sf_temp will just be a directory on the vm to be used for temporary files.

7. Mount the drive locations

   sudo mount -a

Create a Dummy Sound Card Device

Have the virtual machine create a dummy sound device to be passed to docker containers
running Eclipse. This is necessary because eclipse generates an error if there is no sound
device. The container has no sound device by default, and neither does a virtual machine
(apparently the expectation is that a remote desktop client will supply the sound device).
For some reason the RDP is not reliably creating said device, so we need to have a dummy in
the container, but there is no bundled ALSA (kernel) sound driver in docker, so it is
necessary to create the dummy device in the VM, then pass it to the container using the
sound switches defined in the test harness, which will execute the ensure-dummy-sound-device-exists
script

1. From the linux vm, create the snd-dummy.conf file

   sudo bash -c '(echo "# create dummy sound device"; echo "snd-dummy") > /etc/modules-load.d/snd-dummy.conf'

2. Add scoretest to the audio group

   sudo usermod -aG audio $USER

3. Reboot

   sudo reboot

1. Create the ensure-dummy-sound-device-exists script

   sudo nano ~/oaTestHarness/workspace/ensure-dummy-sound-device-exists

   paste the content of the file below into ensure-dummy-sound-device-exists

Media:ensure-dummy-sound-device-exists.txt

1. sudo chmod +x ensure-dummy-sound-device-exists

Setup VNC

1. Install VNCServer
2. Detailed instructions are here https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-vnc-on-ubuntu-20-04

   sudo apt update

   sudo apt install xfce4 xfce4-goodies

   when prompted for a default display manager, press ENTER

   sudo apt install tightvncserver

   vncserver

   enter the scoretest password (scteorste1)

   reply n to not create a view only password

3. Configure the VNC Server

   vncserver -kill :1

   mv ~/.vnc/xstartup ~/.vnc/xstartup.bak

   nano ~/.vnc/xstartup

   #!/bin/bash

   xrdb $HOME/.Xresources

   startxfce4 &

4. Restart the VNC Server

   chmod +x ~/.vnc/xstartup

   vncserver

5. Running VNC as a System Service

   sudo nano /etc/systemd/system/vncserver@.service

6. Add the following lines to the file

   [Unit]

   Description=Start TightVNC server at startup

   After=syslog.target network.target

   [Service]

   Type=forking

   User=scoretest

   Group=score

   WorkingDirectory=/home/scoretest

   PIDFile=/home/scoretest/.vnc/%H:%i.pid

   ExecStartPre=-/usr/bin/vncserver -kill :%i > /dev/null 2>&1

   ExecStart=/usr/bin/vncserver -depth 16 -geometry 1920x1080 :%i

   ExecStop=/usr/bin/vncserver -kill :%i

   [Install]

   WantedBy=multi-user.target

7. Make the system aware of the new file

   sudo systemctl daemon-reload

   sudo systemctl enable vncserver@1.service

8. Stop the server if it's still running

   vncserver -kill :1

9. Start it as a service

   sudo systemctl start vncserver@1

10. Verify it has started

    sudo systemctl status vncserver@1

11. If it started correctly you should see output like this:

    vncserver@1.service - Start TightVNC server at startup

    Loaded: loaded (/etc/systemd/system/vncserver@.service; enabled; vendor preset: enabled)

    Active: active (running) since Wed 2022-05-18 10:23:32 MST; 1h 28min ago

    Process: 880 ExecStartPre=/usr/bin/vncserver -kill :1 > /dev/null 2>&1 (code=exited, status=2)

    Process: 926 ExecStart=/usr/bin/vncserver -depth 24 -geometry 1280x800 :1 (code=exited, status=0/SUCCES>

    Main PID: 941 (Xtightvnc)

    Tasks: 152 (limit: 9360)

    Memory: 312.1M

    CGroup: /system.slice/system-vncserver.slice/vn

    ...

12. Reboot the machine

    sudo reboot

13. Login using TightVNC Viewer

    testlinux01::5901

    scteorste1

Build and start the OA Test Harness

1. Check out test harness code from SVN and build the test harness

   Open a terminal

   cd ~

   mkdir ddci_eclipse_test_harness_dev

   cd ddci_eclipse_test_harness_dev

   svn co https://deos.ddci.com/svn/DDCI/products/openarbor/utils/dds-manager/dds-manager-e4/ dds-manager-e4

   cd dds-manager-e4

   make MAVEN_OFFLINE=""

   ./oaTestHarness -n