DDS Build for Deos

Overview

This document describes how to Build the DDC-I Developer Suite (DDS) and create media that exactly matches a Deos customer's install. If you are simply looking to install already created media - refer to the DDS_Install_for_Deos_Maintainers how-to.

Building DDS

A DDS build creates a folder which contains what should be placed on the ftp site for customers to download.

64-bit DDS' are built on the ddsbuild machine. 32-bit DDS' were for Windows 7 and older and are no longer built.

Building a 64-bit DDS

Build_a_64bit_DDS

The License File

A prior version of the DDS installer used the license file to determine which components are to be installed on the user's system. This is no longer the case. Everything on the media-set (i.e., the DDS) is automatically installed, and the user has only minimal control over the install. This essentially means that every customer will receive a unique DDS distribution.

You no longer need to get the specific customer license to perform the install. Most of you should skip to Installing and Running the Media. However, if you want to create a new license which corresponds to the customer's license, then the following information is still technically correct.

If you wish to match the customer's license you should start with an existing license. You can use either
\\ddsbuild.ddci.com\DDCI_integration\all_prods.lic as a starting-license-file and skip to Customize the License for Deos; otherwise follow the directions below for obtaining a specific customer's license to use as a starting-license-file.

Get the Customer License

The customer license files can be found in \\nx3000.ddci.com\utils\flexnet\Generated-Customer-Licenses. Each customer and partner will have a folder containing their current license.

Tips for Finding a Customer License

Each customer folder is of the format <codename>-<company>.
Each partner folder is of the format partner-<company>.

Customize the License for Deos

1. From Windows Explorer navigate to \\nx3000.ddci.com\utils\flexnet\LicenseGenerator_current
2. Run the License Generator by launching the runme.bat file i.e.,
   \\nx3000.ddci.com\utils\flexnet\LicenseGenerator_current\runme.bat
3. At the bottom of the app, in the middle of the window, click the second button from the bottom that says: Read Existing License...
4. Point the dialog at your customer license, starting license file
   1. Reading an existing license file will select the same set of features and product versions in the generator
   2. These presets can be changed if needed
5. On the left side of the dialog, ensure the following (from top to bottom)
   1. A correct, fully qualified file name for your new license file that refers to the installation media folder name e.g., C:\<local folder>\dds-chino-deos-fourpeaks-20130329.lic
   2. An expiration date (the default expiration is acceptable)
   3. Host: Windows (this is the default)
   4. Type: DEMO (this is the default)
6. In the "middle" ensure the correct set of features are selected. For example, for a chino-fourpeaks distribution:
   1. DEOS (check)
   2. GCC
      1. MIPS (check)
      2. PPC (check)
      3. X86 (check)
   3. HEARTOS (unchecked)
   4. OPENARBOR (check)
   5. SCORE (unchecked)
   6. TADS (unchecked)
   7. TRAC
      1. MIPS
         1. C (check)
         2. CPP (check)
      2. PPC
         1. C (check)
         2. CPP (check)
      3. X86
         1. C (check)
         2. CPP (check)
7. On the "right" ensure the product versions are specified correctly
8. Double check there are no WARNINGS listed on the lower-left

Note:TRAC is not supported at this time in the kismet baseline so those warnings can be ignored

1. Click the Generate License File button at the middle/bottom of the window

You now have the-deos-license-file for use in the below.

Note: By selecting DEMO as the license type we have just created a license that can be used on *any* machine (hence why the expiration date is so important).

Installing and Running the Media

After creating the media, you can install it on your local workstation or on a DeosTower for any developer to use. Run setupDDCI.exe, located in the D:\DDCI_integration\DDS-<customer-id>-deos-<distribution>-yyyymmdd, to perform the install. You will be prompted for a location to install the DDS components. When specifying the "Destination Folder", append the customer's media name to "DDC-I-", e.g., the Destination Folder for C:\dds-media\hotdish-20100208 is C:\DDC-I-hotdish-20100208.

Note: We are no longer distributing the VMware Player, but you can reuse an existing, installed player.

The installation will create shortcuts on your desktop and the Start Menu for OpenArbor and the DESK. The name on each will correspond to the destination folder name, e.g. "DDC-I-hotdish-20100208".

If you wish to use the custom license file created above, you will need to edit the system environment variable, DDCIFLEX_LICENSE_FILE and set the value to the pathname of the new license file. Remember to restore the value of the environment variable when you have finished running this installation. Note: The default license already defined by DDCIFLEX_LICENSE_FILE will likely be adequate to run this new installation, so setting the environment variable is unnecessary in most cases.

When running an installation of OpenArbor, you must specify a workspace. Please create a default "workspace" for the installation with the appropriate name; e.g. "C:\OpenArbor\workspace-hotdish-20100208". Additional workspaces should be created within this directory as needed when debugging/accessing customer issues.

Note: OpenArbor does not support simultaneous access for multiple users. In order for multiple developers to access the same customer media, the media must be copied from the deostower on to the developer's workstation. The quickest way to perform a copy is with rsync. For directions on using rsync to copy a customer's media, please refer to section "Using rsync" on the DeosTower_Remote_Desktop_Connection wiki.

Modifying an existing Archive

There are times when we want to use an archive to create a DDS for a new customer or change component content for an existing customer. keeping all other content the same. But, if the customer or component didn't exist, there was no way to create the DDS from an existing archive. Well, now there is. NOTE:These steps assume the new customer package or component has been created and the ftp sever updated following the directions in README.txt.

Make Copy of Archive

There are two possible scenarios: the new DDS needs to have the same date as the original DDS or the new DDS needs to have the date it is created.

Scenario 1:Same date as original DDS for new DDS. This is used for a customer that is downstream from an existing customer.
Log into the distribution server.
ssh -l deosftp deos.ddci.com -p 47734
Change directory to the archive where the folder is located that needs to be copied. The below is an example command copying the archive for dds-trailhead-deos-jupiter-20200511:
cd pub/Workstation/cygwin-20180401-ddci-dist-jupiter/x86_64/archive/
Copy the existing archive and add A. This will be the new archive folder to be modified.
cp -R 1589230173 1589230173A

Scenario 2:Current date for new DDS. In this example, the timestamp is for 09/21/2020. This is used for "headwater" customers.
Get the timestamp conversion for the date the new DDS is to be built. This site is great https://timestamp.online/. When the site is accessed, the timestamp is the time the page is accessed.
Log into the distribution server.
ssh -l deosftp deos.ddci.com -p 47734
Change directory to the archive where the folder is located that needs to be copied. The below is an example command copying the archive for dds-trailhead-deos-jupiter-20200511:
cd pub/Workstation/cygwin-20180401-ddci-dist-jupiter/x86_64/archive/
Copy the existing archive to a new folder with timestamp for 09(month)/ 21(day)/ 2020(uear) 10(hours)/000(mins)/16(secs). This will be the new archive folder to be modified.
cp -R 1589230173 1600704016

Copy the setup.bz2 to a local folder

Create a local folder to copy the setup.bz2 file to and using a bash window, cd to that folder. In the example below, the local folder is the same as the folder created during the copy.
cd /cygdrive/c/1589230173A/
Copy the setup.bz2 to /cygdrive/c/1589230173A/

Decompress setup.bz2 and Edit

Decompress the setup.bz2 using the command below:
bzip2 -d setup.bz2
Edit setup using a text editor with added content in the setup.bz2 file created in a new archive when ./deos2cygwin.py is executed. Copy and decompress this setup.bz2 file the same way in order to access the new customer/component that was added.

Compress setup file

Once you are done editing the file, compress it.
bzip2 -z setup

Create a README.txt

Document in README.txt what changes were made to the new archive. Examples
New customer definition, customer-ferguson, added.
New component/version of existing component added.

Copy Files to New Archive Folder

Use secure copy to copy the files to the new archive folder previously created. Below are examples of copying setup.bz2 and README.txt.
scp -P 47734 setup.bz2 deosftp@deos.ddci.com:pub/Workstation/cygwin-20180401-ddci-dist-jupiter/x86_64/archive/1589230173A/
scp -P 47734 README.txt deosftp@deos.ddci.com:pub/Workstation/cygwin-20180401-ddci-dist-jupiter/x86_64/archive/1589230173A/

Modifying setup.xz

If there is a setup.xz file as well as setup.bz2, update it as well following the steps above except use the following to decompress and compress the file:
xz --decompress setup.xz
xz -z setup

Build DDS with Updated Archive

You are all done! Now you can build the DDS following the Building DDS instructions using the --archive switch. Make sure the customer_info.json file has also been updated if adding a new customer package. Example build command below:
./populateDDS.py --progress --archive=1589230173A customer-ferguson

Patching a Prior DDS Build

Often we wish to ship a customer exactly what they had before, except... well, you get the idea; just one or two new components, but everything else is the same. This section details the steps necessary to patch an existing media-set (i.e., installation) creating a new media-set. The advantage to the customer is a complete install vs. a multi step patching process which is difficult to replicate correctly across their disperse team. The advantage to DDC-I is reduced QA/test burden.

The patched components must have been built with deos2cygwin.py at some point. Therefore, you may need to add an unreleased link, run deos2cygwin.py, and then remove the link in order to have the -x.tar.bz2 file needed by the installer.

Note: it is best to perform the below while on the local Phoenix network (i.e., RDC in, if necessary).

Copy/Rename from Ship to Integration

First, copy the media-set from the ship area to the Integration area giving it a new name.
Ship Area: \\nx3000.ddci.com\ship\dds\windows\approved
Integration Area: \\ddsbuild.ddci.com\DDCI_integration

Assuming ship is mapped as drive S and Integration is mapped as drive Z, the below is an example command:
cp -r /cygdrive/s/dds/windows/approved/DDS-chino-deos-fourpeaks-20130830 /cygdrive/z/DDS-chino-deos-fourpeaks-20130905

Make writable

The directions below will have you modify files and directories that are read only. Before modifying a file or directory you must use the chmod command to change the attributes of the file/directory so that it can be modified. Use the following command:

chmod ugo+rwX <file path>/<file or directory name>

Update setup.ini

The string ROOT will be used to refer to the newly created folder in the Integration site (e.g., DDS-chino-deos-fourpeaks-20130905). You will find the setup.ini file in the following location:
ROOT/deos-heartos-posix/x86_64/

If the release in question is old, then the location will be:
ROOT/deos-heartos-posix/x86/ or
ROOT/deos-heartos-posix/

Updating an existing package

Edit the file and seacrh for @ <package-name>. For example, if you wish to update the ansi package, search for:

 @ ansi

You will find a section similar to the below:

 @ ansi
 sdesc: "The Deos ANSI library."
 category: Deos
 requires: desk desk-python-tools
 version: 4.1.4-1
 install: deos/ansi/ansi-4.1.4-1.tar.bz2 205869 9c75c42154a3d6732e0f3df7cb20c9d5

1. Change the version field to reflect the new version to be shipped.
2. Change the install field. The easiest way to do this is to copy it from the appropriate setup.ini on the FTP server, e.g. deosftp@deos.ddci.com/pub/Workstation/cygwin-20120401-ddci-dist-fourpeaks/setup.ini. Note: If the values are not part of a setup.ini you can either add a temporary unreleased link to see what the lines should be or run md5sum and /bin/ls -l to get the size and checksum of the tar.bz2 file to place in the setup.ini file.
3. Save the changes.

Adding a package

1. Using an existing setup.ini (see above) that contains the entry for the package to be added, cut and paste the section into the destination setup.ini. Note: It is likely that you will need to insert the new package description such that alphabetical order is preserved.
2. If needed, remove the [test] line from the freshly inserted package description. This is needed if you copied from an unreleased setup.ini entry.
3. In order for the customer to get the newly added package installed, it needs to be listed in the cygwin dependency chain. So, open the ROOT/deos-heartos-posix/packages text file to determine the top-level package the cygwin setup program will install. You can either add your new package name here (a comma separated list), or perhaps better, edit the requires line of the appropriate package descriptions within the setup.ini file.
4. Save the changes.

Removing a package

1. Simular to Updating an existing package, seacrh for @ <package-name> and remove the section (see above for an example of a section).
2. Simular to Adding a package, remove it from the ROOT/deos-heartos-posix/packages text file to and/or edit the requires line of the appropriate package descriptions within the setup.ini file.
3. Save the changes.

Update Cygwin Packages

Updating an existing package

Copy the cygwin package (aka tar ball) from the ftp server to the Integration area. Continuing with ansi as the example:
From: deosftp@deos.ddci.com/pub/Workstation/cygwin-20120401-ddci-dist-fourpeaks/deos/ansi/ansi-4.1.4-1.tar.bz2
To: '

Note: you can remove the old package from the Integration folder.

Adding a package

1. Create the appropriate subdirectories under Root/deos-heartos-posix/
2. Copy the packages (.tar.bz2) to the new directory. Note: Ensure the package name numerology matches that stated in the setup.ini. E.g., if the <setup.ini> line for ansi contains this:

 install: deos/ansi/ansi-4.1.4-1.tar.bz2 205869 9c75c42154a3d6732e0f3df7cb20c9d5

Make sure ansi-4.1.4-1.tar.bz2 is the one copied.

Removing a package

1. Find the appropriate subdirectories under Root/deos-heartos-posix/ and remove them

Update Docs Folder

Updating or Adding

The media-set contains a folder that includes the content from desk/help. Unzip the install from the ftp site and replace the help content in the Integration folder with the correct help content for the updated package. For example:

1. unzip deosftp@deos.ddci.com//pub/deos-products/ansi/4.1.4/ansi-4.1.4.zip someplace local
2. copy the contents of the local desk/help folder to ROOT/docs/deos/help

Removing

Use the components archive file (e.g., deosftp@deos.ddci.com//pub/deos-products/ansi/4.1.4/ansi-4.1.4.zip) to determine its contribution to the desk/help tree and remove those files from ROOT/docs/deos/help.

Update Cover letter & Labels Text Files

There are two text files in ROOT:

• cover_letter.txt
  • Update the project name on the RE: line
  • Update the DDS Version on the line below the RE:
  • Update the the contents if necessary.
• label.txt
  • Update the version number
  • Update the DDS name
  • Update the CCN (Customer Code Name)

Build Docker with Archive

If you need to build a new customer docker release from an archive, use the following command:
$./build-docker --timestamp=1734367908 --save=dds-docker-zhangmen-deos-kismet-20240403 --name=customer-zhangmen kismet come-ctl6-x86-64 deos653 pcie-scanner dvms-sata-ahci desk-x86-64 desk-aarch64 deos-general-release
where
--timestamp is the archive timestamp
--save is the name of the image
--name is the customer name
kismet is the baseline
packages from the customer-<codename> packages
NOTE: for docker, replace _64 with -64

TODO add instructions to put DDS-docker together and export documents

Patching a Prior Docker Build

Often we wish to ship a customer exactly what they had before, except... well, you get the idea; just one or two new components, but everything else is the same. This section details the steps necessary to patch an existing media-set (i.e., installation) creating a new media-set. The advantage to the customer is a complete install vs. a multi step patching process which is difficult to replicate correctly across their disperse team. The advantage to DDC-I is reduced QA/test burden.

The patched components must have been built with deos2cygwin.py at some point. Therefore, you may need to add an unreleased link, run deos2cygwin.py, and then remove the link in order to have the -x.tar.bz2 file needed by the installer.

Note: it is best to perform the below while on the local Phoenix network (i.e., RDC in, if necessary).

Copy/Rename from Ship to Integration

First, copy the media-set from the ship area to the Integration area giving it a new name.
Ship Area: \\nx3000.ddci.com\ship\dds\ubuntu\approved or \\nx3000.ddci.com\ship\dds\ubuntu\other
Integration Area: \\ddsbuild.ddci.com\DDCI_integration

Assuming ship is mapped as drive S and Integration is mapped as drive Z, the below is an example command:
cp -r /cygdrive/s/dds/ubuntu/other/DDS-docker-gator-deos-kismet-20240524 /cygdrive/z/DDS-docker-gator-deos-kismet-20240611

Install and Run Docker

$ cd DDS-docker-gator-deos-kismet-20240611
$ ./install-dds
$ docker run -it --rm --name=foo dds-docker-gator-deos-kismet-20240524

Execute apt-get

root@##:/# cat /etc/apt/sources.list.d/ddci-dist-kismet.list
root@##:/# sed --in-place -e 's/old timestamp/new timestamp/' /etc/apt/sources.list.d/ddci-dist-kismet.list root@##:/# apt-get update
root@##:/# apt-get install <component>

• Visually verify correct version updated
• Don't exit until the Export Container steps are completed

root@##:/# exit

Export Container

In a new bash window cd to where the new tar.xz will go
$cd /cygdrive/c/DDCI_integration/DDS-docker-gator-deos-kismet-20240611
$docker container ls (to confirm foo is present)
$docker container export foo | xz -T0 > dds-docker-gator-deos-kismet-20240611.tar.xz
$rm -rf dds-docker-gator-deos-kismet-20240524

Update Docs Folder

Update DDS-docker-gator-deos-kismet-20240611\docs\deos\help with release-notes and/or user guide for updated component

Update Cover letter

In ROOT, update:

• cover_letter.txt
  • Update the project name on the RE: line
  • Update the DDS Version on the line below the RE:
  • Update the the contents if necessary.

Request QA Approval

Diff Ship with Integration

In order to ensure QA that we can skip a complete test run for a cygwin release, perform a sorted diff on the original approved tree against the newly created Integration tree:

BASELINE=multicore
CUSTOMER=harrys
NEW_DATE=20180419
OLD_DATE=20180205
diff -rq //nx3000.ddci.com/ship/dds/windows/approved/DDS-${CUSTOMER}-deos-${BASELINE}-${OLD_DATE} //ddsbuild.ddci.com/DDCI_integration/DDS-${CUSTOMER}-deos-${BASELINE}-${NEW_DATE} | sort > DDS-${CUSTOMER}-deos-${BASELINE}-${NEW_DATE}.txt

Submit Request

Notify QA of your new release and wait for approval. An email should suffice, as long as it contains an attachment with the sorted diff file (created above), and a link to the release page. The following email has worked in the past:

A new candidate DDS for release has been created. The <insert release wiki page link here> release page captures the following information for your consideration:
* customer program
* release milestone
* previous distribution
* candidate distribution
* test reports for any changed or newly added components
* additional testing performed (if any)
* removed components (if any)
* test summary
* known problems

I am requesting that you approve this release for delivery.

If there is anything else I should provide you, please let me know.

Thanks

Congratulations! Time for a nap.

Release

Once approval has been granted and logged, one may use the secure FTP site to upload the release to the customer

If the release contains achieved, contractual milestones, the release announcement needs to contain:A message like the below where we deliver a contractual milestone should reference the milestone number, so that

• The milestone number as described in the statement of work
• A request that the customer respond, acknowledging receipt. Use contractual terms.

Example:

 This release completes the following DDC-I Software deliverables:
 1) DDC-I Quote Section 11.2.3 Item 11: [this is BSP Drop2, the first of three drops mentioned in the referenced section/item]
 2) DDC-I Quote Section 11.2.3 Item 15: 1- Preliminary Cache Trasher
 3) DDC-I Quote Section 11.2.3 Item 15A: 1- Target Platform Cache Trasher
 Please respond with an e-mail confirming you have received the above.

• Once the customer's acknowledgment is received, it should be logged in the Project Folder, category Deliverable.

Tool Maintenance

Updating the Wise Installer

In the unlikely event that a change needs to be made to the WISE installer, here's how you do it:

1. Reserve the ddsbuild machine on x9
2. Use Microsoft's Remote Desktop to logon to the ddsbuild machine (username: scoretest password: scteorste1)
3. Use Windows Explorer to double-click D:\DDCI_integration\DDS\DDCI_empty.wse. The "Wise InstallBuilder" dialog will now be in action.
4. Update the script as needed and save your changes.
5. From a cygwin terminal, run ./mk_dds_install. This will create a new setupDDCI.EXE
6. Check in your changes to SVN.