OpenArbor Development

From DDCIDeos
Jump to navigationJump to search

This is how to setup a development environment for developing OpenArbor, debugging OpenArbor, running the automated test system, and other project-management activities.

For a quick guide to building and installing OpenArbor, see OpenArbor Build.

See also OpenArbor Testing.

It is assumed that all command line operations will be done in a bash shell in a desk maintainer environment.

Setting Up A Development Environment

Windows

To setup a development environment on a Windows machine, see OpenArbor_Development_Windows.

Docker

To setup a development environment on a Linux machine, or in WSL see OpenArbor_Development_Docker.

Version Numbers and Branches

TODO: This section is too long, and this process is too complicated. We need more direct explanations (with code examples), and someday, we should automate some of this.

OpenArbor's version number is 4 components, "<major>.<minor>.<patch>.r<revision>". The revision component is automatically inferred in the Makefile from svn metadata. The base version is the first 3 components. The base version is stored in the workspace, in build-utils/version.txt, and it is maintained by manual editing.

Creating a Branch

These are the reasons to create an OpenArbor branch:

  • If you're about to start a long project that will create regressions that aren't planned to be fixed for months (such as for DDCI_PCR:478), then create a feature branch. Name the branch a short phrase, preferably a single word, that describes the objective of the branch, such as "timemap". The name of a branch should follow the company-wide convention of all-lowercase names with words separated by hyphens.
  • If an OpenArbor release is approaching, and we want to put the code into "feature freeze" (aka "code freeze"), then create a release branch. Name the branch the current base version, such as "5.2.0". After branching, bump the minor version in mainline, such as to "5.3.0". This will create two branches that have different base versions.

For the remainder of this discussion, we will use the name "OTHER" to refer to the name of the branch being managed.

First, create a PCR for managing the branch. See DDCI_PCR:2194 for an example. For the remainder of this discussion, we'll refer to this PCR as "PCR nnnn".

Create a branch like this: 

svn cp https://deos.ddci.com/svn/DDCI/products/openarbor/openarbor/branches/{mainline,OTHER} -m "PCR nnnn. creating branch OTHER from branch mainline"

It is our official policy to only create branches from mainline.

After creating the release branch, add a section for the new version to workspace/com.ddci.common.ide/OpenArbor_docs/release_notes/release_notes.docbook. Commit the release notes changes to PCR nnnn.

NOTE: it is important to make at least 1 commit in a release branch before creating a release from the branch with make stable. The <revision> portion of the version number is the "Last Changed Rev" according to svn metadata, and if there have been no commits since creating a release branch, then this revision will point to a commit back on mainline rather than a commit in the release branch. It is important that the <revision> version point to a commit in the proper branch so that the dds-manager automatically guesses which openarbor branch to use for the test system.

Develop OA Plugins in Another Branch

If you haven't already, get the branch on your local machine with this: 

svn up /cygdrive/c/ddci_eclipse/branches

If you'd like to preserve all your workspace preferences, you can copy over the entire .metadata directory. Be sure to close any eclipse instances before doing this. 

rm -rf /cygdrive/c/ddci_eclipse/branches/OTHER/workspace/.metadata
cp -r /cygdrive/c/ddci_eclipse/branches/{mainline,OTHER}/workspace/.metadata

Follow the #Workspace configuration section, using the OTHER branch instead of mainline.

Done with a release branch

When a release has been approved by QA or is in some other way determined to be done, follow these steps.

  • Merge any changes made in the release branch into mainline.
  • Find the revision number of the commit that created the branch. If the above process was followed, there should be a PCR for managing the branch, which we'll call PCR nnnn, and the branch creation will be committed against the PCR. We'll call the revision that created the branch START.
cd /cygdrive/c/ddci_eclipse/branches/mainline
  • Make sure the working copy is clean. svn st should say nothing.
  • Perform the merge in your working copy of mainline.
svn merge https://deos.ddci.com/svn/DDCI/products/openarbor/openarbor/branches/OTHER -rSTART:HEAD
  • Review and resolve any conflicts (see #Resolving Conflicts below).
  • Watch out for conflicts in build-utils/version.txt.
  • When merging a release branch, preserve the mainline version, which should be ahead of the release branch.
  • When merging a feature branch, use engineering judgement to decide what the base version should be.
  • Checkin the changes. Note that doing this from within eclipse will miss some important metadata. Always do this step from the command line.
svn ci -m "PCR nnnn. merging branch OTHER into branch mainline"
  • Delete the branch
svn rm https://deos.ddci.com/svn/DDCI/products/openarbor/openarbor/branches/OTHER -m "PCR nnnn. deleting branch OTHER"
  • Resolve PCR nnnn as fixed.

Keeping a feature branch up to date

Due to the nature of a feature branch (see above), it is expected that mainline will be updated throughout the lifetime of a feature branch. To keep the feature branch up-to-date with the mainline changes, follow these steps. Recall that there should be a PCR dedicated to managing the feature branch; we'll call this PCR nnnn.

  • Determine the start revision for the merge. We'll call this revision START. Refer to PCR nnnn for the merge history of the feature branch thus far.
    • Look for the most recent PCR nnnn comment like "merging branch mainline into branch OTHER". START is the revision of this commit.
    • If no such comment exists, START is the revision when the branch was created.
cd /cygdrive/c/ddci_eclipse/branches/OTHER
  • Make sure the working copy is clean. svn st should say nothing.
  • Perform the merge in your working copy of OTHER.
svn merge https://deos.ddci.com/svn/DDCI/products/openarbor/openarbor/branches/mainline -rSTART:HEAD
  • Review and resolve any conflicts (see #Resolving Conflicts below).
  • Checkin the changes. Noting the revision range in the commit makes future update merges a lot easier.
svn ci -m "PCR nnnn. merging branch mainline into branch OTHER revisions nnnnn to nnnnn"

Done with a feature branch

When the feature branch is stable enough for internal use, follow these steps. Recall that there should be a PCR dedicated to managing the feature branch; we'll call this PCR nnnn. 

svn rm https://deos.ddci.com/svn/DDCI/products/openarbor/openarbor/branches/mainline -m "PCR nnnn. deleting branch mainline"
  • Move the feature branch to be the new mainline.
svn mv https://deos.ddci.com/svn/DDCI/products/openarbor/openarbor/branches/{OTHER,mainline} -m "PCR nnnn. renaming branch OTHER to branch mainline"
  • Resolve PCR nnnn as fixed.

Recover a deleted branch

Refer to the PCR dedicated to managing the branch and note the revision number that deleted the branch. To recover, copy the branch using the revision number just before the revision that deleted the branch.

From PCR 3138, revision 39414 deleted branch 6.2.0: Committed SVNRevision 39414.

deleting branch 6.2.0

To restore, cp the branch @ the revision just before the deletion: 

  • svn cp https://deos.ddci.com/svn/DDCI/products/openarbor/openarbor/branches/{6.2.0@39413,6.2.0} -m"PCR 3138. restore 6.2.0 branch"

Resolving Conflicts

Conflicts create a number of files, some that end with ".working", so searching for those files should point you at all the conflicts.

  • If you are resolving conflicts and know which one you want to keep, my-conflict is mainline, and theirs-conflict is the release branch.
  • Look for the "<<<<<<", "======", and ">>>>>>" in conflicted files. Edit the files and make them what they're supposed to be.
  • Delete the extraneous files (there are always 3 extraneous files per conflict: "*.left.r*", "*.right.r*", and "*.working")

Adding new icons

New icons should be created in the com.ddci.common.ide.icons folder. The other plugins and the docs directory have an svn external definition to get the icons at a particular svn rev. When adding a new icon, the external definition for one of these may need to be updated.

lcj@LCJLap3 /cygdrive/c/ddci_eclipse/branches/mainline/workspace/docs
$ svn propset svn:externals "/svn/DDCI/products/openarbor/openarbor/branches/mainline/workspace/com.ddci.common.ide/icons@97987 icons" .
property 'svn:externals' set on '.'

Debugging / Searching Eclipse Source code

Open the Plug-ins View, select all the plug-ins, right-click, and select "Add to Java Search" Then you can use Open Type (Ctrl-Shift-t) to search for class names and the source will be attached so you can read it, set breakpoints and debug.

Debugging / Searching CDT Source code

It is possible to debug the CDT source from Eclipse, as long as you provide the path to the source. Clone the CDT repository to your local machine: git clone --branch CDT_11_2_0 https://github.com/eclipse-cdt/cdt.git.
(Note, this command is for CDT version 11.2.0, adjust accordingly)

Once your debugger reaches CDT code, select the "Edit source lookup path..." button that appears, enter the path to the /cdt repository on your machine, and select 'OK'. Eclipse will automatically find the file needed from the repo and allow you to continue debugging from that point.

Just as with Eclipse source, you can use Open Type (Ctrl-Shift-t) to search for CDT class names and the source will be attached so you can read it, set breakpoints and debug.

Debugging A Child Test in OA

In MiscUtil.launchAnotherOpenArbor(), pass additional JVM args to the headless child OA instance to enable remote debugging using JDWP (Java Debug Wire Protocol). Include the following argument in the JVM args:

"-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=<any-open-port>"

This tells the JVM to load the JDWP agent and listen for debug connections over TCP/IP (dt_socket) on the port defined by address. It will suspend execution at startup (suspend=y), until a debugger attaches, providing time to attach with a 'Remote Java Application' from the eclipse environment.

After making this change, launch a debug on a test that launches another child OA. Execution will stop once the suspend mentioned above occurs. At this point, the parent OA console will report "launching child test".

In the eclipse environment, open Debug Configurations, and create a new 'Remote Java Application'. Set the Project to 'com.ddci.openarbortests', the Port to <any-open-port>, and the Host to 'localhost'.

Launch the debug to attach to the JVM of the child OA. From this point, the debugger can control execution, set breakpoints, etc. in the child JVM.

OpenArbor Source Guide

OpenArbor is mostly comprised of Eclipse features. An Eclipse feature is a grouping of one or more plugins that can be easily loaded, managed and branded as a single unit. Each feature is a Feature Project in Eclipse, containing a few files that define the feature. Most interesting, is the feature.xml file. Double-clicking this file and viewing the "Plug-ins" tab in the resulting editor gives a list of the plugins that comprise that feature.

There are two features:

  • com.ddci.common_feature

The common feature contains all the plugins that are delivered to customers.

  • common feature plugins
    • com.ddci.common.editors - Editors for Ada, Asm, Disassembly, Dump, Suprograms
    • com.ddci.common.help - Context Help for the IDE
    • com.ddci.common.ide - "main" plugin for the OpenArbor IDE views
    • com.ddci.deos.common - Deos support plugin for building Deos xml files and Deos Views
    • com.ddci.deos.targetmanager - Load List, Status Monitor support plugin
    • com.ddci.gcc.asm - Gnu Asm
    • com.ddci.gnu.c - Gnu C Compiler
    • com.ddci.gnu.common - Compile/Link Options, Error handling for compile/linking
    • com.ddci.gnu.gcc_cpp - Gnu C++ Compiler
    • com.ddci.mld - multi-language debugger support plugin
    • com.ddci.rtems - OAR (DDC-I partner) support plugin
    • com.ddci.score.ada - Deos for Ada plugin
  • com.ddci.openarbortest_feature

The openarbortest feature contains the OpenArbor automated test suite. This feature is not delivered to customers. The feature is installed separately when running formal tests for releases.

Conventions and Style

Refactor as you work
These policies are not just for new code. Clean up old code if you happen to be working in it.

  • boolean vs Boolean: always use boolean when possible. A parameter or field of type Boolean means that the value can be null. The same applies to all the primitive wrapper classes, such as Integer.
  • exception breakpoints: it's advisable to add a Java Exception breakpoint for NullPointerException, TimeoutException, and WidgetNotFoundException. These are the most common exceptions that indicate OA developer error, and it's usually good for maintainers to be able to examine the context in which these exceptions where thrown.
  • no pokemon programming: Never catch and ignore the above exception types (that includes catching Exception). This is so that they can be useful for the purpose stated above. It's acceptable to catch and rethrow the exceptions wrapped in another type or something. As long as the problem is loudly brought to the attention of an OA developer. In cases where it seems appropriate to catch such exceptions, consider these alternatives:
    • check for bad values: Instead of catching NullPointerException, check for null. Instead of catching IndexOutOfBoundsException, check the bounds. Etc.
    • throw a different exception: In cases where an exception of some type is appropriate, consider throwing a custom class that callers can catch to distinguish certain error conditions from programmer errors. For example, throw a SomethingParseException when serialized data does not conform to the format you expected, instead of allowing an ArrayIndexOutOfBoundsException to be thrown.
  • throw null: For code that should never be reached, the statement "throw null;" will result in throwing a NullPointerException. If the above recommendations are followed, any execution of that line of code will effectively be a breakpoint, and alert OA developers.
  • Name things for readability, not writability:
    • Use correct names rather than abbreviated names for fields and constants. Rather than "retVal", use "returnValue" or "result". Rather than "lst", use "list". Rather than "relPath", use "relativePath".
    • Some abbreviations are well established enough to use: arg, param, abs, etc. Parameters and local variables can be shortened as long as the meaning is still clear within context.
    • Data should be nouns; methods should be verbs (except for trivial getters, which effectively turn the methods into fields).
    • Collections (and arrays, etc.) should be plural.
    • Don't say "toggle" when you mean "set".
    • Don't use "m_" prefixes. Prefer "this." when necessary.
    • Don't use hungarian notation. Instead of "szName", use "nameString" or just "name". Instead of "alArgs", use "argList" or just "args".
    • The noun "target" is ambiguous. (We need to do something about that in our codebase.) Consider "targetArchitecture", "remoteTarget", "debugTarget", etc.
  • Parent classes should never reference child classes.
    • Rather than if (this instanceof Child) ..., the child should override a method to provide the different behavior.
  • Use CopyOnWriteArrayList for listener lists, so that modification and access can happen from any thread without worry.
  • Methods should return Lists rather than arrays.
    • When returning an empty list, use Collections.emptyList().
    • Unless otherwise specified, modification of a List returned from a method is forbidden. If needed, make a copy, i.e. new ArrayList<>(list).
      • This should be enforced by returning an unmodifiable collection.
  • System.*.println does not belong in production code. Catch it, log it, or throw it.
  • Several areas of the codebase have a lot of Hungarian notation. Don't perpetuate this pattern, and refactor it away wherever you encounter it.

Wiki Generation

Each release wiki page (such as OpenArbor 5.3.0) has a section whose contents are generated by a script.

The wiki generator script is here: https://deos.ddci.com/svn/DDCI/products/openarbor/openarbor/branches/mainline/workspace/build-utils/generate_wiki_details.py

Invoke the tool like this: 

/path/to/generate_wiki_details.py --target=TARGET_RELEASE > /dev/clipboard

where TARGET_RELEASE is the target milestone, such as 5.3.0. The output file /dev/clipboard means that the contents will be copied to your clipboard. Next edit the auto generated section of the wiki page, select all, paste, (optionally review the changes,) and save.

The wiki generator recognizes three bug states: Open, need tests, and completed. This is based off the flags on the bugs, specifically marking Code, Other, and TestCases + or -. If all are blank, it's still open. If Code and Other are filled in, it needs tests written. If all are filled in, it's completed.

Python Development

If you'd like to use python's built-in command line debugger, insert this line of code as a "breakpoint": 

import pdb; pdb.set_trace()

Then run the script from the command line (obviously). When the "breakpoint" line of code is encountered, you'll get a command prompt. Here are the commands you want to know:

  • s(tep): step into
  • n(ext): step over
  • r(eturn): step return
  • c(ont(inue)): go
  • l(ist): show some source code, and then show some more source code.
  • any python expression: evaluate it and print it.

See also: http://docs.python.org/2/library/pdb.html#debugger-commands

MLD Communication Debugging

If something's wrong with OpenArbor and MLD talking to each other, here are some debugging tips.

Enable a communication dump

  1. Launch OpenArbor with the --enableDebugLog option. This can be done from the command line, or by putting it in OpenArbor.ini before the -vmargs line.
  2. Reproduce the problem while running OpenArbor with this option enabled.
  3. The dump is located in the workspace/.metadata/scoremessages_<project>.log

You can view the log directly, or use this viewer utility: https://deos.ddci.com/svn/DDCI/products/openarbor/utils/abridge_messages.py

Invoke it with: 

python abridge_messages.py path/to/scoremessages_<project>.log | less -RXFS

This sets up `less` to view the log and enable horizontal scrolling (with left and right arrows).

Read the documentation in abridge_messages.py for more info.

Enable the MldCommandQueue tracer

Invoke OpenArbor with the switch: --mldCommandQueueLogger PATH

Retrieved from "https://ddci.zapto.org/ddciWiki/index.php?title=OpenArbor_Development&oldid=75073"