cdproc

From DDCIDeos
Jump to navigationJump to search

Technology for processing cd.xml files.

There is a teams chat and a wiki for OpenArbor's use of cdproc: OpenArbor_cdproc


Tasks/Issues

Ron test related issues:

  1. Current test infrastructure does not support TESTREG_INPUTS_NO_STDAPPS.
    • vfile 56 and 57 are good test cases.
  2. OA makefiles don't support warning suppression.
  3. QUIET not always working (when using OA makefiles)
  4. PIA resource startaddress expressions either not supported or not working, e.g., addition.
  5. MFS start address not specifiable
  6. Need ability to "unique-ify" registry names for specific tests
    • Partially implemented in cdproc, still not implemented in OA makefiles.
  7. Striping support.
    • Ron says low priority for tests since can address via .fp for now.

Problem statement

Much duplication between OpenArbor and the maintainer environment

  1. Building executables
    1. openarbor-common
    2. build-utils

  2. Building "platforms" (registries, config files, etc.)

    1. openarbor-common
    2. test-utils
    3. rtems-mini-env
    4. kernel debugger
    5. various component test environments
    6. test_elfchk
    as a consequence, maintaining the various environments is very expensive.

Some modeling is just not right

  1. cd.xml files are missing content, e.g., names.dat
  2. some components are not even represented in cd.xml files, e.g., 653
    • Most components have cd elements, e.g., itFiles, piaFiles, ioiFiles, vFiles, etc., 653 does not.
  3. There is nothing that corresponds to a system configuration:
    • A set of software, including registry and configuration files, that could be deployed on multiple hardware platforms.
    • Currently a platform project and a BSP are conflated with a system configuration.
      • A platform project is the only "cd like thing" where a registry can be defined.
    • A good example is a particular kernel test, e.g. tpk001, etc.
  4. accounting for LwIP apps like behavior.

OpenArbor is the only way to process cd.xml files

  1. Makes using the command line difficult, even with headless build.
    • As a consequence, scripting things like the execution of a series of tests is messy.
  2. Makes supporting other IDEs more difficult
  3. Adding a component that requires system configuration, e.g., vfile, dvms, etc. means that openarbor has to be "in the loop" for defining the integration, not just the GUI associated with editing the defined files.
    • Not scaleable, there are more developers adding stuff than OpenArbor staff.
    • Divorces component developer from the system integration considerations.
      • Since Aaron is not java literate there is another artificial limitation.

Examples

Trivial Example

 foo$ run-docker -it --rm --name=cdproc ubuntu-kismet-dev
 dkr:cdproc foo$ cdproc-makefile qemu-arm
 dkr:cdproc foo$ make boot
 dkr:cdproc foo$ startqemu    # Should show QEMU running the kernel and nothing else, i.e., a blank display.

 # To add network standard apps:
 dkr:cdproc foo$ cdproc-makefile --component development-support qemu-arm
 dkr:cdproc foo$ make boot
 dkr:cdproc foo$ startqemu    # Should show QEMU running with network and idle displays.

Mfs Example

To execute this example, the unreleased version of the desk-cdproc package is required. To install it do the following:

  1. unrelease --sudo install desk-cdproc
 dkr:cdproc foo$ rm -r *
 dkr:cdproc foo$ mkdir etc
                 # copy for editing...
 dkr:cdproc foo$ cp /desk/platform/qemu-arm/etc/openarbor.options etc/
 dkr:cdproc foo$ sed --in-place -e '/<\/componentDescriptor>/i\  <kfsFile>SCATHitmap.kfs.xml</kfsFile>'  etc/openarbor.options
 dkr:cdproc foo$ svn cat https://ddci.zapto.org/scm/Deos/products/desk/cdproc/branches/mainline/devel/SCATHitmap.kfs.xml > etc/SCATHitmap.kfs.xml
 dkr:cdproc foo$ sed --in-place -e '/<\/kfs>/i\    <kernelFileRegex component=".*vfile.*">.*\.so</kernelFileRegex>'  etc/SCATHitmap.kfs.xml 
 dkr:cdproc foo$ cdproc-makefile --debugComponent=kernel --extra-component=development-support --desk-root={/desk,$(pwd)} etc/openarbor.options 
 dkr:cdproc foo$ make 
 dkr:cdproc foo$ ls *SCAT*
 SCATHitmap.dkfs  SCATHitmap_loadlist.txt
 dkr:cdproc foo$ grep kernelFile SCATHitmap_loadlist.txt
 -kernelFile=libvfile-prl.so,/desk/arm/appbin/libvfile-prl.so
 -kernelFile=libvfile.so,/desk/arm/appbin/libvfile.so
 -kernelFile=libvfilesupport.so,/desk/arm/appbin/libvfilesupport.so

Proposals

cdproc

Move the generation of (the equivalent of) project-config.mk to a python library that would be invoked by openarbor.

  1. First for platforms, (maybe) later for executable projects.
  2. Prototype implementation is available at https://ddci.zapto.org/scm/Deos/products/desk/cdproc/branches/mainline/code/

KFS XML

Add a new XML file that declares how to construct KFS files. Initially for MFS files, later for LFS, or at least for the collection of files in an LFS.

The "official"/proposed schema is at https://ddci.zapto.org/scm/Deos/products/desk/cdproc/branches/mainline/code/kfs-1.0.xsd

This is a summary: 

<kfs>
  <type>LFS|MFS
  <underlyingResource>  Name of a platform resource.  
      The resource supplies the storage geometry.

  <kernelFileRegex component=regex scope={default|inclusive}>file_regex
      The component is a regular expression that matches the names of the components
      that this kernelFileRegex applies to.  The regular expression must match the entire
      component name, i.e., as if the regular expression started with '^' and ended with '$'.
      When empty means the regular expression refers to all components.

      scope means:
        inclusive: Files matched by this regular expression are always included in the KFS.
        default:   Files matched by this regular expression are included in the KFS
                   only when the file is not matched by some KFS's scope='inclusive' kernelFileRegex.
</kfs>

examples: 

mylib-binaries.kfs.xml:
  // All the .exe or lib*.so files from the "my-library" component.
  <kfs>
      <type>MFS</type
      <underlyingResource>executables</underlyingResource>
      <kernelFileRegex component="my-library">.*\.exe|lib.*\.so</kernelFileRegex>
  </kfs>
mylib-data.kfs.xml:
  // All the files from the my-library component that are not in executables
  <kfs>
      <type>MFS</type
      <underlyingResource>non-executables</underlyingResource>
      <kernelFileRegex component="my-library" scope="default"></kernelFileRegex>
  </kfs>

Enhance headless build

why? Because re-implementing OpenArbor's cd.xml processing will be expensive and has no (limited?) direct customer benefit, i.e., we don't get paid for this.

Perhaps we could just re-characterize stuff more directly with OpenArbor concepts, e.g., define a test as a project of some type.

Define tests via cd.xml

Regardless of the implementation, tests should be defined more in terms of cd.xml concepts.

  1. A test is defined, at least in part, by a cd.xml file.
  2. suggest a "system configuration".

Ron's Testing Notes

kernel tests:

  1. Relies on vfile general usage
  2. Some tests are download mode tests (multi-mode tests)
  3. Various memory pool configurations (more than 1 pool currently broken)
  4. Various customizations for multi-core (multiple schedulers)
  5. Test PRLs using MFS address (common/platform/config) a.k.a. kernel mode driver
  6. Interceptor based tests
  7. Kernel test specific resources
  8. MFS resources - PIAble
  9. Architecture specific tests
  10. Info: Tests query the registry for magic numbers.
  11. Info: Tests hack the registry for error conditions.

vfile tests:

  1. Needed custom vfile usage fp xml
  2. Used custom rules by adding searchPath to PI_OPTIONS for test directory.

dvms tests (DAL-D):

  1. Needed custom dvms config usage fp xml
  2. Needed custom dvms vfile xml

general issues for tests:

  1. TLS usage and compiler/link flags
  2. Missing resources - PIAble
  3. Rules for tools with more than one output (fixed in make 4.0 &:)


Retrieved from "https://ddci.zapto.org/ddciWiki/index.php?title=Cdproc&oldid=72358"