Python3 Port
From DDCIDeos
Description
This project is to update all the python scripts in the DDS to use python3.
Porting
linux> 2to3-2.7 --fix=all -w *.py cygwin> 2to3 --fix=all -w *.py
Review result. It is normally fine to accept the changes to print() and exception syntax (such changes are typically compatible with both python2 and python3).
Changes to imported module names should only be accepted when changing a command's shebang line to use python3. Such changes should normally not be made to importable modules unless all the programs using the module have been changed to use python3.
Fix the pylint warnings. Pylint is good at finding functions that no longer exist in python3 and many common syntax errors.
Issues not fixed by the 2to3 script:
string.split("someString", ...) --> "someString".split(...)
string.replace same as split
string.join(some_list, "someString") --> "someString".join(some_list)
file(name, options) --> open(name, options)
- In python2 there was little distinction between bytes and str. Python3 is much more strict. Typically this affects:
- open() not having correct "b"inary or "t"text options and the use of the resulting data read from the file.
- Use of cStruct and CHAR data, in python2 the result of unpack was str, in python3 the result is bytes.
- subprocess execution, specifically input or output is typically binary unless universal_newlines=True
- Integer division using operator "/" now yields a floating point. To retain integer math you must use "//".
- Update the shebang line (see PEP 0394 for background):
- For programs the shebang line should be:
#!/usr/bin/python3
- For modules:
#!/usr/bin/python3 # Shebang line is ignored since this is a module. Adding python3 to record it has been ported to python3.
- Once the port activity is complete, the two lines could be removed. For now it helps track script status.
- For programs the shebang line should be:
- Components using HTMLgen can use the (newly DDCI installed) htmlgen2 module. Sadly the interface is not very compatible.
- For an examples of the changes see the:
- The hypdump port triggered the generation of the ddcihtml porting module. Talk to Aaron if you decide to use it so the interface can be formalized.
- Update externals to latest python-wrapper
- The new python-wrapper sniffs whether the script is using python2 or 3 and generates an appropriate wrapper.
- (optional) If any of the installed scripts are commands, and they are not referenced by other programs in the DESK using their .py suffix (none are supposed to be, but latent defects abound), remove the .py suffix from the command. See SVNRevision 80463, notably startqemu and makefile.mk for the steps. In brief; svn mv foo.py foo, remove .py suffix from makefile.
- If the script references DESK commands using .py suffix (e.g., calls "putall.py"), change the script to not use the suffix (e.g., call "putall" instead).
- This change has nothing to do with python3, but calling via .py suffix is a latent defect in older scripts.
- Review the pylint output for the script.
- Ideally use a distribution newer than cygwin 2018, e.g., a docker image.
- If you must use Cygwin 2018's pylint note that it spuriously generates warnings of this sort (which are bogus):
Unnecessary parens after 'print' keyword (superfluous-parens)
- (optional) If the script uses getopt or optparse convert to argparse.
- Converting scripts that use toolcrib is also acceptable, but the effort is larger.
- Some examples of conversion of DDCI scripts are startqemu and strip_deos.
- (optional) Change indentation to 4 spaces per level
- This complies with PEP8 and makes pylint much happier.
- On Linux (e.g., from within a docker container):
- sudo apt-get install python3-pip
- sudo pip3 install reindent
- reindent *.py
- It may make sense to do this step after testing the port, and with a separate commit.
Specific cases
- MakeBoot extensions: See qemu-aarch64's makeboot_ext.py for an example.
- Testing can be done via: python3 /desk/bin/makeboot.py makeboot args ...
Tasks
XML-Tools means tools based on xml-tools-common. Currently xml-tools-common is not sufficiently converted to support python3.
See also current PCR status:
| Task | Priority | Assignee | XML-Tools? | Status | Remarks |
|---|---|---|---|---|---|
| abc-tool | 5 | JK | No | Done-ish | GK 20230825: Noted that abc-tool/package2cygwin.py has "requires: automake bash diffutils perl python2 %s" |
| afdx-driver-cvt | 5 | Yes | Pending | Component is OBSOLETE. | |
| afdx-library-cvt | 5 | Yes | Pending | Component is OBSOLETE. | |
| cffsdump | 5 | No | Pending | ||
| codetrace | 8 | No | Done | ||
| cffsapi653p2 | 5 | GK | No | Done | PCR:15056 unreleased version 3.0.2 |
| crittime-tool | 5 | AL | No | Done | |
| deos653config | 5 | GK | Yes | Done | unreleased version 1.27.6 |
| deos653-cvt | 5 | Yes | Done | ||
| deosbook-devel | 8 | No | Done | ||
| desk-python-tools | 3 | AL | No | Done | Adds python3 to Cygwin. |
| dvms-config | 3 | CP | Yes | Done | stable version 1.3.1 |
| hyperstart(modules) | 1 | AL | No | Done | |
| hyperstart (scripts) | 3 | AL | No | Done | |
| integ-tool-command-line | 3 | CF | Yes | Done | PCR:14379 Released 3.62.0 |
| ioi-config-tool | 5 | GK | Yes | Done | unreleased version 3.6.8 |
| ioi-cvt | 5 | GK | Yes | Done | unreleased version 1.9.0 |
| ist-arinc653-examples | 5 | AL | No | Done-ish | Not released |
| ist-config | 5 | GK | Yes | Done | unreleased version 1.4.1 |
| ist-ioi-examples | 4 | AL | No | Done-ish | Not released |
| kernel | 3 | Ron | No | Done-ish | Python scripts and modules in kernel code and docs are ported. Test scripts are still pending. |
| kfs-cvt | 5 | CF | Yes | Done | |
| openarborEclipse_64 | 5 | AL/LCJ | No | Done | DDCI_PCR:4600 |
| openarborMaintainerTools | 8 | AL/LCJ | No | Done | DDCI_PCR:4594 |
| pci-config | 5 | GK | Yes | Done | unreleased version 1.1.1 |
| qemu-aarch64-boot | 3 | AL | No | Done | |
| qemu-arm-boot | 3 | AL | No | Done | PCR:14574 |
| qemu-ppc-boot | 3 | RLR | No | Done | |
| qemu-x86-boot | 3 | AL | No | Done | PCR:14575
|
| registry-cvt-win32 | 5 | Yes | Done | ||
| score-other-docs | 5 | No | Pending | ||
| socket-examples | 4 | AL | No | Done-ish | Not released. |
| test-infrastructure | 8 | AL | No | Done | |
| trace32-extension | 5 | No | Pending | ||
| traceaid | 8 | No | Done | ||
| vfileconfig | 3 | GK | Yes | Done | unreleased version 1.8.0 |
| x9-client | 8 | AL | No | Done-ish | PCR:14568 Client code ported and unreleased, server code still py2. |
| xml-tools-common | 8 | GK | Yes | Done | PCR:13965 |
| build-utils | 8 | AL | No | Done | |
| test-utils | 8 | AL | No | Done | PCR:14576 |
| deos2cygwin | 9 | No | Pending | ||
| review-process-scripts | 9 | AL | No | Done |
Tracking Progress/Status
| Date | Total /desk python files | Ported to Python3 |
|---|---|---|
| 2021-10-05 | 129 | 56 |
| 2021-10-29 | 129 | 74 |
| 2022-04-25 | 161 | 89 |
| 2023-06-12 | 218 | 132 |
| 2023-07-11 | 224 | 147 |
To get the above numbers paste the following into a desk console:
pyFiles=$(for f in $(find /desk -type f | grep -v __pycache__); do if file $f | grep -i Python > /dev/null; then echo $f; fi; done)
printf "|-\n| %s || %s || %s\n" $(date '+%Y-%m-%d') \
$(ls $pyFiles | wc -l) \
$(for f in $pyFiles; do echo $f: $(head -1 $f); done | grep 'python3' | wc -l)
To get the list of packages that have python2 files, and how many files in each package:
pkgPyFiles=$(for f in $(for f in $pyFiles; do echo $f: $(head -1 $f); done | grep -v 'python3' | cut -d: -f1); do dpkg -S $f | sed -e 's/ /:/' ; done | sort -u) pkgs=$(echo "$pkgPyFiles" | cut -d: -f1 | sort -u) for p in $pkgs; do printf "%3d %s\n" $(echo "$pkgPyFiles" | grep "$p:" | wc -l) $p; done | sort --key=2d | sort --key=1rn
Results on Aaron's maintainer docker image 2023-07-11:
25 ist-config-win32 23 ioi-config-tool 14 ioi-cvt 9 desk-python-tools 1 deos653-config-win32 1 ist-arinc653-examples 1 ist-ioi-examples 1 qemu-x86-64-boot 1 qemu-x86-boot 1 socket-examples 77 Total
Task Raw Data
The following is a list of all the packages from a cygwin setup.ini file circa 2021-10-01:
Non-boot components DDCI responsible for:
abc-tool afdx-driver-cvt afdx-library-cvt cffsdump codetrace crittime-tool deos653-config-win32 deos653-cvt deosbook-devel desk-python-tools heartos-python-tools heartos-test-infrastructure hyperstart integ-tool-command-line ioi-config-tool ioi-cvt ist-arinc653-examples ist-config-win32 ist-ioi-examples kernel kfs-cvt openarborEclipse_32 openarborEclipse_64 openarborMaintainerTools pci-config registry-cvt-win32 score-other-docs socket-examples test-infrastructure trace32-extension traceaid vfileconfig x9-client
BSP Boot components:
arrow-sockit-boot bsp_mpc603e_pqii82xx bsp_mpc603e_psim bsp_mpc603e_rattler8280 celestial chino-fcm cm-crb-boot dpspl017x-boot dvmsconfig ep440c-platform-boot ep8280-ppc-platform-boot ep8343m-boot fit-pc1-boot fit-pc2-boot harrys-boot hosmer-boot hotdish-aid-boot imb-a180-h-boot intel-mc-boot kalbi-boot louiev2-boot lutefisk-aid-ppc-platform-boot lutefisk-iv-boot maple-boot minnow-turbot-dual-boot minnow-turbot-quad-boot mpc8315e-boot mpc8349e-mitx-gp-platform-configuration nai75arm1-boot nitrogen6x-som-boot p1010rdb-boot p2041rdb-boot pmc5675-platform-configuration qemu-arm-boot qemu-mips-boot qemu-ppc-boot qemu-x86-boot rattler8280-ppc-platform-configuration vmware-boot vpx6187-boot wade8078-boot xcalibur1002-platform-configuration zcu102 zu5
Supplier provided components:
deos-rapita deos-rtems-examples deos-rtems-tools-arm deos-rtems-tools-ppc deos-rtems-tools-x86 gcc-aarch64-elf-libs gcc-common gcc-common-libs gcc-ppc-elf-libs gcc-ppc-elfspe-libs gcc-x86-elf-libs gcc-x86_64-elf-libs gdb-cross-debuggers htmlgen2 pyPdf
