View source for Python3 Port

From DDCIDeos
You do not have permission to edit this page, for the following reason:
The action you have requested is limited to users in the group: Users.
You can view and copy the source of this page.
= Description = 
This project is to update all the python scripts in the DDS to use python3.

= Porting =

  linux>  2to3-2.7 --fix=all -w *.py
  cygwin> 2to3     --fix=all -w *.py

Review result.  It is normally fine to accept the changes to print()
and exception syntax (such changes are typically compatible with both
python2 and python3).

Changes to imported module names should only be accepted when changing a
command's shebang line to use python3.  Such changes should normally
not be made to importable modules unless all the programs using the
module have been changed to use python3.

Fix the pylint warnings.  Pylint is good at finding functions that no longer exist
in python3 and many common syntax errors.

Issues not fixed by the 2to3 script:
  string.split("someString", ...)  --> "someString".split(...)
  string.replace same as split
  string.join(some_list, "someString")  --> "someString".join(some_list)
  file(name, options)   -->  open(name, options)

# In python2 there was little distinction between bytes and str.  Python3 is much more strict.  Typically this affects:
## open() not having correct "b"inary or "t"text options and the use of the resulting data read from the file.
## Use of cStruct and CHAR data, in python2 the result of unpack was str, in python3 the result is bytes.
## subprocess execution, specifically input or output is typically binary unless universal_newlines=True
# Integer division using operator "/" now yields a floating point.  To retain integer math you must use "//".
# Update the shebang line (see [https://www.python.org/dev/peps/pep-0394/#recommendation PEP 0394] for background):
## For programs the shebang line should be:
##: <pre>#!/usr/bin/python3</pre>
## For modules:
##: <pre>#!/usr/bin/python3&#xa;# Shebang line is ignored since this is a module.  Adding python3 to record it has been ported to python3.</pre>
##: Once the port activity is complete, the two lines could be removed.  For now it helps track script status.
# Components using HTMLgen can use the (newly DDCI installed) htmlgen2 module.  Sadly the interface is not very compatible.
## For an examples of the changes see the:
### [{{SERVER}}/viewscm/Deos/products/desk/python-tools/branches/mainline/code/deosdoc.py?r1=75678&r2=75677&pathrev=75678 changes made to deosdoc]
### [{{SERVER}}/viewscm/Deos/products/kernel/hypstart/branches/mainline/code/python/hypdump.py?r1=75604&r2=75729 changes make to hypdump]
## The hypdump port triggered the generation of the [{{SERVER}}/viewscm/Deos/products/kernel/hypstart/branches/mainline/code/python/ddcihtml.py?revision=75732&view=markup ddcihtml porting module].  Talk to Aaron if you decide to use it so the interface can be formalized.
# Update externals to latest python-wrapper
## The new python-wrapper sniffs whether the script is using python2 or 3 and generates an appropriate wrapper.
# (optional) If any of the installed scripts are commands, and they are not referenced by other programs in the DESK using their .py suffix (none are supposed to be, but latent defects abound), remove the .py suffix from the command.  See [https://deos.ddci.com/viewscm/Deos?view=revision&revision=80463 SVNRevision 80463], notably startqemu and makefile.mk for the steps.  In brief; svn mv foo.py foo, remove .py suffix from makefile.
# If the script references DESK commands using .py suffix (e.g., calls "putall.py"), change the script to not use the suffix (e.g., call "putall" instead).
## This change has nothing to do with python3, but calling via .py suffix is a latent defect in older scripts.
# Review the pylint output for the script.
## Ideally use a distribution newer than cygwin 2018, e.g., a docker image.
## If you must use Cygwin 2018's pylint note that it spuriously generates warnings of this sort (which are bogus):
##:  <pre>Unnecessary parens after 'print' keyword (superfluous-parens)</pre>
# (optional) If the script uses getopt or optparse convert to [https://docs.python.org/3/library/argparse.html#upgrading-optparse-code argparse].
## Converting scripts that use toolcrib is also acceptable, but the effort is larger.
## Some examples of conversion of DDCI scripts are [https://deos.ddci.com/viewscm/Deos/products/desk/python-tools/branches/mainline/code/startqemu?r1=80463&r2=80462&pathrev=80463 startqemu] and [https://deos.ddci.com/viewscm/Deos/products/desk/python-tools/branches/mainline/code/strip_deos.py?r1=80466&r2=80465&pathrev=80466 strip_deos].
# (optional) Change indentation to 4 spaces per level
## This complies with [https://peps.python.org/pep-0008/ PEP8] and makes pylint much happier.
## On Linux (e.g., from within a docker container):
### sudo apt-get install python3-pip
### sudo pip3 install reindent
### reindent *.py
## It may make sense to do this step after testing the port, and with a separate commit.

== Specific cases ==

# MakeBoot extensions: See [{{SERVER}}/viewscm/Deos/products/bsp/qemu-aarch64/branches/mainline/boot/code/makeboot_ext.bangvar.py?r1=75736&r2=75735&pathrev=75736 qemu-aarch64's makeboot_ext.py] for an example.
## Testing can be done via: python3 /desk/bin/makeboot.py makeboot args ...

<!--
Random notes:
  for f in *.py; do sed --in-place '1s/python/python3/' $f; done
  for f in *.py; do sed --in-place '1s/env python/python/' $f; done
-->

== Python 3.12 ==

Ubuntu 2024 uses python 3.12 as the python3 program.  Python 3.12 has some additional porting requirements.  The following are ones we've seen.

# Backslashes within strings that are not python defined escape sequences generate warnings.  For example "\g" in a regular expression generates a warning since python does not define "g" as an escape sequence, even though it is a valid regular expression escape.  The easiest way to resolve this is to make the string an "r-string", for example:  <tt>r"\g"</tt>.
# the [https://docs.python.org/3/library/functions.html#open open()] function no longer accepts the "U" mode character.  See this [https://stackoverflow.com/questions/56791545/what-is-the-non-deprecated-version-of-open-u-mode stackoverflow posting] and the [https://docs.python.org/3/whatsnew/3.11.html python3.11 release notes].

See also:
* https://docs.python.org/3/whatsnew/3.11.html
* https://docs.python.org/3/whatsnew/3.12.html

= Tasks =

XML-Tools means tools based on xml-tools-common.

<font size="3">See also [{{server}}/bugzilla/showdependencytree.cgi?id=13715&hide_resolved=1 current PCR status]: </font>

{| class="wikitable sortable mw-collapsible" border="1"
|- align="center" style="background: silver"
! Task     !! Priority !! Assignee !! XML-Tools? !! Status !! Remarks

|-
| abc-tool				||  5 || JK  || No  || {{Work2}}
| GK 20230825: Noted that abc-tool/package2cygwin.py has "requires: automake bash diffutils perl python2 %s"

|-
| afdx-driver-cvt			||  5 ||     || Yes || Pending
| Component is OBSOLETE.

|-
| afdx-library-cvt			||  5 ||     || Yes || Pending
| Component is OBSOLETE.

|-
| arinc653-runtme tests			||  3 || MS  || No || Pending
| create-mms-pi-xml.py fixed on experimental Revision 94263 

|-
| cffsdump				||  5 ||     || No  || Pending
|

|-
| codetrace				||  8 ||     || No  || {{Done}}
|

|-
| cffsapi653p2				||  5 || GK  || No  || {{Done}}
| [[PCR:15056]] unreleased version 3.0.2

|-
| crittime-tool				||  5 || AL  || No  || {{Done}}
| 
|-
| deos653config		||  5 ||  GK || Yes || {{Done}}
| unreleased version 1.27.6

|-
| deos653-cvt				||  5 ||     || Yes || {{Done}}
|

|-
| deosbook-devel			||  8 ||     || No  || {{Done}}
|

|-
| desk-python-tools     	      	||  3 || AL  || No  || {{Done}}
| Adds python3 to Cygwin.

|-
| dvms-config                           ||  3 || CP  || Yes || {{Done}}
| stable version 1.3.1

|-
| hyperstart(modules)                   ||  1 || AL  || No  || {{Done}}
| 

|-
| hyperstart (scripts)                  ||  3 || AL  || No  || {{Done}}
| 

|-
| integ-tool-command-line		||  3 || CF  || Yes || {{Done}}
| [[PCR:14379]] Released 3.62.0

|-
| ioi-config-tool			||  5 || GK  || Yes || {{Done}}
| unreleased version 3.6.8

|-
| ioi-cvt				||  5 || GK  || Yes || {{Done}}
| unreleased version 1.9.0

|-
| ist-arinc653-examples			||  5 || AL  || No  || {{Done}}
| released in [[Deos_Sales_Kismet_1A]]

|-
| ist-config       			||  5 || GK  || Yes || {{Done}}
| unreleased version 1.4.1

|-
| ist-ioi-examples			||  4 || AL  || No  || {{Done}}
| released in [[Deos_Sales_Kismet_1A]]

|-
| kernel				||  3 || Ron || No  || {{Work2}}
| Experimental branch: problems.py is only remaining file. Others pending verification.

|-
| kfs-cvt				||  5 || CF  || Yes || {{Done}}
|

|-
| openarborEclipse_64			||  5 ||| AL/LCJ || No  || {{Done}}
| [[DDCI_PCR:4600]]

|-
| openarborMaintainerTools		||  8 || AL/LCJ || No  || {{Done}}
| [[DDCI_PCR:4594]]

|-
| pci-config				||  5 || GK  || Yes || {{Done}}
| unreleased version 1.1.1

|-
| qemu-aarch64-boot			||  3 || AL  || No  || {{Done}}
|

|-
| qemu-arm-boot				||  3 || AL  || No  || {{Done}}
| [[PCR:14574]]

|-
| qemu-ppc-boot				||  3 || RLR || No  || {{Done}}
|

|-
| qemu-x86-boot				||  3 || AL  || No  || {{Done}}
| [[PCR:14575]]


|-
| registry-cvt-win32			||  5 ||     || Yes || {{Done}}
|

|-
| score-other-docs			||  5 ||     || No  || Pending
|

|-
| socket-examples			||  4 || AL  || No  || {{Work2}}
| Not released.

|-
| test-infrastructure			||  8 || AL  || No  || {{done}}
|

|-
| trace32-extension			||  5 ||     || No  || {{Done}}
|

|-
| traceaid				||  8 ||     || No  || {{Done}}
|

|-
| vfileconfig				||  3 ||  GK  || Yes  || {{Done}}
| unreleased version 1.8.0

|-
| x9-client				||  8 || AL  || No  || {{Work2}}
| [[PCR:14568]]  Client code ported and unreleased, server code still py2.<br>
DeprecationWarning: 'telnetlib' is deprecated and slated for removal in Python 3.13

|-
| xml-tools-common			||  8 || GK  || Yes || {{Done}}
| [[PCR:13965]]

|-
| build-utils				||  8 || AL  || No  || {{Done}}
|

|-
| test-utils				||  8 || AL  || No  || {{Work2}}
| [[PCR:14576]] Will investigate pylint warnings and cleanup.
|-
| deos2cygwin				||  9 || AL  || No  || {{Done}}
|
|-
| review-process-scripts		||  9 || AL  || No  || {{Done}}
|

|}

= Tracking Progress/Status =

{| class="wikitable sortable mw-collapsible" border="1" style="text-align: right"
|- align="center" style="background: silver"
! Date     !! Total /desk python files !! Ported to Python3
|-
| 2021-10-05 || 129  || 56

|-
| 2021-10-29 || 129 || 74

|-
| 2022-04-25 || 161 || 89

|-
| 2023-06-12 || 218 || 132

|-
| 2023-07-11 || 224 || 147

|-
| 2023-09-27 || 166 || 155

|-
| 2024-05-14 || 172 || 163

|-
| 2024-10-16 || 212 || 198

|}

To get the above numbers paste the following into a desk console:
<pre>pyFiles=$(for f in $(find /desk -type f | grep -v __pycache__); do if file $f | grep -i ':.*Python' > /dev/null; then echo $f; fi; done)
printf "|-\n| %s || %s || %s\n" $(date '+%Y-%m-%d')  \
    $(ls $pyFiles | wc -l) \
    $(for f in $pyFiles; do echo $f: $(head -1 $f); done | grep 'python3' | wc -l)
</pre>

To get the list of packages that have python2 files, and how many files in each package:

<pre>pkgPyFiles=$(for f in $(for f in $pyFiles; do echo $f: $(head -1 $f); done | grep -v 'python3' | cut -d: -f1); do dpkg -S $f | sed -e 's/ /:/' ; done | sort -u)
pkgs=$(echo "$pkgPyFiles" | cut -d: -f1 | sort -u)
for p in $pkgs; do printf "%3d %s\n" $(echo "$pkgPyFiles" | grep "$p:" | wc -l) $p; done | sort --key=2d | sort --key=1rn
</pre>

Results on Aaron's maintainer docker image:

<table>
<tr><th>2023-07-11</th>  <th>2023-09-27</th> <th>2024-05-14</th> <th>2024-10-16</th> </tr>
<tr>
<td>
<pre>
 25 ist-config-win32
 23 ioi-config-tool
 14 ioi-cvt
  9 desk-python-tools
  1 deos653-config-win32
  1 ist-arinc653-examples
  1 ist-ioi-examples
  1 qemu-x86-64-boot
  1 qemu-x86-boot
  1 socket-examples
 77 Total
</pre>
</td>

<td  style="vertical-align:top"><pre>
  8 desk-python-tools
  1 ist-arinc653-examples
  1 ist-ioi-examples
  1 socket-examples
</pre></td>

<td  style="vertical-align:top"><pre>
  8 desk-python-tools
  1 socket-examples
</pre></td>

<td  style="vertical-align:top"><pre>
 14 ioi-cvt
</pre></td>

</tr>
</table>

= Task Raw Data =

<!--
The following raw data provides an incomplete list of components that have .py scripts (generated by these commands from Aaron's "current install"):

  for f in $(find /desk -iname '*.py'); do  dpkg -S $f; done | sort | awk '{print $1; }' | sort -u | sed -e 's/://'
  for f in $(find /desk -iname '*.py'); do  dpkg -S $f; done | sort

The command output is below:
-->

The following is a list of all the packages from a cygwin setup.ini file circa 2021-10-01:

Non-boot components DDCI responsible for:
<pre class="mw-collapsible mw-collapsed" style="width:50%">
abc-tool
afdx-driver-cvt
afdx-library-cvt
cffsdump
codetrace
crittime-tool
deos653-config-win32
deos653-cvt
deosbook-devel
desk-python-tools
heartos-python-tools
heartos-test-infrastructure
hyperstart
integ-tool-command-line
ioi-config-tool
ioi-cvt
ist-arinc653-examples
ist-config-win32
ist-ioi-examples
kernel
kfs-cvt
openarborEclipse_32
openarborEclipse_64
openarborMaintainerTools
pci-config
registry-cvt-win32
score-other-docs
socket-examples
test-infrastructure
trace32-extension
traceaid
vfileconfig
x9-client
</pre>

BSP Boot components:
<pre class="mw-collapsible mw-collapsed" style="width:50%">
arrow-sockit-boot
bsp_mpc603e_pqii82xx
bsp_mpc603e_psim
bsp_mpc603e_rattler8280
celestial
chino-fcm
cm-crb-boot
dpspl017x-boot
dvmsconfig
ep440c-platform-boot
ep8280-ppc-platform-boot
ep8343m-boot
fit-pc1-boot
fit-pc2-boot
harrys-boot
hosmer-boot
hotdish-aid-boot
imb-a180-h-boot
intel-mc-boot
kalbi-boot
louiev2-boot
lutefisk-aid-ppc-platform-boot
lutefisk-iv-boot
maple-boot
minnow-turbot-dual-boot
minnow-turbot-quad-boot
mpc8315e-boot
mpc8349e-mitx-gp-platform-configuration
nai75arm1-boot
nitrogen6x-som-boot
p1010rdb-boot
p2041rdb-boot
pmc5675-platform-configuration
qemu-arm-boot
qemu-mips-boot
qemu-ppc-boot
qemu-x86-boot
rattler8280-ppc-platform-configuration
vmware-boot
vpx6187-boot
wade8078-boot
xcalibur1002-platform-configuration
zcu102
zu5
</pre>

Supplier provided components:
<pre class="mw-collapsible mw-collapsed" style="width:50%">
deos-rapita
deos-rtems-examples
deos-rtems-tools-arm
deos-rtems-tools-ppc
deos-rtems-tools-x86
gcc-aarch64-elf-libs
gcc-common
gcc-common-libs
gcc-ppc-elf-libs
gcc-ppc-elfspe-libs
gcc-x86-elf-libs
gcc-x86_64-elf-libs
gdb-cross-debuggers
htmlgen2
pyPdf

</pre>
<p> <br> </p>
[[Category:Projects]]
[[Category:Active_Projects]]
<!-- Uncomment one of these
[[Category:Completed_Projects]]
[[Category:On_Hold_Projects]]
[[Category:Active_Projects]]
-->
Pages included on this page:
Return to Python3 Port.
Retrieved from "https://ddci.zapto.org/ddciWiki/Python3_Port"
View source for Python3 Port

Navigation menu

Page actions

Page actions

Personal tools

Navigation

Search

DDCI Tools & Resources

DDC-I private

Shared IP

Other

Tools
