Common XenServer/XCP-ng Python classes

The xcp directory contains the Common XenServer and XCP-ng Python packages. They are for use in XenServer and XCP-ng Dom0 only and deal with logging, Hardware/PCI, networking, and other Dom0 tasks.

The package name is python-libs which is also the rpm package name in XenServer. XCP-ng packages it as xcp-python-libs (koji).

It supports Python 2.7 and is currently in progress to get further fixes for >= 3.6. It depends on six, and on Python 2.7, also configparser and pyliblzma.

Test-driven Development (TDD) Model

Please see [CONTRIBUTING.md] for installing a local development environment.

This package has CI which can be run locally but is also run in GitHub CI to ensure Test-driven development.

CI features:

• Combined coverage testing of Python 2.7 and Python 3.8 code branches
• Automatic Upload of the combined coverage to CodeCov (from the GitHub Workflow)
• Checking of the combined coverage against the diff to master: Fails if changes are not covered!
• Pylint report in the GitHub Action Summary page, with Warning and Error annotations, even in the code review.
• Check that changes don't generate pylint warnings (if warning classes which are enabled in .pylintrc)
• Static analysis using mypy, pylint, pyright and pytype

This enforces that any change (besides whitespace):

• Has code coverage,
• does not introduce a pylint warning which is not disabled in .pylintrc
• does not introduce a type of static analysis warning which is currently suppressed.

Status Summary

Pylint results from GitHub CI in GitHub Actions page

A step of the GitHub workflow produces a browser-friendly pylint report: From the Actions tab, open a recent workflow run the latest and scroll down until you see the tables!

Configuration files

• pyproject.toml: Top-level configuration of the package metadata and dependencies
• tox.ini: Secondary level configuration, defines of the CI executed by tox
• pytest.ini: The defaults used by pytest unless overruled by command line options
• .github/workflows/main.yml: Configuration of the GitHub CI matrix jobs and coverage upload
• .github/act-serial.yaml: Configuration for the jobs run by the local GitHub actions runner act
• .pylintrc: Configuration file of Pylint

Installation and Setup of a development environment

For the installation of the general development dependencies, visit INSTALL.md

Updating tests using pytest-watch (ptw)

• pip install pytest-watch - ptw watches changed files and runs pytest after changes are saved.
  • Then run ptw on the code/tests you work on, e.g.: ptw tests/test_pci_* and edit the files.

Example development workflow

• Run the tests for at also with LC_ALL=C python3.6 -m pytest to check for any ascii codec issues by Python3.6
• Test with python2.7 -m pytest
• Run mypy (without any arguments - The configuration is in pyproject.toml)
• Run ./pytype_runner.py
• Run tox -e py36-lint and fix any Pylint warnings
• Run tox -e py310-covcombine-check and fix any missing diff-coverage.
• Run tox for the full CI test suite
• Run act for the full CI test suite in local containers (similar to GitHub action containers)
• Commit with --signoff on a new branch and push it and check the triggered GitHub Action run succeeds.
• Open a new PR

The list of virtualenvs configured in tox can be shown using this command: tox -av

$ tox -av
default environments:
py36-lint              -> Run in a py36 virtualenv: Run pylint and fail on warnings remaining on lines in the diff to master
py311-pyright          -> Run in a py311 virtualenv: Run pyright for static analysis
py38-pytype            -> Run in a py38 virtualenv: Run pytype for static analysis, intro: https://youtu.be/abvW0mOrDiY
py310-covcombine-check -> Run in a py310 virtualenv: Generate combined coverage reports with py27-test coverage merged Run mypy for static analysis

additional environments:
cov                    -> Run in a python virtualenv: Generate coverage html reports for this environment
covcp                  -> Run in a python virtualenv: Copy the generated .coverage and coverage.xml to the UPLOAD_DIR dir
fox                    -> Run in a python virtualenv: Generate combined coverage html reports and open them in firefox
mdreport               -> Run in a python virtualenv: Make a test report (which is shown in the GitHub Actions Summary Page)
test                   -> Run in a python virtualenv: Run pytest in this environment with --cov for use in other stages

If you have only one version of Python3, that works too. Use: tox -e py<ver>-test

Static analysis using mypy, pylint, pyright and pytype

The preconditions for using static analysis with mypy (which passes now but has only a few type comments) and pyright are present now and mypy is enabled in tox which runs the tests in GitHub CI as well. But of course, because they code is largely still not yet typed, no strict checks can be enabled so far. However, every checker which is possible now, is enabled.

Checking the contents of not typed functions is enabled for all but four modules which would need more work. Look for check_untyped_defs = false in pyproject.toml.

The goal or final benefit would be to have it to ensure internal type correctness and code quality but also to use static analysis to check the interoperability with the calling code.

Type annotations: Use Type comments for now

Python2.7 can't support the type annotation syntax, but until all users are migrated, annotations in comments (type comments) can be used. They are supported by tools like mypy and pyright (VS Code):

Quoting from https://stackoverflow.com/questions/53306458/python-3-type-hints-in-python-2:

Function annotations were introduced in PEP 3107 for Python 3.0. The usage of annotations as type hints was formalized in PEP 484 for Python 3.5+.

Python < 3.0 does support the type hints syntax, but PEP 484 introduces type comments that are equally supported and are otherwise ignored. These type comments look like this:

def get_default_device(use_gpu=True):
    # type: (bool) -> cl.Device
    ...

Many type checkers support this syntax: mypy, pyright, pytype

As proof, these examples show how the comment below triggers the checks:

--- a/xcp/xmlunwrap.py
+++ b/xcp/xmlunwrap.py
@@ -29,1 +29,2 @@ class XmlUnwrapError(Exception):
 def getText(nodelist):
+    # type:(Element) -> str

mypy:

$ mypy xcp/xmlunwrap.py
xcp/xmlunwrap.py:31: error: Name "Element" is not defined
xcp/xmlunwrap.py:38: error: Incompatible return value type (got "bytes", expected "str")

pyright (used by VS Code by default):

$ pyright xcp/xmlunwrap.py|sed "s|$PWD/||"
...
pyright 1.1.295
xcp/xmlunwrap.py
  xcp/xmlunwrap.py:32:13 - error: "Element" is not defined (reportUndefinedVariable)
  xcp/xmlunwrap.py:38:12 - error: Expression of type "Unknown | bytes" cannot be assigned to return type "str"
    Type "Unknown | bytes" cannot be assigned to type "str"
      "bytes" is incompatible with "str" (reportGeneralTypeIssues)
  xcp/xmlunwrap.py:81:38 - error: Argument of type "Unknown | None" cannot be assigned to parameter "default" of type "str" in function "getStrAttribute"
    Type "Unknown | None" cannot be assigned to type "str"
      Type "None" cannot be assigned to type "str" (reportGeneralTypeIssues)
3 errors, 0 warnings, 0 informations
Completed in 0.604sec

See #23 for the context of this example.

Guidelines

Character set encoding/string handling: See README-Unicode.md for details on Unicode support.

Users

• host-installer
• host-upgrade-plugin (koji):
  • /etc/xapi.d/plugins/prepare_host_upgrade.py
• xapi (xapi-core.rpm and xenopsd.rpm)
  • /etc/xapi.d/extensions/pool_update.apply
  • /etc/xapi.d/extensions/pool_update.precheck
  • /etc/xapi.d/plugins/disk-space
  • /etc/xapi.d/plugins/install-supp-pack
  • /opt/xensource/libexec/host-display
  • /opt/xensource/libexec/mail-alarm
  • /opt/xensource/libexec/usb_reset.py
  • /opt/xensource/libexec/usb_scan.py
  • /usr/libexec/xenopsd/igmp_query_injector.py
• xenserver-release-config/xcp-ng-release-config
  • /opt/xensource/libexec/fcoe_driver
  • /opt/xensource/libexec/xen-cmdline
• https://github.com/xcp-ng-rpms/interface-rename
  • /etc/sysconfig/network-scripts/interface-rename.py
  • /opt/xensource/bin/interface-rename
• pvsproxy (Proprietary)
  • /usr/libexec/xapi-storage-script/volume/org.xen.xapi.storage.tmpfs/memoryhelper.py
• https://github.com/xenserver/linux-guest-loader (not installed by default anymore)
  • /opt/xensource/libexec/eliloader.py
• https://github.com/xcp-ng-rpms/vcputune
  • /opt/xensource/bin/host-cpu-tune
• The ACK xapi plugin. See: #21

Verification:

# rpm -qf $(grep -r import /usr/libexec/ /usr/bin /etc/xapi.d/ /opt/xensource/|grep xcp|cut -d: -f1|grep -v Binary) --qf '%{name}\n'|sort -u|tee xcp-python-libs-importers.txt
host-upgrade-plugin
interface-rename
pvsproxy
vcputune
xapi-core
xenopsd
xenserver-release-config
# grep -s import $(rpm -ql xapi-core)|grep xcp|cut -d: -f1
/etc/xapi.d/extensions/pool_update.apply
/etc/xapi.d/extensions/pool_update.precheck
/etc/xapi.d/plugins/disk-space
/etc/xapi.d/plugins/disk-space
/etc/xapi.d/plugins/install-supp-pack
/opt/xensource/libexec/host-display
/opt/xensource/libexec/mail-alarm
/opt/xensource/libexec/usb_reset.py
/opt/xensource/libexec/usb_scan.py