packages/lang/python/python3-host.mk, branch openwrt-22.03

python: Add proper support for pyproject.toml-based builds

2023-10-31T09:03:43Z

This removes the changes made in 61f202c0170785addbbc449e4de61cc5886f0833 and adds actual support for pyproject.toml-based (PEP 517) builds of Python packages. Packages can force the use of the old build process by setting PYTHON3_PKG_FORCE_DISTUTILS_SETUP:=1; this should only be a temporary workaround until the package can be updated/fixed to use the new build process. Signed-off-by: Jeffery To (cherry picked from commit 5156c0c82bcb18481bf588ba4b1ff67c12422a80)

python: Better host pip options

2023-10-13T06:23:39Z

pip by default will read system-wide and per-user configuration files[1]. Setting PIP_CONFIG_FILE=/dev/null instructs pip to not read any config files[2]. pip will spawn child processes of itself to do work, but not all options are passed down to the child processes[3]. Setting global options as environment variables[4] ensures they are passed down to any child processes. [1]: https://pip.pypa.io/en/stable/topics/configuration/#configuration-files [2]: https://pip.pypa.io/en/stable/topics/configuration/#pip-config-file [3]: https://github.com/pypa/pip/issues/9081#issue-733819665 [4]: https://pip.pypa.io/en/stable/topics/configuration/#environment-variables Signed-off-by: Jeffery To (cherry picked from commit 8c2abb74031e2403dde83536a8e7f13c63cdd4ab)

python: Add pyproject.toml-based builds for host Python packages

2023-10-13T06:23:16Z

Using pip to install host packages with pyproject.toml-based (PEP 517) builds is problematic: * If build isolation is used, pip will create an isolated build environment, install any build dependencies for the requested package, then build the requested package. It does not appear currently possible to have pip install the build dependencies with hash-checking mode enabled[1]. * If build isolation is not used, any build dependencies must be installed in the build environment before invoking pip to build the requested package[2]. This would require creating a package dependency resolution system to install build dependencies, and any dependencies of dependencies, in the correct order. * It is very difficult to patch the packages installed by pip. This adds a new include file (python3-host-build.mk) with recipes to install host Python packages with pyproject.toml-based builds. This is backwards-compatible with packages that require running setup.py. Besides addressing the above issues (the OpenWrt build system already resolves dependencies between packages, checks all source downloads against known hashes, and supports patching packages), host packages also: * Capture package licensing and maintainer information * Enable uscan checking for package updates/CVEs * Are a known concept for OpenWrt packagers/developers The existing functionality of using host pip to install packages will remain for now, but should be considered deprecated and expected to be removed in the future. This also updates Py3Build/CheckHostPipVersionMatch for the case where the host-pip-requirements directory does not exist or is empty. [1]: https://pip.pypa.io/en/stable/user_guide/#changes-to-the-pip-dependency-resolver-in-20-3-2020 [2]: https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-no-build-isolation Signed-off-by: Jeffery To (cherry picked from commit fe78c07a316b1722b8d35b63772a7067dfd87c5e)

python-build: add support for pyproject.toml files

2023-10-13T06:23:08Z

A new PEP 517 (https://www.python.org/dev/peps/pep-0517/) has defined that Python packages can be shipped without any `setup.py` file, and that a `pyproject.toml` file is sufficient. A `setup.py` shim layer is suggested as a method for running the build. For these cases, we will add a support in the OpenWrt build-system to provide the default `setup.py` shim layer in case this file does not exist, but there is a `pyproject.toml` file. We also seem to need to tweak the shim layer with the PKG_VERSION, otherwise the detected version is 0.0.0. We will need to see if this will be fixed later in setuptools{-scm}. Signed-off-by: Alexandru Ardelean (cherry picked from commit 61f202c0170785addbbc449e4de61cc5886f0833)

python: Unset Python environment variables

2023-10-13T06:21:12Z

This will prevent the user's environment variables from affecting host Python, removing the need to manually override these variables. It is also not necessary to set PYTHONPATH (when not working on target Python packages) because the given directories are already included in Python's search path by default. Signed-off-by: Jeffery To (cherry picked from commit 6ef46bb9194b5d3f5fc17471d8869bcae729d215)

python: Use locked for host pip

2020-08-31T10:48:47Z

This also removes PKG_BUILD_PARALLEL:=0 that was added for packages that use HOST_PYTHON3_PACKAGE_BUILD_DEPENDS. Signed-off-by: Jeffery To

python3: Use hash-checking mode when installing host pip packages

2020-08-17T15:05:49Z

In hash-checking mode[1], pip will verify downloaded package archives (source tarballs in our case) against known SHA256 hashes before installing the packages. As a consequence, this requires the use of requirements files[2] and pinning packages to known versions. The syntax for package Makefiles has changed slightly; HOST_PYTHON3_PACKAGE_BUILD_DEPENDS no longer accepts requirement specifiers like "foo>=1.0", only requirements file names (which are the same as package names in the most common case). This also updates affected packages, in particular: * python-zipp: "setuptools_scm[toml]" has been split into "setuptools-scm toml" to reuse the requirements file for setuptools-scm (the extra depends installed by "setuptools_scm[toml]" is toml). * python-pycparser: This previously used ply 3.10, whereas the requirements file will now install 3.11. [1]: https://pip.pypa.io/en/stable/reference/pip_install/#hash-checking-mode [2]: https://pip.pypa.io/en/stable/user_guide/#requirements-files Signed-off-by: Jeffery To

python3: Add config PYTHON3_HOST_PIP_CACHE_WORLD_READABLE

2020-08-15T22:09:45Z

This adds a config option PYTHON3_HOST_PIP_CACHE_WORLD_READABLE; if enabled, chmod will be run after pip install to make all files/directories in the host pip cache world-readable. Supersedes https://github.com/openwrt/packages/pull/13012. Signed-off-by: Jeffery To

python3: fix host compilation with clang

2020-08-02T07:49:30Z

Matched rpath parameter with Makefile. Signed-off-by: Rosen Penev

python3: Use default _PYTHON_HOST_PLATFORM

2020-07-08T09:11:42Z

This lets the Python build process set _PYTHON_HOST_PLATFORM instead of forcing an explicit value. Also: * Save the target _PYTHON_HOST_PLATFORM value during Build/InstallDev for use when building target Python packages (in python3-package.mk). * Use the (mostly) default PYTHON_FOR_BUILD value, instead patch configure to remove the platform triplet from the sysconfigdata file name. * Remove the "CROSS_COMPILE=yes" make variable (there is no indication that this variable is necessary). * Force host pip to build packages from source instead of downloading binary wheels. Previously, host pip can download universal (platform-independent) wheels but not platform-specific wheels, because of the custom _PYTHON_HOST_PLATFORM value. (Packages that do not have universal wheels would be compiled from source.) With a correct _PYTHON_HOST_PLATFORM, host pip can install platform-specific wheels as well. However, the pre-built shared object (.so) files in these wheels will have the host's platform triplet in their file names. When target Python packages are built (using the target's _PYTHON_HOST_PLATFORM), Python will not use these shared object files. By forcing host pip to build packages from source, the built shared object files will not have the platform triplet in their file names. (Host Python has been patched to remove the platform triplet from file names.) This allows these packages to be used when building target Python packages. (The net effect of this complete change is that platform-dependent packages will continue to be compiled from source, while platform-independent packages will now also be compiled from source.) Signed-off-by: Jeffery To