Upgrade setuptools, setuptools_scm, pip (2020-06), add package wheel; remove zope_interface

[mkoeppe]

Big upgrade steps, enabled by drop python 2 support.

• ​https://pypi.org/project/pip/ 20.1.1
• ​https://pypi.org/project/setuptools/ 47.1.1
• ​https://pypi.org/project/setuptools-scm/ 4.1.2

Add new packages to satisfy dependencies:

• ​https://pypi.org/project/wheel/ 0.34.2

These are the latest versions as of 2020-06-05; all claim to support Python >= 3.5 (note #29033)

We adjust spkg-install of several packages by removing pyproject.toml to remove the need for more extensive build infrastructure. We disable the build of zope_interface (no longer needed, see #29754).

Tarballs: See checksums.ini [upstream_url]. (To configure Sage to download from the upstream URL, use ./configure --enable-download-from-upstream-url.)

Previous updates: #26969, #27886, #26008

[mkoeppe]

New commits:

​886693e	Upgrade pip to 20.1.1
​80374c5	Upgrade setuptools to 47.1.1
​50c6962	Upgrade setuptools_scm to 4.1.2

[mkoeppe]

pip installation fails, for example on ubuntu-bionic-standard (​https://github.com/mkoeppe/sage/runs/741110905)

     Preparing wheel metadata: finished with status 'error'
ERROR: Command errored out with exit status 1: /sage/local/bin/python3 /sage/local/var/tmp/sage/build/pip-20.1.1/src/src/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmp32gsdf31 Check the logs for full command output.

[mkoeppe]

Seems we need to add the wheel package

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​7c81b8c	build/pkgs/wheel: New
​70378ed	build/pkgs/pip/dependencies: Add wheel, simplify $(PYTHON) python3 to $(PYTHON)
​10a2bac	build/pkgs/setuptools, build/pkgs/pip: Remove python2 support from dependencies, install scripts

[mkoeppe]

Tests run at ​https://github.com/mkoeppe/sage/pull/36/checks

[mkoeppe]

On debian-bullseye-standard:

  [pkgconfig-1.5.1]     File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
  [pkgconfig-1.5.1]       return _bootstrap._gcd_import(name[level:], package, level)
  [pkgconfig-1.5.1]     File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  [pkgconfig-1.5.1]     File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  [pkgconfig-1.5.1]     File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
  [pkgconfig-1.5.1]     File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  [pkgconfig-1.5.1]     File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  [pkgconfig-1.5.1]     File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  [pkgconfig-1.5.1]     File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
  [pkgconfig-1.5.1]     File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  [pkgconfig-1.5.1]     File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  [pkgconfig-1.5.1]     File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  [pkgconfig-1.5.1]     File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
  [pkgconfig-1.5.1]   ModuleNotFoundError: No module named 'poetry'

[mkoeppe]

pkgconfig is at the most current version. Looks like giving it a modern setuptools causes it to demand more.

[mkoeppe]

poetry, of course, has a LONG list of additional dependencies... ​https://github.com/python-poetry/poetry/blob/master/pyproject.toml

[mkoeppe]

We will postpone adding poetry to #29810 and instead patch pkgconfig so it installs more prosaically using setuptools.

[mkoeppe]

​https://github.com/matze/pkgconfig/commits/master

[mkoeppe]

See #28883 for the last pkgconfig update, which commented on the issues with poetry already.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6755adb

build/pkgs/pkgconfig/spkg-install.in: Remove pyproject.toml so that pip installs using setup.py

[mkoeppe]

Tests run again at ​​https://github.com/mkoeppe/sage/pull/36/checks

[mkoeppe]

Several failures (for example in ​https://github.com/mkoeppe/sage/runs/744000563):

  [entrypoints-0.3]   ModuleNotFoundError: No module named 'flit'
  [testpath-0.4.2]   ModuleNotFoundError: No module named 'flit'
  [zope_interface-4.6.0]   ImportError: cannot import name 'Feature' from 'setuptools' (/sage/local/lib/python3.7/site-packages/setuptools/__init__.py)
  [terminado-0.8.1]   ModuleNotFoundError: No module named 'flit'

[mkoeppe]

Possible that zope_interface can be removed (#29754)

[mkoeppe]

entrypoints is current, ​https://pypi.org/project/testpath/ can be upgraded to 0.4.4, ​https://pypi.org/project/terminado/ to 0.8.3

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​29327d3	build/pkgs/entrypoints: Do not provide setup.py, not needed for entrypoints 0.3
​bcc79a5	build/pkgs/entrypoints/spkg-install.in: Remove pyproject.toml so that setup.py is used
​361b98b	build/pkgs/terminado/spkg-install.in: Remove pyproject.toml so that setup.py is used
​5040d7e	build/pkgs/testpath/spkg-install.in: Remove pyproject.toml so that setup.py is used

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​371ede6

build/pkgs/zope_interface: Change to a dummy script package

[mkoeppe]

Builds fine on homebrew-macos-standard. Ready for review.

[dimpase]

lgtm

[mkoeppe]

Thanks!

[jhpalmieri]

Not a big deal, but I'm curious: with pip, why only the change to set versions=3, instead of removing the loop for vers in $versions ... and replacing $vers with 3 in spkg-install.in?

[mkoeppe]

I like to keep diffs minimal. Also minor uncertainty what the python binary will be called when python 4.0 comes out...

[jhpalmieri]

I think that this ticket is leading to the following messages in the sagetex install log:

Successfully built sagetex
Installing collected packages: sagetex
  Created temporary directory: /private/var/folders/cp/n8wtqs490tq5psknff1hv9qr0000gn/T/pip-unpacked-wheel-hw7nz29q
  *** Error compiling '/private/var/folders/cp/n8wtqs490tq5psknff1hv9qr0000gn/T/pip-unpacked-wheel-hw7nz29q/sagetex-3.4.data/data/share/texmf/tex/latex/sagetex/extractsagecode.py'...
    File "/private/var/folders/cp/n8wtqs490tq5psknff1hv9qr0000gn/T/pip-unpacked-wheel-hw7nz29q/sagetex-3.4.data/data/share/texmf/tex/latex/sagetex/extractsagecode.py", line 48
      except getopt.GetoptError, err:
                               ^
  SyntaxError: invalid syntax
  
  *** Error compiling '/private/var/folders/cp/n8wtqs490tq5psknff1hv9qr0000gn/T/pip-unpacked-wheel-hw7nz29q/sagetex-3.4.data/data/share/texmf/tex/latex/sagetex/makestatic.py'...
    File "/private/var/folders/cp/n8wtqs490tq5psknff1hv9qr0000gn/T/pip-unpacked-wheel-hw7nz29q/sagetex-3.4.data/data/share/texmf/tex/latex/sagetex/makestatic.py", line 48
      except getopt.GetoptError, err:
                               ^
  SyntaxError: invalid syntax
  

Successfully installed sagetex-3.4
Removed build tracker: '/private/var/folders/cp/n8wtqs490tq5psknff1hv9qr0000gn/T/pip-req-tracker-3d0gjhof'
Removing old SageTex version(s)

real	0m3.793s
user	0m1.162s
sys	0m0.458s
Copying package files from temporary location /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.2.beta3/local/var/tmp/sage/build/sagetex-3.4/inst to /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.2.beta3/local
Successfully installed sagetex-3.4
Deleting temporary build directory
/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.2.beta3/local/var/tmp/sage/build/sagetex-3.4
Finished installing sagetex-3.4.spkg

Is it the new wheel package?

(This is on OS X using the system Python, version 3.7.7.)

[mkoeppe]

pip likes to install via bdist_wheel now. Something is getting compiled that was not compiled before.

I think it reveals that the sagetex code is not compatible with python 3.

[dimpase]

I am going to make a new release of sagetex soon.

[jhpalmieri]

Replying to dimpase:

I am going to make a new release of sagetex soon.

Good, that should take care of it. I'm surprised that sagetex still works, given these syntax errors.

[dimpase]

probably it's from some file that we normally didn't install, or perhaps it was using some compatibility kludge that is now gone.