Upgrade to Python 3.7.x

[Jeroen Demeyer]

Tarball: ​https://www.python.org/ftp/python/3.7.3/Python-3.7.3.tar.xz

[Jeroen Demeyer]

New commits:

​2c67605

Upgrade to Python 3.7.0

[Erik Bray]

I don't think we should be in any hurry on this. All the last ~year's work of Python 3 porting has been targeting Python 3.6, and I think I'd rather stick with that before risking throwing everything out of whack again.

Between Python 3.6 and 3.7 I suspect the differences, if any, will be minor. But I'd rather try to stabilize on 3.6 first (to which we're quite close), then deal with those differences, rather than change Python versions again. Also I don't know that Python 3.7 has been ported to Cygwin yet (although it does contain a number of useful fixes for Cygwin, I do know).

That said, if you know any specific changes in Python 3.7 (other than the aforementioned Cygwin fixes) that would actively make our porting effort easier I'd consider it...

[Frédéric Chapoton]

Jeroen has said elsewhere that islice will accept Sage integers instead of just ints.

I am again becoming tired and frustated by the slow pace of progress towards py3. And by the large amount of difficult work that remains..

[Erik Bray]

Ah, I remember making an issue about that, but I don't think I followed what the resolution was.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​1bfcdc8

Upgrade to Python 3.7.0

[Erik Bray]

Well there's this ​https://github.com/python/cpython/pull/1918 But unless there are other fixes I'm not seeing it's annoying that this was only fixed for islice, where I believe there is other code affected by the same issue (e.g. range). Also we've already worked aroudn the majority of those cases (sigh...)

[Erik Bray]

Replying to chapoton:

I am again becoming tired and frustated by the slow pace of progress towards py3. And by the large amount of difficult work that remains..

Unfortunately I'm not convinced that upgrading Python yet again is going to make that go any faster. In any case I don't actually have that much "difficult" work remaining. Just dozens of little issues. My Python 3 branch has < 600 modules failing, and most of it appears to minor issues (at one point I had it down to ~400 and I don't know why it ballooned up again, but one or two small changes in just the right places can do that with Sage...)

Also I'll add if you're not using a patched pynac you're going to have lots, lots more "serious" looking issues. We still need to get a pynac upgrade in (I've been manually installing pynac with my Python 3 fixes every time I re-build sage).

[Jeroen Demeyer]

Replying to embray:

That said, if you know any specific changes in Python 3.7 (other than the aforementioned Cygwin fixes) that would actively make our porting effort easier I'd consider it...

#25391 is a pretty serious issue which is fixed by a Python upgrade (probably also by upgrading to 3.6.6 but I haven't tested that).

[Jeroen Demeyer]

Replying to embray:

I don't think we should be in any hurry on this. All the last ~year's work of Python 3 porting has been targeting Python 3.6, and I think I'd rather stick with that before risking throwing everything out of whack again.

First of all, I do not think that this 3.6 -> 3.7 will throw everything out of whack.

However, regardless of that, we'll have to upgrade sooner or later. I don't see the point of having a perfectly working Sage on an outdated Python version. And for the many issues that we still have to fix, it would be better to make sure that they work on Python 3.7 from the start.

[Frédéric Chapoton]

I still think we have very hard remaining problems, among which

• sorting issues in graphs
• hashing and comparisons, especially for groups
• coercion framework have broken parts

[vklein]

I get the following error

$ ./configure --with-python=3
...
$ make build
...
[python3-3.7.0] Testing importing of various modules...
[python3-3.7.0] Traceback (most recent call last):
[python3-3.7.0]   File "<string>", line 1, in <module>
[python3-3.7.0]   File "/home/vklein/odk/sage/local/var/tmp/sage/build/python3-3.7.0/src/Lib/ctypes/__init__.py", line 7, in <module>
[python3-3.7.0]     from _ctypes import Union, Structure, Array
[python3-3.7.0] ModuleNotFoundError: No module named '_ctypes'
[python3-3.7.0] ctypes module failed to import
[python3-3.7.0] math module imported OK
[python3-3.7.0] hashlib module imported OK
[python3-3.7.0] crypt module imported OK
[python3-3.7.0] readline module imported OK
[python3-3.7.0] socket module imported OK
[python3-3.7.0] Error: One or more modules failed to import.
...

full python3.7 install log in attachement

[François Bissey]

Failure to build _ctypes usually means a problem with ffi (sometimes called libffi) may be there is minimum version requirement?

[vklein]

Indeed i don't get the error after installing libffi-dev. Thanks !

Note : sage make build works fine in python 3.6 without that.

[vklein]

next error during the build is :

Installing collected packages: ipykernel
  Running setup.py install for ipykernel: started
    Running command /home/vklein/odk/sage/local/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-qoxsao1r-build/setup.py';f=getattr(tokenize
, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" --no-user-cfg install --record /tmp/pip-vc23lh
pc-record/install-record.txt --single-version-externally-managed --compile
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-qoxsao1r-build/setup.py", line 91, in <module>
        from ipykernel.kernelspec import write_kernel_spec, make_ipkernel_cmd, KERNEL_NAME
...
      File "/home/vklein/odk/sage/local/lib/python3.7/site-packages/pexpect/spawnbase.py", line 224
        def expect(self, pattern, timeout=-1, searchwindowsize=-1, async=False):
                                                                       ^
    SyntaxError: invalid syntax
    Running setup.py install for ipykernel: finished with status 'error'
Cleaning up...

Issue reference : ​https://github.com/pypa/pipenv/issues/956

[François Bissey]

To be honest we haven't updated pexpect in vanilla sage for a while. It may be a good idea to do that - in a separate ticket.

[vklein]

Do we need to update pexpect or to patch pipenv ?

[Jeroen Demeyer]

No, pexpect.

[François Bissey]

I am at pexpect-4.2.1 in sage-on-gentoo (without any patches) and sage is at 4.1.0 + some patches.

[Jeroen Demeyer]

Replying to fbissey:

I am at pexpect-4.2.1 in sage-on-gentoo (without any patches) and sage is at 4.1.0 + some patches.

The patches in Sage are for performance. Without those patches, pexpect is really a lot slower.

[vklein]

Let's do that #25700.

[Erik Bray]

Replying to chapoton:

I still think we have very hard remaining problems, among which

• sorting issues in graphs
• hashing and comparisons, especially for groups
• coercion framework have broken parts

Most of which Python 3.7 won't help with. The one area where it might is sorting of dicts, since dict insertion order is now preserved (hooray!)

Well, I'm not completely opposed to upgrading, just skeptical, and don't want to rush into it. When I get a chance I'll compare how my python3 branch fares with and without it.

[vklein]

Replying to embray:

Well, I'm not completely opposed to upgrading, just skeptical, and don't want to rush into it. When I get a chance I'll compare how my python3 branch fares with and without it.

One of my goal reviewing this ticket is to produce a kind of diff between the doctests failures in 3.6 and 3.7. It can be helpful to decide what to do.

[vklein]

And now it's numpy's turn :

$ ./configure --with-python=3
...
$ make build
...
/home/vklein/odk/sage/local/lib/python3.7/distutils/dist.py:274: UserWarning: Un
known distribution option: 'define_macros'
  warnings.warn(msg)
numpy/random/mtrand/mtrand.c: In function '__Pyx__ExceptionSave':
numpy/random/mtrand/mtrand.c:45208:19: error: 'PyThreadState {aka struct _ts}' h
as no member named 'exc_type'
     *type = tstate->exc_type;
                   ^
numpy/random/mtrand/mtrand.c:45209:20: error: 'PyThreadState {aka struct _ts}' h
as no member named 'exc_value'
     *value = tstate->exc_value;
...

numpy's issue ​#10500. I am not sure what is the right call here i will try with numpy 1.14.5

[vklein]

When updating to numpy 1.14.5 with this following patch :

diff --git a/numpy/distutils/system_info.py b/numpy/distutils/system_info.py
index 93a8e6f..6e31d30 100644
--- a/numpy/distutils/system_info.py
+++ b/numpy/distutils/system_info.py
@@ -1740,7 +1740,7 @@ class blas_info(system_info):
             lib = self.has_cblas(info)
             if lib is not None:
                 info['language'] = 'c'
-                info['libraries'] = [lib]
+                info['libraries'] = lib
                 info['define_macros'] = [('HAVE_CBLAS', None)]
         self.set_info(**info)
 
@@ -1772,16 +1772,16 @@ class blas_info(system_info):
                 # check for cblas lib, and if not present check for blas lib.
                 try:
                     c.link_executable(obj, os.path.join(tmpdir, "a.out"),
-                                      libraries=["cblas"],
+                                      libraries=["libraries"],
                                       library_dirs=info['library_dirs'],
                                       extra_postargs=info.get('extra_link_args', []))
-                    res = "cblas"
+                    res = info["libraries"]
                 except distutils.ccompiler.LinkError:
                     c.link_executable(obj, os.path.join(tmpdir, "a.out"),
                                       libraries=["blas"],
                                       library_dirs=info['library_dirs'],
                                       extra_postargs=info.get('extra_link_args', []))
-                    res = "blas"
+                    res = ["blas"]
             except distutils.ccompiler.CompileError:
                 res = None
         finally:

I get the following error :

Found local metadata for numpy-1.14.5.p0
Using cached file /home/vklein/odk/sage/upstream/numpy-1.14.5.zip
numpy-1.14.5.p0
====================================================
Setting up build directory for numpy-1.14.5.p0
Finished extraction
Applying patches from ../patches...
Applying ../patches/numpy-1.14.5-no-hardcode-blas.patch
patching file numpy/distutils/system_info.py
****************************************************
Host system:
Linux tuono 4.13.0-45-generic #50~16.04.1-Ubuntu SMP Wed May 30 11:18:27 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
****************************************************
C compiler: gcc
C compiler version:
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/5/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 5.4.0-6ubuntu1~16.04.10' --with-bugurl=file:///usr/s
hare/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suff
ix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=po
six --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcx
x-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --en
able-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/
usr/lib/jvm/java-1.5.0-gcj-5-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-amd64 -
-with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share
/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --wit
h-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-g
nu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 
****************************************************

Note: if you need reliable uninstall behavior, then install
with pip instead of using `setup.py install`:

  - `pip install .`       (from a git repo or downloaded source
                           release)
  - `pip install numpy`   (last NumPy release on PyPi)


blas_opt_info:
blas_mkl_info:
Disabled blas_mkl_info: (MKLROOT is None)
Disabled blas_mkl_info: (MKLROOT is None)
customize UnixCCompiler
  libraries mkl_rt not found in []
  NOT AVAILABLE

blis_info:
customize UnixCCompiler
  libraries blis not found in ['/home/vklein/odk/sage/local/lib', '/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-li
nux-gnu']
  NOT AVAILABLE

openblas_info:
customize UnixCCompiler
Disabled openblas_info: (OPENBLAS is None)
customize UnixCCompiler
  libraries openblas not found in []
  NOT AVAILABLE

atlas_3_10_blas_threads_info:
Disabled atlas_3_10_blas_threads_info: (PTATLAS is None)
customize UnixCCompiler
  libraries tatlas not found in []
  NOT AVAILABLE

atlas_3_10_blas_info:
Disabled atlas_3_10_blas_info: (ATLAS is None)
customize UnixCCompiler
  libraries satlas not found in []
  NOT AVAILABLE

atlas_blas_threads_info:
Disabled atlas_blas_threads_info: (PTATLAS is None)
customize UnixCCompiler
  libraries ptf77blas,ptcblas,atlas not found in []
  NOT AVAILABLE

atlas_blas_info:
Disabled atlas_blas_info: (ATLAS is None)
customize UnixCCompiler
  libraries f77blas,cblas,atlas not found in []
  NOT AVAILABLE

blas_info:
customize UnixCCompiler
customize UnixCCompiler
C compiler: gcc -DNDEBUG -g -fwrapv -O3 -Wall -D__CEPHES_COMPLEX_H -fPIC

creating /tmp/tmpyq7p80or/tmp
creating /tmp/tmpyq7p80or/tmp/tmpyq7p80or
compile options: '-I/home/vklein/odk/sage/local/include -c'
gcc: /tmp/tmpyq7p80or/source.c
gcc /tmp/tmpyq7p80or/tmp/tmpyq7p80or/source.o -L/home/vklein/odk/sage/local/lib -llibraries -o /tmp/tmpyq7p80or/a.o
ut
/usr/bin/ld: cannot find -llibraries
collect2: error: ld returned 1 exit status
/usr/bin/ld: cannot find -llibraries
collect2: error: ld returned 1 exit status
gcc /tmp/tmpyq7p80or/tmp/tmpyq7p80or/source.o -L/home/vklein/odk/sage/local/lib -lblas -o /tmp/tmpyq7p80or/a.out
Running from numpy source directory.
/home/vklein/odk/sage/local/var/tmp/sage/build/numpy-1.14.5.p0/src/numpy/distutils/system_info.py:624: UserWarning:
 
    Atlas (http://math-atlas.sourceforge.net/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [atlas]) or by setting
    the ATLAS environment variable.
  self.calc_info()
/usr/bin/ld: cannot find -lblas
collect2: error: ld returned 1 exit status
/usr/bin/ld: cannot find -lblas
collect2: error: ld returned 1 exit status
Traceback (most recent call last):
  File "/home/vklein/odk/sage/local/lib/python3.7/distutils/unixccompiler.py", line 197, in link
    self.spawn(linker + ld_args)
  File "/home/vklein/odk/sage/local/var/tmp/sage/build/numpy-1.14.5.p0/src/numpy/distutils/ccompiler.py", line 89, 
in <lambda>
    m = lambda self, *args, **kw: func(self, *args, **kw)
  File "/home/vklein/odk/sage/local/var/tmp/sage/build/numpy-1.14.5.p0/src/numpy/distutils/ccompiler.py", line 152,
 in CCompiler_spawn
    raise DistutilsExecError('Command "%s" failed with exit status %d%s' % (cmd, s, msg))
distutils.errors.DistutilsExecError: Command "gcc /tmp/tmpyq7p80or/tmp/tmpyq7p80or/source.o -L/home/vklein/odk/sage
/local/lib -llibraries -o /tmp/tmpyq7p80or/a.out" failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/vklein/odk/sage/local/var/tmp/sage/build/numpy-1.14.5.p0/src/numpy/distutils/system_info.py", line 17
77, in has_cblas
    extra_postargs=info.get('extra_link_args', []))
  File "/home/vklein/odk/sage/local/lib/python3.7/distutils/ccompiler.py", line 734, in link_executable
    debug, extra_preargs, extra_postargs, None, target_lang)
  File "/home/vklein/odk/sage/local/lib/python3.7/distutils/unixccompiler.py", line 199, in link
    raise LinkError(msg)
distutils.errors.LinkError: Command "gcc /tmp/tmpyq7p80or/tmp/tmpyq7p80or/source.o -L/home/vklein/odk/sage/local/li
b -llibraries -o /tmp/tmpyq7p80or/a.out" failed with exit status 1

Advices or help are welcome. I get the same errors with a sage with python2.7.

[Jeroen Demeyer]

Replying to vklein:

numpy's issue ​#10500.

The solution seems to be recythonizing. Maybe we should just do that when building numpy.

[vklein]

Ok i will try.

[Jeroen Demeyer]

Replying to vklein:

Ok i will try.

If you look in the setup.py file of numpy, you'll see

    from setuptools import setup
    if run_build:
        from numpy.distutils.core import setup
        cwd = os.path.abspath(os.path.dirname(__file__))
        if not os.path.exists(os.path.join(cwd, 'PKG-INFO')):
            # Generate Cython sources, unless building from source release
            generate_cython()

So I guess that removing the PKG-INFO file in spkg-install will force recythonization.

[vklein]

Your trick almost work. The problem is that the numpy file ​tools/cythonize.py is not in numpy's source releases.

[vklein]

I confirm that numpy's install work if we remove PKG-INFO and add cythonize.py file.

[vklein]

Adding a patch to fix numpy's ​issue ​#10500.

---

New commits:

​6e0c5b4

Trac #25680: Patch numpy to fix issue with python3.7

[Jeroen Demeyer]

Can we please deal with numpy in a separate ticket.

[vklein]

I don't know. The patch doesn't make sense without python 3.7.

[Jeroen Demeyer]

Replying to vklein:

I don't know. The patch doesn't make sense without python 3.7.

It's needed for Python 3.7, but the fix is unrelated to it.

[Jeroen Demeyer]

If you're patching the sources anyway, I would prefer to patch setup.py instead of removing PKG-INFO.

[vklein]

Sure ! It will be cleaner to patch setup.py.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

[vklein]

scipy errors seems pretty similar to numpy's:

scipy/cluster/_vq.c: In function '__Pyx__ExceptionSave':
    scipy/cluster/_vq.c:9728:19: error: 'PyThreadState {aka struct _ts}' has no member named 'exc_type'
         *type = tstate->exc_type;
                       ^
    scipy/cluster/_vq.c:9729:20: error: 'PyThreadState {aka struct _ts}' has no member named 'exc_value'
         *value = tstate->exc_value;

With likely the same cause. In setup.py file of scipy :

        if not os.path.exists(os.path.join(cwd, 'PKG-INFO')):
            # Generate Cython sources, unless building from source release
            generate_cython()

[vklein]

Building sage with this ticket and it's dependencies works fine.

I have made two run of sage -t -a --long, one with a sage python3.6 and one with this ticket. Comparing the logs of the two runs (and weakly idenfying a failure by the tuple (<sage file with the failure>, <line of the failure>) with a little script, i get the following results:

nb doctests failures with python3.6: 42990
nb doctests failures with python3.7: 43643
new python3.7 failures : 1366
python3.6 failures not appearing with python 3.7 : 713

Two exemple failures among 1366 :

File "src/sage/combinat/finite_state_machine.py", line 10190, in sage.combinat.finite_state_machine.FiniteStateMachine.asymptotic_moments
Failed example:
    moments = T.asymptotic_moments()
Exception raised:
    Traceback (most recent call last):
      File "/home/vklein/odk/sage/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 573, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/vklein/odk/sage/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 983, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.combinat.finite_state_machine.FiniteStateMachine.asymptotic_moments[2]>", line 1, in <module>
        moments = T.asymptotic_moments()
      File "/home/vklein/odk/sage/local/lib/python3.7/site-packages/sage/combinat/finite_state_machine.py", line 10540, in asymptotic_moments
        if not final_component.digraph().is_aperiodic():
      File "/home/vklein/odk/sage/local/lib/python3.7/site-packages/sage/graphs/digraph.py", line 3330, in is_aperiodic
        import networkx
      File "/home/vklein/odk/sage/local/lib/python3.7/site-packages/networkx/__init__.py", line 114, in <module>
        import networkx.generators
      File "/home/vklein/odk/sage/local/lib/python3.7/site-packages/networkx/generators/__init__.py", line 13, in <module>
        from networkx.generators.geometric import *
      File "/home/vklein/odk/sage/local/lib/python3.7/site-packages/networkx/generators/geometric.py", line 27, in <module>
        from scipy.spatial import cKDTree as KDTree
      File "/home/vklein/odk/sage/local/lib/python3.7/site-packages/scipy/spatial/__init__.py", line 95, in <module>
        from .qhull import *
      File "qhull.pyx", line 2224, in init scipy.spatial.qhull
    AttributeError: 'cython_function_or_method' object has no attribute '__func__'

sage -t --long src/sage/combinat/dyck_word.py
**********************************************************************
File "src/sage/combinat/dyck_word.py", line 873, in sage.combinat.dyck_word.DyckWord.plot
Failed example:
    w.plot()
Expected:
    Graphics object consisting of 1 graphics primitive
Got:
    doctest:warning
      File "/home/vklein/odk/sage/src/bin/sage-runtests", line 127, in <module>
        err = DC.run()
      File "/home/vklein/odk/sage/local/lib/python3.7/site-packages/sage/doctest/control.py", line 1172, in run
        self.run_doctests()
      ...
      File "/home/vklein/odk/sage/local/lib/python3.7/collections/__init__.py", line 52, in __getattr__
        DeprecationWarning, stacklevel=2)
            File "/home/vklein/odk/sage/local/lib/python3.7/warnings.py", line 99, in _showwarnmsg
        msg.file, msg.line)
    :
    DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    Graphics object consisting of 1 graphics primitive

Note : To see the script and logs used ​https://github.com/vinklein/Trac25680

[Jeroen Demeyer]

Python 3.7 seems to require libffi as new dependency, so I guess we should add that as Sage package.

---

New commits:

​1c35bac	Upgrade pexpect
​7a7cb17	Trac #25755: Patch numpy to fix issue with ptyhon3.7
​6e291b1	Trac #25755: update package version
​48ceb84	Trac #25755: recythonized numpy in every case
​9c0cb67	Add cython as dependency of numpy
​7386ef8	Update scipy to 1.1.0
​b94a1bc	Remove deprecated iprint parameter
​0dcadcc	Remove scipy testsuite for now
​e074fa3	Merge commit '1c35bacd91a87db59d19ab78c2ecbfa17e38e240'; commit '9c0cb67a0d4a6f2b590143537c60d6e1d7cb84e8'; commit '0dcadcc4311e780e134c6912c3636c35e2d9d8b3' into t/25680/upgrade_to_python_3_7_0
​24349a4	Upgrade to Python 3.7.0

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​8f6d830	Add libffi package
​3b8e438	Merge commit '8f6d8304420e0894d5e8221238adcb09ab7edbed' into t/25680/upgrade_to_python_3_7_0
​2f7648b	Upgrade to Python 3.7.0

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​4e1b2be

Upgrade to Python 3.7.0

[Jeroen Demeyer]

Replying to fbissey:

Failure to build _ctypes usually means a problem with ffi (sometimes called libffi) may be there is minimum version requirement?

It seems that older Python versions vendored libffi but this was changed in Python 3.7. See ​https://docs.python.org/3/whatsnew/3.7.html#build-changes

[vklein]

The build works fine with the new dependencies.

[Erik Bray]

Ok, since this will be delayed until 8.4 (at least) I rescind my earlier concerns that this will hinder the Python 3 port. After a thorough reading of the release notes I don't think there's anything that will hurt, and there are some things that will almost certainly help (such a dict insertion order preservation).

I just need to see how Python 3.7 does on Cygwin (originally I was aiming to have CPython fully supported on Cygwin by 3.7, but I just didn't have the time, though we did get some important big fixes in).

[Frédéric Chapoton]

branch does not apply

[vklein]

@embray Have you some news on Python3.7 on Cygwin ?

[Erik Bray]

My only news is I have multiple upstream PRs necessary for 3.7 to at least build, at a bare minimum, on Cygwin, and they have been stalled for quite a bit. I was just thinking yesterday of poking python-dev soon for someone to look at them.

I actually now maintain an official CPython buildbot for Cygwin, but it's all but useless until those fixes are accepted: ​https://buildbot.python.org/all/#/builders/164

[Erik Bray]

The more I think about it, the more I'm also hesitant to make 3.7 the Python for Sage until 3.7 is in more distros. But I'm not sure. It would be easier to depend on an arbitrary Python 3 (or at least 3.4+) if we were also dropping Python 2 support completely. I don't think we've set a schedule for doing that, but I doubt that will happen in Sage until after the Python 2 EoL.

[vklein]

Ok i see (​https://github.com/python/cpython/pulls/embray).

So does that mean you think we should fix problems like itertools.islice in sage rather than waiting for 3.7 ?

[Erik Bray]

Replying to vklein:

Ok i see (​https://github.com/python/cpython/pulls/embray).

Wow, I have several PRs pending... FWIW the only two that are major blockers for Python on Cygwin are:

​https://github.com/python/cpython/pull/4348 ​https://github.com/python/cpython/pull/8712

Of course, that alone doesn't necessarily need to block Sage's Python 3 spkg--as it is I believe we already include some custom patches for Cygwin, and I'm okay with that for now. I just brought it up since you asked about the status on Cygwin. Also, beyond those patches, there may be other issues. I actually haven't spent much time on it since I was hoping to get some movement on those bare minimum issues before I do much more... *sigh*

So does that mean you think we should fix problems like itertools.islice in sage rather than waiting for 3.7 ?

Yes, and for two reasons:

1) We've already fixed those problems in several cases; probably even the majority of them (I have a few more in my Python 3 branch I haven't even made tickets for yet).

2) We will likely have to work around that anyway for downstream distros, unless we want to make 3.7 a hard requirement for Sage. But that might be tough, at least in the near term...

[Samuel Lelièvre]

Python 3.7.1 was released.

[Samuel Lelièvre]

Python 3.7.2rc1 was released: ​Python insider blog post: Python 3.7.2rc1.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​3d98676

Upgrade to Python 3.7.2

[vklein]

When running sage in py3 a difference with python3.6 is that we get a bunch of doctests errors of type DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working. They mainly comes form matplotlib package but also from packages jinja2, networkx and werkzeug.

[Jeroen Demeyer]

Replying to vklein:

When running sage in py3 a difference with python3.6 is that we get a bunch of doctests errors of type DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working. They mainly comes form matplotlib package but also from packages jinja2, networkx and werkzeug.

We could just hide that warning.

[vklein]

Python version	Files having failures	failing doctests
3.6.6	166	717
3.7.2	387	1104

Numbers of DeprecationWarning: Using or importing the ABCs from 'collections'... : 230.

new python3.7 failures : 407

python3.6 failures not appearing with python 3.7 : 18

Link for the doctest logs ​here

[Jeroen Demeyer]

By the way, I don't think that increasing the number of doctest failures should prevent this ticket from being merged. We'll need to support Python 3.7 sooner or later, so we might as well do it sooner.

[vklein]

Yes i agree with that. I am ok to give positive review for this ticket as you can now run sage with this python version.

@embray can you confirm that it's ok for Cygwin now ? ((​https://github.com/python/cpython/pull/4348 ​​https://github.com/python/cpython/pull/8712 have been merged)

[Frédéric Chapoton]

oh, great, 400 new doctests to fix.. really great.. I was afraid to start lacking things to do...

Does this pass the python3 test target ?

If not, it will fail on the buildbot that checks exactly that.. and maybe the release manager will say niet.

[Jeroen Demeyer]

Like I said, we would need to fix those failures at some point anyway. I don't think that we want to stay on Python 3.6 forever.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​99cbc2d

Disable collections.abc warnings

[Jeroen Demeyer]

Most remaining failures are of the form

sage -t --long src/sage/rings/number_field/galois_group.py
**********************************************************************
File "src/sage/rings/number_field/galois_group.py", line 20, in sage.rings.number_field.galois_group
Failed example:
    G == loads(dumps(G))
Exception raised:
    Traceback (most recent call last):
      File "/home/dimpase/sage-python3/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 671, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/dimpase/sage-python3/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1095, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.rings.number_field.galois_group[3]>", line 1, in <module>
        G == loads(dumps(G))
      File "sage/misc/persist.pyx", line 968, in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:7256)
        return unpickler.load()
      File "/home/dimpase/sage-python3/local/lib/python3.7/site-packages/sage/structure/sequence.py", line 573, in extend
        self._require_mutable()
      File "/home/dimpase/sage-python3/local/lib/python3.7/site-packages/sage/structure/sequence.py", line 740, in _require_mutable
        if self._is_immutable:
      File "/home/dimpase/sage-python3/local/lib/python3.7/site-packages/sage/structure/sequence.py", line 879, in __getattr__
        raise AttributeError("'Sequence_generic' object has no attribute '%s'"%name)
    AttributeError: 'Sequence_generic' object has no attribute '_is_immutable'
**********************************************************************

[vklein]

The filter works ! So it will be more something like 177 new doctests failures rather than 407.

[Jeroen Demeyer]

Also

DeprecationWarning: time.clock has been deprecated in Python 3.3 and will be removed from Python 3.8: use time.perf_counter or time.process_time instead

[Jeroen Demeyer]

Replying to chapoton:

oh, great, 400 new doctests to fix.. really great.. I was afraid to start lacking things to do...

It's not as bad as it sounds. Those 400 doctests are mostly just instances of either of two failures. So fixing those two should bring that number down to a very small number.

[Frédéric Chapoton]

does not build on my patchbot (and others)

[Jeroen Demeyer]

Replying to chapoton:

does not build

Details please...

[François Bissey]

The only thing I can see from the patch bot is that the python 3.7 tarball is not mirrored so the bots fail to fetch it.

[Frédéric Chapoton]

yes, sorry for the noise..

Is there good hope to fix here the

AttributeError: 'Sequence_generic' object has no attribute '_is_immutable

errors ? I remember that the Galois group code is a bit antique and somewhat ugly..

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​d4a4913	Upgrade to Python 3.7.2
​f7bb957	Disable collections.abc warnings

[Jeroen Demeyer]

I'm looking into this:

sage: L = Sequence(range(10))
sage: L
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
sage: loads(dumps(L))
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-3-21e74d398e6e> in <module>()
----> 1 loads(dumps(L))

/home/dimpase/sage-python3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:7256)()
    966 
    967     unpickler = SageUnpickler(io.BytesIO(s))
--> 968     return unpickler.load()
    969 
    970 

/home/dimpase/sage-python3/local/lib/python3.7/site-packages/sage/structure/sequence.py in extend(self, iterable)
    571             [1, 2, 3, 0, 1, 2, 3]
    572         """
--> 573         self._require_mutable()
    574         v = [self.__universe(x) for x in iterable]
    575         list.extend(self, v)

/home/dimpase/sage-python3/local/lib/python3.7/site-packages/sage/structure/sequence.py in _require_mutable(self)
    738             ValueError: object is immutable; please change a copy instead.
    739         """
--> 740         if self._is_immutable:
    741             raise ValueError("object is immutable; please change a copy instead.")
    742 

/home/dimpase/sage-python3/local/lib/python3.7/site-packages/sage/structure/sequence.py in __getattr__(self, name)
    877             return self.__hash
    878         else:
--> 879             raise AttributeError("'Sequence_generic' object has no attribute '%s'"%name)
    880 seq = Sequence
    881 

AttributeError: 'Sequence_generic' object has no attribute '_is_immutable'

This is causing the vast majority of all new doctest failures.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​e233a9e	Implement pickling for Sequence_generic
​0d4139c	Upgrade to Python 3.7.2
​830749c	Disable collections.abc warnings

[Jeroen Demeyer]

There is just one failure with ptest-python3: a timeout in src/sage/parallel/map_reduce.py

[vklein]

Is this timeout somehow related with python3.7 ? If not with #27334 i am ok to set this ticket in positive review.

[Jeroen Demeyer]

I'm not entirely sure what is causing that timeout... but I also don't think that it should prevent merging this ticket.

[vklein]

Ok then.

[Frédéric Chapoton]

Does this pass the refreshed python3 test target of 8.7.b5 (that checks more modules)?

If not, it will fail on the buildbot that checks exactly that.. and maybe the release manager will say niet.

[Jeroen Demeyer]

I would argue that we can always change the list of passing doctests in this ticket. Otherwise, we are aiming for a moving target and this ticket won't ever be fixed.

[Frédéric Chapoton]

yes, sure. I would very much like this ticket to be fixed, and I do not want to do anything against it. But still maybe somebody should check if the passing doctests needs to be changed ?

[Volker Braun]

Ironically, this fails on py3:

sage -t --long src/sage_setup/docbuild/__init__.py
**********************************************************************
File "src/sage_setup/docbuild/__init__.py", line 98, in sage_setup.docbuild.builder_helper
Failed example:
    build_many(build_ref_doc, [("docname", "en", "html", {})])
Expected:
    Traceback (most recent call last):
    ...
    Exception: ('Non-exception during docbuild: abort pool operation', BaseException('abort pool operation',))
Got:
    <BLANKLINE>
    multiprocessing.pool.RemoteTraceback: 
    """
    Traceback (most recent call last):
      File "/var/lib/buildbot/slave/sage3_git/build/local/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 129, in f
        runsphinx()
      File "<doctest sage_setup.docbuild.builder_helper[3]>", line 2, in raiseBaseException
        raise BaseException("abort pool operation")
    BaseException: abort pool operation
    <BLANKLINE>
    During handling of the above exception, another exception occurred:
    <BLANKLINE>
    Traceback (most recent call last):
      File "/var/lib/buildbot/slave/sage3_git/build/local/lib/python3.7/multiprocessing/pool.py", line 121, in worker
        result = (True, func(*args, **kwds))
      File "/var/lib/buildbot/slave/sage3_git/build/local/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
        return list(map(*args))
      File "/var/lib/buildbot/slave/sage3_git/build/local/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 73, in build_ref_doc
        getattr(ReferenceSubBuilder(doc, lang), format)(*args, **kwds)
      File "/var/lib/buildbot/slave/sage3_git/build/local/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 742, in _wrapper
        getattr(DocBuilder, build_type)(self, *args, **kwds)
      File "/var/lib/buildbot/slave/sage3_git/build/local/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 138, in f
        raise Exception("Non-exception during docbuild: %s"%(e,), e)
    Exception: ('Non-exception during docbuild: abort pool operation', BaseException('abort pool operation'))
    """
    <BLANKLINE>
    The above exception was the direct cause of the following exception:
    <BLANKLINE>
    Traceback (most recent call last):
      File "/var/lib/buildbot/slave/sage3_git/build/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 671, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/var/lib/buildbot/slave/sage3_git/build/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1095, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage_setup.docbuild.builder_helper[7]>", line 1, in <module>
        build_many(build_ref_doc, [("docname", "en", "html", {})])
      File "/var/lib/buildbot/slave/sage3_git/build/local/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 282, in build_many
        ret = x.get(99999)
      File "/var/lib/buildbot/slave/sage3_git/build/local/lib/python3.7/multiprocessing/pool.py", line 657, in get
        raise self._value
    Exception: ('Non-exception during docbuild: abort pool operation', BaseException('abort pool operation'))
**********************************************************************
1 item had failures:
   1 of  10 in sage_setup.docbuild.builder_helper
    [36 tests, 1 failure, 0.65 s]

[John Palmieri]

Here's what I got for the test suite on OS X:

Ran 93 tests in 225.537s

FAILED (errors=3, skipped=1)
8 tests failed again:
    test_normalization test_robotparser test_site test_ssl
    test_sysconfig test_urllib test_urllib2 test_urllib2_localnet

== Tests result: FAILURE then FAILURE ==

389 tests OK.

8 tests failed:
    test_normalization test_robotparser test_site test_ssl
    test_sysconfig test_urllib test_urllib2 test_urllib2_localnet

16 tests skipped:
    test_dbm_gnu test_devpoll test_epoll test_gdb test_msilib
    test_multiprocessing_fork test_ossaudiodev test_spwd
    test_startfile test_tix test_tk test_ttk_guionly test_winconsoleio
    test_winreg test_winsound test_zipfile64

8 re-run tests:
    test_normalization test_robotparser test_site test_ssl
    test_sysconfig test_urllib test_urllib2 test_urllib2_localnet

3 tests run no tests:
    test_dtrace test_future4 test_largefile

Total duration: 48 min 44 sec
Tests result: FAILURE then FAILURE
make[2]: *** [test] Error 2
An error occurred while testing Python

[Erik Bray]

Please also don't set this back to positive review until I've had a chance to try it. I didn't even know this had been closed. I don't know why you added me to the reviewers when I haven't even commented on this ticket in 5 months (note: I'm not opposed to it but I haven't even looked at it since it started having activity again).

[Erik Bray]

(For some reason the workflow was in an odd state where I couldn't set it directly back to needs review)

[Erik Bray]

There are two critical fixes needed for Python 3.7 to even build on Cygwin:

• ​One of them has been backported to the 3.7 branch, and I confirmed is in 3.7.2.

• ​The other has not been backported to 3.7, which is annoying. I'm not sure why not; I'll have to look into what I have to do to request a backport. In the meantime this patch needs to be included, at the very least.

[vklein]

Replying to embray:

Please also don't set this back to positive review until I've had a chance to try it. I didn't even know this had been closed. I don't know why you added me to the reviewers when I haven't even commented on this ticket in 5 months (note: I'm not opposed to it but I haven't even looked at it since it started having activity again).

Sorry but i thought you already had validated 3.7.2 for OSX. I was wrong.

[John Palmieri]

For comparison, here is what happens when I run the tests with Python 3.6.6:

Ran 93 tests in 225.364s

FAILED (errors=3, skipped=1)
10 tests failed again:
    test_asyncio test_normalization test_pyexpat test_robotparser
    test_site test_ssl test_sysconfig test_urllib test_urllib2
    test_urllib2_localnet

== Tests result: FAILURE then FAILURE ==

382 tests OK.

10 tests failed:
    test_asyncio test_normalization test_pyexpat test_robotparser
    test_site test_ssl test_sysconfig test_urllib test_urllib2
    test_urllib2_localnet

15 tests skipped:
    test_dbm_gnu test_devpoll test_epoll test_gdb test_msilib
    test_ossaudiodev test_spwd test_startfile test_tix test_tk
    test_ttk_guionly test_winconsoleio test_winreg test_winsound
    test_zipfile64

10 re-run tests:
    test_asyncio test_normalization test_pyexpat test_robotparser
    test_site test_ssl test_sysconfig test_urllib test_urllib2
    test_urllib2_localnet

Total duration: 49 min 37 sec
Tests result: FAILURE then FAILURE
make[2]: *** [test] Error 2
An error occurred while testing Python

[Frédéric Chapoton]

The second Cygwin fix seems to have been merged, no ?

​https://github.com/python/cpython/pull/12063

[Erik Bray]

Replying to chapoton:

The second Cygwin fix seems to have been merged, no ?

​https://github.com/python/cpython/pull/12063

Yes, 6 days ago, after I made noise about it.

[Frédéric Chapoton]

Should we wait for a release of python 3.7 ?

[Erik Bray]

Not necessarily, we can still include the patch now.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​8c12c27	Upgrade to Python 3.7.2
​bb0564d	Disable collections.abc warnings
​b94ad2c	Fix "abort pool operation" docbuild doctest

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​b08639c	Upgrade to Python 3.7.2
​2d6bb9a	Disable collections.abc warnings
​a42152c	Fix "abort pool operation" docbuild doctest

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​36c93f1	Upgrade to Python 3.7.2
​b03a9f8	Disable collections.abc warnings
​e5f3023	Fix "abort pool operation" docbuild doctest

[Jeroen Demeyer]

I'm getting a lot of failures due to

sage: gap_reset_workspace()
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-1-01843f720c90> in <module>()
----> 1 gap_reset_workspace()

/home/jdemeyer/sage-python3/local/lib/python3.7/site-packages/sage/interfaces/gap.py in gap_reset_workspace(max_workspace_size, verbose)
   1577     # Create new workspace with filename WORKSPACE
   1578     g = Gap(use_workspace_cache=False, max_workspace_size=None)
-> 1579     g.eval('SetUserPreference("HistoryMaxLines", 30)')
   1580     from sage.tests.gap_packages import all_installed_packages
   1581     for pkg in all_installed_packages():

/home/jdemeyer/sage-python3/local/lib/python3.7/site-packages/sage/interfaces/gap.py in eval(self, x, newlines, strip, split_lines, **kwds)
    581             if not input_line.endswith(';'):
    582                 input_line += ';'
--> 583         result = Expect.eval(self, input_line, **kwds)
    584         if not newlines:
    585             result = result.replace("\\\n","")

/home/jdemeyer/sage-python3/local/lib/python3.7/site-packages/sage/interfaces/expect.py in eval(self, code, strip, synchronize, locals, allow_use_file, split_lines, **kwds)                                                                                                                                                                    
   1349                 elif split_lines:
   1350                     return '\n'.join([self._eval_line(L, allow_use_file=allow_use_file, **kwds)
-> 1351                                         for L in code.split('\n') if L != ''])
   1352                 else:
   1353                     return self._eval_line(code, allow_use_file=allow_use_file, **kwds)

/home/jdemeyer/sage-python3/local/lib/python3.7/site-packages/sage/interfaces/expect.py in <listcomp>(.0)
   1349                 elif split_lines:
   1350                     return '\n'.join([self._eval_line(L, allow_use_file=allow_use_file, **kwds)
-> 1351                                         for L in code.split('\n') if L != ''])
   1352                 else:
   1353                     return self._eval_line(code, allow_use_file=allow_use_file, **kwds)

/home/jdemeyer/sage-python3/local/lib/python3.7/site-packages/sage/interfaces/gap.py in _eval_line(self, line, allow_use_file, wait_for_prompt, restart_if_needed)
    741         try:
    742             if self._expect is None:
--> 743                 self._start()
    744             E = self._expect
    745             #import pdb; pdb.set_trace()

/home/jdemeyer/sage-python3/local/lib/python3.7/site-packages/sage/interfaces/gap.py in _start(self)
   1272 
   1273         # set random seed
-> 1274         self.set_seed(self._seed)
   1275 
   1276     def _function_class(self):

/home/jdemeyer/sage-python3/local/lib/python3.7/site-packages/sage/interfaces/gap.py in set_seed(self, seed)
   1177         if seed is None:
   1178             seed = self.rand_seed()
-> 1179         self.eval("Reset(GlobalMersenneTwister,%d)" % seed)
   1180         self.eval("Reset(GlobalRandomSource,%d)" % seed)
   1181         self._seed = seed

/home/jdemeyer/sage-python3/local/lib/python3.7/site-packages/sage/interfaces/gap.py in eval(self, x, newlines, strip, split_lines, **kwds)
    581             if not input_line.endswith(';'):
    582                 input_line += ';'
--> 583         result = Expect.eval(self, input_line, **kwds)
    584         if not newlines:
    585             result = result.replace("\\\n","")

/home/jdemeyer/sage-python3/local/lib/python3.7/site-packages/sage/interfaces/expect.py in eval(self, code, strip, synchronize, locals, allow_use_file, split_lines, **kwds)
   1349                 elif split_lines:
   1350                     return '\n'.join([self._eval_line(L, allow_use_file=allow_use_file, **kwds)
-> 1351                                         for L in code.split('\n') if L != ''])
   1352                 else:
   1353                     return self._eval_line(code, allow_use_file=allow_use_file, **kwds)

/home/jdemeyer/sage-python3/local/lib/python3.7/site-packages/sage/interfaces/expect.py in <listcomp>(.0)
   1349                 elif split_lines:
   1350                     return '\n'.join([self._eval_line(L, allow_use_file=allow_use_file, **kwds)
-> 1351                                         for L in code.split('\n') if L != ''])
   1352                 else:
   1353                     return self._eval_line(code, allow_use_file=allow_use_file, **kwds)

/home/jdemeyer/sage-python3/local/lib/python3.7/site-packages/sage/interfaces/gap.py in _eval_line(self, line, allow_use_file, wait_for_prompt, restart_if_needed)
    753             # garbage should be filtered out by this point); here we decode
    754             # them (on Python 3), currently just using the default encoding
--> 755             normal, error = bytes_to_str(normal), bytes_to_str(error)
    756 
    757             if len(error):

/home/jdemeyer/sage-python3/local/lib/python3.7/site-packages/sage/cpython/string.pxd in sage.cpython.string.bytes_to_str (build/cythonized/sage/cpython/string.c:1515)()
     29 
     30 
---> 31 cpdef inline str bytes_to_str(b, encoding=None, errors=None):
     32     r"""
     33     Convert ``bytes`` to ``str``.

/home/jdemeyer/sage-python3/local/lib/python3.7/site-packages/sage/cpython/string.pxd in sage.cpython.string.bytes_to_str (build/cythonized/sage/cpython/string.c:1399)()
     58         return <str>b
     59     else:
---> 60         return _cstr_to_str(<bytes>b, encoding, errors)
     61 
     62 

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 15: invalid start byte

[Jeroen Demeyer]

But why is GAP producing a byte 0x80 in its output?

[Frédéric Chapoton]

what is the locale used ?

[Jeroen Demeyer]

Replying to chapoton:

what is the locale used ?

I have no idea, how do I figure that out?

[Jeroen Demeyer]

In any case, Python 3 was working quite well on this system before. So it's a regression since a few betas ago.

[Frédéric Chapoton]

Maybe using

echo $LANG
echo $LC_ALL

[Frédéric Chapoton]

Does the current branch include the patches required for Cygwin ?

[Jeroen Demeyer]

Replying to chapoton:

Does the current branch include the patches required for Cygwin ?

Yes, I added the patches that @embray mentioned.

I have not looked further into the GAP issue. Maybe somebody else should try it.

[John Palmieri]

I don't see the GAP problem when I do a Python 3 build with this branch on OS X:

┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 8.7.beta7, Release Date: 2019-03-10               │
│ Using Python 3.7.2. Type "help()" for help.                        │
└────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Warning: this is a prerelease version, and it may be unstable.     ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
sage: gap_reset_workspace()
sage: 

and

$ ./sage -t src/sage/interfaces/gap.py 
too many failed tests, not using stored timings
Running doctests with ID 2019-03-13-13-25-14-a8d0606b.
Git branch: t/25680/upgrade_to_python_3_7_0
Using --optional=dochtml,mpir,python2,sage
Doctesting 1 file.
sage -t src/sage/interfaces/gap.py
    [222 tests, 6.84 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 8.3 seconds
    cpu time: 1.9 seconds
    cumulative wall time: 6.8 seconds
$ ./sage -t src/sage/interfaces/gap_workspace.py
too many failed tests, not using stored timings
Running doctests with ID 2019-03-13-13-25-35-b861aafd.
Git branch: t/25680/upgrade_to_python_3_7_0
Using --optional=dochtml,mpir,python2,sage
Doctesting 1 file.
sage -t src/sage/interfaces/gap_workspace.py
    [8 tests, 0.01 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 0.0 seconds
    cpu time: 0.0 seconds
    cumulative wall time: 0.0 seconds

[Frédéric Chapoton]

I have two questions here:

• do we have the approval of Erik (about Windows) ?
• does the branch pass all the tests in the current python3 target ?

EDIT: I have no problem with gap either.

[Jeroen Demeyer]

I reduced the problem to the shell command

echo 'Reset(GlobalMersenneTwister,500837550);' | ./sage --gap -r -b -p -T -E local/share/sage/ext/gap/sage.g

This is outputting binary garbage which Python 3 cannot convert to a string.

I don't think it's caused by this ticket, but it does prevent me from working on Python 3.

[Jeroen Demeyer]

Even in an interactive GAP session, I'm getting the same binary garbage:

 *********   GAP 4.10.0 of 01-Nov-2018
 *  GAP  *   https://www.gap-system.org
 *********   Architecture: x86_64-pc-linux-gnu-default64
 Configuration:  gmp 6.0.0, readline
 Loading the library and packages ...
 Packages:   Alnuth 3.1.0, AtlasRep 1.5.1, AutPGrp 1.10, CRISP 1.4.4, CTblLib 1.2.2, FactInt 1.6.2, FGA 1.4.0, GAPDoc 1.6.2, IRREDSOL 1.4, LAGUNA 3.9.0, 
             Polycyclic 2.14, PrimGrp 3.3.2, SmallGrp 1.3, Sophus 1.24, TomLib 1.2.7, TransGrp 2.0.4
 Try '??help' for help. See also '?copyright', '?cite' and '?authors'
gap> Reset(GlobalMersenneTwister,0);
�>[>�1P[�\"�2�ʦ\005�t>���T��\020@�8��l���\023�%��\023hK�q\033�'�\026fB\025��P��\n~otM��^�����É)\<[\000\b\<G\rs���\n���Q[LN�\030\<�\013�L~8\
x*\t�x�p������l\b�\022P`���^��\021p�=j��I8j��rV\023�~N�^l�9�'�Ᾱ�@^LI[\
I\027��j\023���#\03122q\034���Uv��n�p��\\014�H\bׯ2�9W�.�Ni\034���4J�Tb_�\�2\013�I\000JM�n6ODt`_\007!\023:�*�������_*�qj4�kʪ�-F<pr��H��xR\005�uS�\026\n\
�\027���'\005FT��a&]\024�\026���d���#\rK��)k�\\��L��I��\033o�F��BY D\>�\006tF���^\032-\n��z���f\020>��p�\004K<�
                                                                                                               �lCJ�o\0278|\027P3�Q%D�Q��V\bU
�hR-���]�~N���\c�a�\025[O\036�\037MJt��5�i�\014\bu=)=f��\034\c��\>����*��\006��{o\
\016jws�1�&we5�[�BFǴzl�'�_Q�\021[x�R[�\021\t����7F˵0[N\t���U�/��UP\005\t������\024���IנL(��7��9��l`��L�\027�Pè�8�TdO,r��`RY\\W\026s�h]\c�j|�d���\016�8�@\
�\027�\006�2.�\005�\n6���7`\005[-)\037�N4�=�W��N�+�\025�2a��\037Tʹ�\0164:*k��z�\c,vԹ��&,\025\<&>Pk
�4��\005����I_��Q�Z����f(2�\Iw��\023H�p�\033\c\\\016�����PTd�C\020��ﾀ�8b\025.����\034qc��\037?���)�6'Kd�\033$��Mz���!t;Njn�whȔ5�#D_�� �!<ؾϛ\031O���\
��2�,h8=\blJ$����ͫ��)���\031-<4��z���dj\<�\t<;�g�&u�<GS\027�2E�|[�0{\t�K�\022}
                                                                             \030��\026cW�e�\020���B��C�?]k&��`�\026�!\017\>\ru5�j) �ܮ�Y\005��� \004\
���C�7        �\>n�oҤ\\\000�\>,�\026dJ�n)S#!l�/�8?�7Q\004Jt�ɭ7�\>*B`jmT�g��\036S|]\006\033�\>|\007\004�\037\033                  �EV�&��\bS�P�R\
/߾��ɢ��a\031{i�\r\020N\030��\032)�&��\"{�־�.���~,��\034����I6\030\034f�\032����(4�^`m&w��\000�D\016,\b�[7/ �\025`\030s@ZW;�\024���æMү4�Hc�\033�\"��\006\<O-�FP흛-�A��KW2>p��������q\b�\022�J*\031P��^���yJ�����\016H�C9�,|�j�?�[U��\c\                            �*�j���W[\023|`��.���h�9e{sy�\rw��\b4�[z\023���m\022�\027ȯ[ЩOž\
\036�޾> J\b�G�4;�\r��W\022i���zs���zW=\004���yT�\007�����Lr30��\005�v�%D�\025�\032\037\c\b
                      1�Ft�1�-�\033[Z\035��t���\030YY�-V�o�g���q\007Lb�'J.R�4jC�C���U%\>vƸ%C]�<��[iK#8.�j>O1T��\cl��[�����B?_\0277�\004\0275�\000d���X\
\014\013�\032ijv�.u��\016+��\017�:_������esͺ\023�H\<z�\n\023E���\r����\024k�\n�W�T(����k8\032rJ|=8\"�|�\004gU�׭����L=;�ow�Ҿ`:,\>�mdQ-Ԫ�Bqy}�-��bA{p\<\
\000\0001234" ]

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​f74ecc0	GAP: silence output from Reset(GlobalMersenneTwister)
​93dfaaf	Upgrade to Python 3.7.2
​700e513	Disable collections.abc warnings
​10442de	Fix "abort pool operation" docbuild doctest

[Erik Bray]

Replying to chapoton:

I have two questions here:

• do we have the approval of Erik (about Windows) ?
• does the branch pass all the tests in the current python3 target ?

EDIT: I have no problem with gap either.

For myself, I've been waiting until some of the other issues with this branch are worked out--i.e. that it's otherwise good-to-go on Linux.

But if you think that's nearly the case I should start now, since building on Windows takes some time.

[Frédéric Chapoton]

I have got 3 doctests failure on the python3 test-suite:

sage -t --long src/sage/interacts/debugger.py  # 1 doctest failed
sage -t --long src/sage_setup/docbuild/__init__.py  # 1 doctest failed
sage -t --long src/sage/parallel/map_reduce.py  # Timed out after testing finished

EDIT: in particular, the "abort pool operation" is not fixed as claimed.

[Erik Bray]

Replying to chapoton:

I have got 3 doctests failure on the python3 test-suite:

sage -t --long src/sage/interacts/debugger.py  # 1 doctest failed
sage -t --long src/sage_setup/docbuild/__init__.py  # 1 doctest failed
sage -t --long src/sage/parallel/map_reduce.py  # Timed out after testing finished

The first one I don't know about.

The second one I think Jeroen just fixed: ​https://git.sagemath.org/sage.git/commit/?id=10442de1f848010ec373ea5afa9332885ffaf59c

The third one is a little concerning: That module tends to have some random failures in it and I have made a few fixes to it in the past. Nevertheless apparently it still has bugs. Though I'm surprised it's timing out (albeit after testing finished, meaning it's probably hanging during interpreter shutdown...

[Frédéric Chapoton]

on the very latest branch, I still get

sage -t --long src/sage_setup/docbuild/__init__.py
**********************************************************************
File "src/sage_setup/docbuild/__init__.py", line 105, in sage_setup.docbuild.builder_helper
Failed example:
    try:
        build_many(build_ref_doc, [("docname", "en", "html", {})])
    except Exception as E:
        "Non-exception during docbuild: abort pool operation" in E
Exception raised:
    multiprocessing.pool.RemoteTraceback: 
    """
    Traceback (most recent call last):
      File "/home/chapoton/sage3/local/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 134, in f
        runsphinx()
      File "<doctest sage_setup.docbuild.builder_helper[3]>", line 2, in raiseBaseException
        raise BaseException("abort pool operation")
    BaseException: abort pool operation

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/home/chapoton/sage3/local/lib/python3.7/multiprocessing/pool.py", line 121, in worker
        result = (True, func(*args, **kwds))
      File "/home/chapoton/sage3/local/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
        return list(map(*args))
      File "/home/chapoton/sage3/local/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 80, in build_ref_doc
        getattr(ReferenceSubBuilder(doc, lang), format)(*args, **kwds)
      File "/home/chapoton/sage3/local/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 761, in _wrapper
        getattr(DocBuilder, build_type)(self, *args, **kwds)
      File "/home/chapoton/sage3/local/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 143, in f
        raise Exception("Non-exception during docbuild: %s"%(e,), e)
    Exception: ('Non-exception during docbuild: abort pool operation', BaseException('abort pool operation'))
    """

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last):
      File "<doctest sage_setup.docbuild.builder_helper[7]>", line 2, in <module>
        build_many(build_ref_doc, [("docname", "en", "html", {})])
      File "/home/chapoton/sage3/local/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 289, in build_many
        ret = x.get(99999)
      File "/home/chapoton/sage3/local/lib/python3.7/multiprocessing/pool.py", line 657, in get
        raise self._value
    Exception: ('Non-exception during docbuild: abort pool operation', BaseException('abort pool operation'))

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/home/chapoton/sage3/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 671, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/chapoton/sage3/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1095, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage_setup.docbuild.builder_helper[7]>", line 4, in <module>
        "Non-exception during docbuild: abort pool operation" in E
    TypeError: argument of type 'Exception' is not iterable
**********************************************************************
1 item had failures:
   1 of   9 in sage_setup.docbuild.builder_helper
    [35 tests, 1 failure, 0.49 s]
----------------------------------------------------------------------
sage -t --long src/sage_setup/docbuild/__init__.py  # 1 doctest failed
----------------------------------------------------------------------

and

sage -t --long src/sage/interacts/debugger.py
**********************************************************************
File "src/sage/interacts/debugger.py", line 102, in sage.interacts.debugger.Debug.curframe
Failed example:
    d.curframe()
Expected:
    <frame object at 0x...>
Got:
    <repr(<sage.interfaces.sage0.SageElement at 0x7f0c32222fc0>) failed: TypeError: replace() argument 2 must be str, not SageFunction>
**********************************************************************
1 item had failures:
   1 of   4 in sage.interacts.debugger.Debug.curframe
    [23 tests, 1 failure, 1.36 s]
----------------------------------------------------------------------
sage -t --long src/sage/interacts/debugger.py  # 1 doctest failed
----------------------------------------------------------------------

EDIT: The failing doctest in debugger works fine when run interactively.

[Frédéric Chapoton]

Here is fix for the docbuild doctest, please add to your branch if you agree:

-- a/src/sage_setup/docbuild/__init__.py
+++ b/src/sage_setup/docbuild/__init__.py
@@ -105,7 +105,7 @@ def builder_helper(type):
         sage: try:
         ....:     build_many(build_ref_doc, [("docname", "en", "html", {})])
         ....: except Exception as E:
-        ....:     "Non-exception during docbuild: abort pool operation" in E
+        ....:     "Non-exception during docbuild: abort pool operation" in str(E)
         True
     """
     def f(self, *args, **kwds):

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​7050a4b

Fix "abort pool operation" docbuild doctest

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​eed6a11

py3: fix for repr of frame objects

[Jeroen Demeyer]

I'm now down to one failure: the timeout in map_reduce.py

[Frédéric Chapoton]

The debugger failure is still there for me.

[Jeroen Demeyer]

Replying to chapoton:

The debugger failure is still there for me.

Can you post the complete output of ./sage -t --long src/sage/interacts/debugger.py?

I get

too few successful tests, not using stored timings
Running doctests with ID 2019-03-21-15-27-35-0a07308a.
Git branch: HEAD
Using --optional=dochtml,memlimit,mpir,python2,sage
Doctesting 1 file.
sage -t --long src/sage/interacts/debugger.py
    [23 tests, 2.38 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 2.4 seconds
    cpu time: 0.1 seconds
    cumulative wall time: 2.4 seconds

[Frédéric Chapoton]

I posted it already, but here it is again:

./sage -t --long src/sage/interacts/debugger.py
too many failed tests, not using stored timings
Running doctests with ID 2019-03-21-15-31-58-e3b050c6.
Git branch: python3.7
Using --optional=dochtml,fricas,gfortran,memlimit,mpir,normaliz,python2,sage
Doctesting 1 file.
sage -t --long src/sage/interacts/debugger.py
**********************************************************************
File "src/sage/interacts/debugger.py", line 102, in sage.interacts.debugger.Debug.curframe
Failed example:
    d.curframe()
Expected:
    <frame ...>
Got:
    <repr(<sage.interfaces.sage0.SageElement at 0x7f773c42f870>) failed: TypeError: replace() argument 2 must be str, not SageFunction>
**********************************************************************
1 item had failures:
   1 of   4 in sage.interacts.debugger.Debug.curframe
    [23 tests, 1 failure, 1.38 s]
----------------------------------------------------------------------
sage -t --long src/sage/interacts/debugger.py  # 1 doctest failed
----------------------------------------------------------------------
Total time for all tests: 1.4 seconds
    cpu time: 0.0 seconds
    cumulative wall time: 1.4 seconds

[Jeroen Demeyer]

I have no idea why that's failing for you...

[Frédéric Chapoton]

Let me try on another machine (will take time)

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​919ca41

parallel/map_reduce.py times out

[Jeroen Demeyer]

This last commit is probably controversial, but it's the best I can do. I have no idea what's going on in map_reduce.py.

[Jeroen Demeyer]

As far as I'm concerned, this is now actually ready for review.

[Frédéric Chapoton]

Could we instead tag #py2 the doctest that triggers the time-out ?

[Jeroen Demeyer]

Replying to chapoton:

Could we instead tag #py2 the doctest that triggers the time-out ?

I'd rather not do that. For me, # py2 means doctests that are clearly meant to pass only on Python 2; where the fact that it only works on Python 2 is not considered a bug.

[Frédéric Chapoton]

Then we should fix that bug, and not hide it under the carpet.

[Jeroen Demeyer]

# py2 is hiding it under the carpet. Removing it from python3-known-passing.txt is just deferring it to another time.

[Frédéric Chapoton]

I get the exact same failure in src/sage/interacts/debugger.py in another machine. So maybe this should not be neglected.

[Jeroen Demeyer]

I believe you, but I cannot reproduce the failure in src/sage/interacts/debugger.py

[John Palmieri]

The appropriate tests pass for me on OS X. (I don't see a failure in interacts/debugger.py, either.)

[Frédéric Chapoton]

So maybe it's time for Erik to launch the Windows build..

[Erik Bray]

Hopefully I'll have time tomorrow but not sure. Kinda crazy day. Monday otherwise.

[Erik Bray]

Well, it compiled fine at least (which was to be expected since the necessary patches are applied) but I haven't had a chance to run any tests yet.

[Jeroen Demeyer]

If you agree to deprecate the whole module src/sage/interacts/debugger.py in #27531, then we could easily mark the failing test as # py2 and not bother with it anymore.

[Jeroen Demeyer]

Replying to chapoton:

Could we instead tag #py2 the doctest that triggers the time-out ?

Your suggestion of checking which specific doctest triggers the time-out is a good one. I'll try that now.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​e646670	Mark a test for deprecated functionality as Python 2 only
​2bf16e8	Mark a few tests as known bug (#27537)

[Jeroen Demeyer]

New attempt, hopefully more acceptable. Still needs to be tested.

[Jeroen Demeyer]

This passes make ptest-python3 for me.

[Frédéric Chapoton]

Replying to jdemeyer:

This passes make ptest-python3 for me.

Same here.

[John Palmieri]

I used the updated version of src/ext/doctest/python3-known-passing.txt from #27519, and I also added "--long" to the appropriate line of Makefile (why don't we have this option already?). Tests passed.

[Jeroen Demeyer]

So, is anybody willing to give positive review here?

[Frédéric Chapoton]

Should we wait for Erik's green light ?

[Erik Bray]

Came home after a weekend afk to find that although python3 succeeded in building, the build stopped at cysignals with a strange problem. I think there might have been something else odd going on with my build though so I'll start it again and then run the tests.

[Erik Bray]

Ticket retargeted after milestone closed (if you don't believe this ticket is appropriate for the Sage 8.8 release please retarget manually)

[Erik Bray]

Seeing lots of

[dochtml] [misc     ] loading pickled environment... failed: 'str' object has no attribute '__dict__'

when building the docs. Probably not directly related to this ticket, but worth making note of.

[Jeroen Demeyer]

Python 3.7.3 was just released, so we might as well upgrade to that.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​9d2c0c4	Upgrade to Python 3.7.3
​438c3f4	Disable collections.abc warnings
​e62ddf8	Fix "abort pool operation" docbuild doctest
​0ad1dc1	Mark a test for deprecated functionality as Python 2 only
​5a09cf1	Mark a few tests as known bug (#27537)

[Erik Bray]

Cool, I'll try it.

[Jeroen Demeyer]

3.7.3 allowed me to remove one more Cygwin patch (the one related to the .exe suffix)

[Erik Bray]

Is there any reason its version is "3.7.3.p0" and not just "3.7.3"? Just because we do still apply some patches? Not that I think it matters much, just curious.

[Jeroen Demeyer]

I thought the rule was to use .pX when there are patches. But it's certainly not enforced strictly.

[Erik Bray]

I think that makes sense--after all it is no longer "purely" the 3.7.3 release at that point.

[Jeroen Demeyer]

I'm getting docbuild failures due to

WARNING: cannot copy static file OSError(40, 'Too many levels of symbolic links')

[John Palmieri]

I don't see this warning. You don't have that stupid mathjax recursive link, do you? Oh, make ptest-python3 passes for me.

[Jeroen Demeyer]

I don't know what was wrong with the docs. I did make doc-clean and then it worked.

[Erik Bray]

I am having a bad problem when running the tests where every test outputs something like:

Exception ignored in: <function _TemporaryFileCloser.__del__ at 0x6ffff4d92f0>
Traceback (most recent call last):
  File "/home/embray/src/sagemath/sage-python3/local/lib/python3.7/tempfile.py", line 448, in __del__
    self.close()
  File "/home/embray/src/sagemath/sage-python3/local/lib/python3.7/tempfile.py", line 444, in close
    unlink(self.name)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpw4zt6tfq'

even if the test succeeds. I think this may be a bug (big shock) in the tempfile module itself. I believe it might even be one that I already have a patch for (I seem to recall fixing some issues in tempfile back when I was trying to get all of the Python test suite to pass on Cygwin) but I will need to look into that before saying anything more.

[Erik Bray]

(Obviously the bug itself is relatively harmless: It's just running a __del__ method that unlinks a file but doesn't ignore errors that occur if the file was already deleted. It's more a nuisance than anything).

[Erik Bray]

I see; this is a recurrence of the problem I originally described here: ​https://trac.sagemath.org/ticket/25107#comment:14

It's resurfaced, because the tempfile module has been refactored, now in such a way that merely setting the _TemporaryFileWrapper.deleted attribute no longer has any effect whatsoever (it should really probably be a property which delegates to the new _TemporaryFileCloser object.

I think this may be a broader issue with NamedTemporaryFile that is not unique just to Cygwin: If you create a NamedTemporaryFile(delete=True) in a parent process, then add it as an attribute on a Process subclass, when the worker process exits the file is deleted. But then when the temporary file is closed on the parent side it tries to re-delete the already deleted file. I think more generally it would be best if ENOENT errors were just ignored in this case.

[Erik Bray]

If you would consider including ​this commit, or something like it, that would resolve the problem I'm having with unhandled exceptions in _TemporaryFileCloser.__del__ .

With this fix, I can see more clearly that there are two test failures remaining on Cygwin:

sage -t --long src/sage_setup/docbuild/__init__.py  # 1 doctest failed
sage -t --long src/sage/manifolds/differentiable/vectorfield.py  # Killed due to segmentation fault

The first one is merely a manifestation of #27514. Still need to rethink exactly how that test is implemented.

The second is more disconcerting, being a segfault (but at least repeatable). I will look into it next.

[Erik Bray]

I think the segfault might be related to openblas issues that I've already fixed, but may not be incorporated in my current build (since I started building this Python 3 branch some time ago, possibly before all the relevant openblas fixes were incorporated).

[Frédéric Chapoton]

I am setting this to positive now, so that we can start again making progress on #27519. We can always fix details in later tickets.

[Erik Bray]

Replying to chapoton:

I am setting this to positive now, so that we can start again making progress on #27519. We can always fix details in later tickets.

Why would you do that? I explicitly pointed out that something was broken and proposed a fix, but that fix wasn't applied yet. Doing this "now" has no direct impact on making progress on #27519. You're just rushing things for no reason and leaving them broken.

[Erik Bray]

Replying to embray:

I think the segfault might be related to openblas issues that I've already fixed, but may not be incorporated in my current build (since I started building this Python 3 branch some time ago, possibly before all the relevant openblas fixes were incorporated).

I meant to follow up here. The segfault was nothing to do with Python 3 (fortunately) and is fixed by #27565 . Not exactly sure why I wasn't seeing that bug before--it's possible I was building OpenBLAS out of its recent develop branch which already had that fixed.

[François Bissey]

Do we have a follow up on that ticket anywhere? In sage-on-gentoo I have a segfault when building the documentation with python 3.7.3 (but not 2.7 or 3.6). Am I alone with that issue?

python3.7: /dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_7/build/cythonized/sage/matrix/matrix1.c:14275: __pyx_pf_4sage_6matrix_7matrix1_6Matrix_60matrix_from_columns: Assertion `PyList_Check(__pyx_v_columns)' failed.

[Jeroen Demeyer]

Interesting. You're building with assertions enabled.

[Jeroen Demeyer]

Can you attach or send me that matrix1.c file? In my version of that file, I don't find such an assertion.

[François Bissey]

Attached as requested (compressed).

[Jeroen Demeyer]

Thanks, that clears things up. Follow-up ticket at #27688.