Build sagelib from build/pkgs/sagelib/src, fix `setup.py sdist`, add spkg-src and tox.ini

[Matthias Köppe]

As a follow-up to #29411, we set up a separate directory build/pkgs/sagelib/src with symlinks into SAGE_ROOT/src so that sagelib's setup.py does not have access to the whole tree SAGE_ROOT/src.

We fix setup.py sdist, which can be tested using spkg-src (which builds a pip-installable tarball in upstream/) and tox:

./sage -sh -c '(cd build/pkgs/sagelib/src && tox -v -v)'
./sage -sh -c 'build/pkgs/sagelib/src/.tox/python/bin/sage-ipython'

The call of ./sage -sh is necessary so that SAGE_LOCAL is set. Moreover, for scripts that use sage-env, also SAGE_ENV_SOURCED is crucial for the correct operation; otherwise sage-env sets things based on SAGE_LOCAL again.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​0459006	.gitignore: Ignore generated files in build/pkgs/*/src
​1ab75a9	Build sagelib using the installed sage_setup

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​3d83c12	build/pkgs/sagelib/spkg-src: New
​a2bde50	src/MANIFEST.in: prune .tox
​4b258d3	src/MANIFEST.in: Remove bitrotted file
​a29cd2d	build/pkgs/sagelib/MANIFEST.in: New

[Matthias Köppe]

sdist is broken: Files named ** end up in the tar file.

[Matthias Köppe]

​https://github.com/sagemath/cypari2/issues/93

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​7f8850a	src/module_list.py: Document that it is obsolete
​26a85a4	Merge branch 't/29702/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup' into t/29701/replace_use_of_module_list_optionalextension
​5d5803e	src/sage/graphs/planarity.pyx: Add forgotten distutils directive
​a0be9b6	Merge branch 't/29790/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files__part_5__sage_graphs_' into t/29701/replace_use_of_module_list_optionalextension
​14396af	Merge tag '9.2.beta2' into t/29701/replace_use_of_module_list_optionalextension
​821e337	Merge branch 't/29701/replace_use_of_module_list_optionalextension' into t/29950/build_sagelib_using_installed_sage_setup
​eeb6ba4	build/pkgs/sagelib/src/requirements.txt, tox.ini: New
​ce8b84d	build/pkgs/sagelib/src/MANIFEST.in: Prune sage/ext/interpreters
​a9a1335	build/pkgs/sagelib/src/requirements.txt: Add packages required for runtime
​c3b8f8f	Suggest tox -v -v

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​cdf501f	build/pkgs/sagelib/spkg-src: New
​ef4f5b8	Move src/MANIFEST.in to build/pkgs/sagelib/MANIFEST.in, fix up
​233f6ba	build/pkgs/sagelib/src/requirements.txt, tox.ini: New
​177cd4c	build/pkgs/sagelib/src/MANIFEST.in: Prune sage/ext/interpreters
​07b29fb	build/pkgs/sagelib/src/requirements.txt: Add packages required for runtime
​ad33705	Suggest tox -v -v
​481e212	.gitignore: Ignore generated files in build/pkgs/*/src
​9955a3f	Build sagelib from build/pkgs/sagelib/src

[Matthias Köppe]

    from .maxima import maxima, Maxima
  File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/build/pkgs/sagelib/src/.tox/python/lib/python3.7/site-packages/sage/interfaces/maxima.py", line 1236, in <module>
    script_subdirectory=None)
  File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/build/pkgs/sagelib/src/.tox/python/lib/python3.7/site-packages/sage/interfaces/maxima.py", line 528, in __init__
    raise RuntimeError('You must get the file local/bin/sage-maxima.lisp')
RuntimeError: You must get the file local/bin/sage-maxima.lisp

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​875940d	trac 29111: more tinkering. Fix doctests in tests/cmdline.py.
​4a3d36e	Move 'sage -app' back to src/bin/sage
​3a0193c	src/bin/sage: Remove handling of 'sage -axiom'
​6b04075	Merge branch 't/29111/specify_a_subset_of_sage_command_line_options_that_are_supported_by_sagelib___rather_than_sage_the_distribution' into t/21559/change-src-bin-installation
​9c7116b	src/bin/sage-list-optional, sage-list-experimental, sage-list-standard: Remove deprecated scripts
​831cc09	Merge branch 't/29920/remove_deprecated_scripts_sage_list_optional__sage_list_experimental__sage_list_standard' into t/21559/change-src-bin-installation
​a56dc35	Merge tag '9.2.beta1' into t/29702/public/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup
​e3eca85	Merge branch 'public/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup' of git://trac.sagemath.org/sage into t/21559/change-src-bin-installation
​7d29141	src/setup.py: Do not install removed script sage-rsyncdist
​08f5175	Merge branch 'u/mkoeppe/change-src-bin-installation' of git://trac.sagemath.org/sage into t/29950/build_sagelib_using_installed_sage_setup

[Matthias Köppe]

./sage -sh -c '(cd build/pkgs/sagelib/src && tox -r -v -v)'
...
RuntimeError: libSingular not found--a working Singular install in $SAGE_LOCAL is required for Sage to work
ERROR: InvocationError for command /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/build/pkgs/sagelib/src/.tox/python/bin/python -c 'import sys; "" in sys.path and sys.path.remove(""); import sage.all' (exited with code 1)

This is because of:

$ build/pkgs/sagelib/src/.tox/python/bin/python
Python 3.7.7 (v3.7.7:d7c567b08f, Mar 10 2020, 02:56:16) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sage.env
>>> sage.env.SINGULAR_SO
>>> sage.env.GAP_SO
>>> sage.env.SAGE_LOCAL
'/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/build/pkgs/sagelib/src/.tox/python'
>>> 

[Matthias Köppe]

Passing the correct SAGE_LOCAL suffices to fix this, of course. But the singular library location should be configured separately (#29024).

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​516358b

build/pkgs/sagelib/src/tox.ini: Pass SAGE_LOCAL

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d62da15	sage_setup.command.sage_build: Add the extensions to the distribution
​5e577ab	Merge branch 't/29701/replace_use_of_module_list_optionalextension' into t/29950/build_sagelib_using_installed_sage_setup
​eaced09	build/pkgs/sagelib: Add remaining Python dependencies for sage.all

[Matthias Köppe]

The sage script relies on PATH to be set correctly when invoking auxiliary scripts such as sage-eval. Hence the following does not work correctly:

$ ./sage -sh -c 'build/pkgs/sagelib/src/.tox/python/bin/sage -c "print(sys.path)"'
['/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/src/bin', '/Library/Frameworks/Python.framework/Versions/3.7/lib/python37.zip', '/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7', '/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/lib-dynload', '/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.7/site-packages']

[Matthias Köppe]

This will be fixed by #30013 (src/bin/sage-env: Make sure SAGE_SCRIPTS_DIR is at the beginning of the PATH).

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​b8e366e

build/pkgs/sagelib/spkg-src: chmod +x

[Matthias Köppe]

All dependencies of this ticket are green. Ready for review.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​c5f7f2f	Merge tag '9.2.beta3' into t/21559/change-src-bin-installation
​0473ef3	Merge branch 't/21559/change-src-bin-installation' into t/29950/build_sagelib_using_installed_sage_setup
​06a3609	sage_setup.find.find_python_sources: Add benchmark doctest
​174626c	Merge branch 't/29786/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files__part_4__sage_rings_' into t/29701/replace_use_of_module_list_optionalextension
​034a7f7	Merge branch 'public/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup' of git://trac.sagemath.org/sage into t/29701/replace_use_of_module_list_optionalextension
​d3c608b	Merge tag '9.2.beta3' into t/29701/replace_use_of_module_list_optionalextension
​7244371	Merge branch 't/29701/replace_use_of_module_list_optionalextension' into t/29950/build_sagelib_using_installed_sage_setup

[Matthias Köppe]

Merged in latest versions #29701, #21559 (on top of 9.2.beta3). Needs review

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​3953671	Merge tag '9.2.beta4' into t/29111/specify_a_subset_of_sage_command_line_options_that_are_supported_by_sagelib___rather_than_sage_the_distribution
​19d7f67	Merge branch 't/29111/specify_a_subset_of_sage_command_line_options_that_are_supported_by_sagelib___rather_than_sage_the_distribution' into t/21559/change-src-bin-installation
​4344f89	Merge branch 't/21559/change-src-bin-installation' into t/29950/build_sagelib_using_installed_sage_setup

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​fcad518	Merge tag '9.2.beta7' into t/29701/replace_use_of_module_list_optionalextension
​55c3fbc	src/sage_setup/clean.py: Fix doctest
​01b96b0	Merge branch 't/29701/replace_use_of_module_list_optionalextension' into t/29950/build_sagelib_using_installed_sage_setup

[Dima Pasechnik]

lgtm

[Matthias Köppe]

Failing doctest needs work as noted in #29701.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​8a19fe2

build/make/Makefile.in (sagelib-clean): Clean the new build location

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​ccc67b0

src/sage_setup: Update cythonized_dir in doctests

[Dima Pasechnik]

ok

[Matthias Köppe]

Thanks!