Really enable cython caching

[jdemeyer]

In order to speed up Cythonization, we should use a cache. Ticket #15430 tried to make this the default, but due to some bug, it never actually worked.

Besides this, this ticket also fixes the silly precision in timing output (time = 2.35585808754 seconds) and disables stdout buffering such that Cython's messages actually appear in the right order.

[jdemeyer]

New commits:

​dfc4bf9

Enable Cython caching by default

[vbraun]

The cycache fix looks good to me.

I don't understand your comment about order, if cython messages are printed to stdout then flushing stdout does not change their order. If they are printed to stderr then unbuffered stdout is wrong, it should be line buffered to avoid half-written lines.

[jdemeyer]

Replying to vbraun:

I don't understand your comment about order, if cython messages are printed to stdout then flushing stdout does not change their order.

Cython mixes print statements with shell commands. Usually, if you do

print "Python message"
os.system("echo hello")

then the "hello" will usually be output before the "Python message" if output is not to a terminal (output to a terminal is unbuffered by default).

[vbraun]

Ok, I see we are using os.system to launch. Still, we should be using line buffering and not unbuffered mode to avoid truncated lines on the Python stream level. Though the lower-level libc buffering is line buffered by default and that probably saves us.

[jdemeyer]

Replying to vbraun:

Ok, I see we are using os.system to launch.

Where "we" is also Cython, not just Sage.

Still, we should be using line buffering and not unbuffered mode to avoid truncated lines on the Python stream level.

How could unbuffered mode lead to truncated lines?

Though the lower-level libc buffering is line buffered by default and that probably saves us.

I think it's the same level, Python doesn't do it own buffering.

[vbraun]

Replying to jdemeyer:

How could unbuffered mode lead to truncated lines?

Not truncated, but different processe's output would not necessarily be separated by newlines 1111222222111111\n222\n vs. 111111111\n2222222\n

Python doesn't do it own buffering.

To be precise, it did until your patch turned it off. But os.fdopen(,,0) does not call libc setvbuf.

[jdemeyer]

Replying to vbraun:

But os.fdopen(,,0) does not call libc setvbuf.

I admit I have not read Python's sources, but why do you think so?

[jdemeyer]

Replying to vbraun:

Replying to jdemeyer:

How could unbuffered mode lead to truncated lines?

Not truncated, but different processe's output would not necessarily be separated by newlines 1111222222111111\n222\n vs. 111111111\n2222222\n

That can happen with or without buffering. This is an OS-level issue, which libc's buffering doesn't solve.

[vbraun]

Ok, os.fdopen does call setvbuf. Cool.

[jdemeyer]

Replying to vbraun:

Ok, os.fdopen does call setvbuf. Cool.

Side comment: this is something which changed in Python 3, where the libc FILE I/O is no longer used.

[leif]

Replying to jdemeyer:

this ticket also fixes the silly precision in timing output (time = 2.35585808754 seconds)

\o/ Thank you very much, it has been annoying me for years.

[leif]

Cython.Compiler.Main.default_options['cache'] = True

is probably the wrong thing; it should be passed in options to cythonize().

But essentially cythonize() appears to be broken w.r.t. this, since it does:

    ...
    c_options = CompilationOptions(**options)
    ...
    options = c_options    
    ...
    if hasattr(options, 'cache'):
        if not os.path.exists(options.cache):
            os.makedirs(options.cache)

and one may end up with

File "/scratch/sage/local/lib/python2.7/site-packages/Cython/Build/Dependencies.py", line 739, in cythonize
if not os.path.exists(options.cache):
File "/scratch/sage/local/lib/python/genericpath.py", line 18, in exists
os.stat(path)
TypeError: coercing to Unicode: need string or buffer, bool found

as reported on sage-devel.

[saraedum]

Is it possible that cython caching is broken again? At least it does not work for me anymore (i.e., when switching back and forth between branches I do not see a speedup.) Do I have to do anything (create cache directories?) to make it work or should it work out of the box?

[vbraun]

See #17851