Port SageNB widgets/interact to Jupyter

[jdemeyer]

Support old SageNB widgets in Jupyter notebooks. This is done by writing wrapper functions with the old SageNB names returning ipywidgets widgets. That is, we essentially do

def input_box(...):
    return ipywidgets.Text(...)

The current implementation should be 100% compatible with ipywidgets, just extending it with Sage functionality.

We also try to support the old SageNB @interact functionality. Many examples of SageNB interacts can be found on ​https://wiki.sagemath.org/interact and we try to support as much as possible in Jupyter.

These are the changes in functionality that I know of:

1. slider(0,10) returns a slider which runs over the integers from 0 to 10 (in SageNB, this ran over floats from 0.0 to 10.0).

2. range_slider() with rational numbers is not implemented: ​https://github.com/ipython/ipywidgets/issues/760 (now implemented in ​https://github.com/jupyter-widgets/ipywidgets/pull/1356)

3. The output is updated only once when a button is clicked multiple times: ​https://github.com/ipython/ipywidgets/issues/763 (the needed hook to implement this is done in ​https://github.com/jupyter-widgets/ipywidgets/pull/1259)

4. Labels do not support HTML, only plain text and LaTeX: ​https://github.com/ipython/ipywidgets/issues/817

5. The visual presentation of widgets can be different.

Tarballs:

• ​https://pypi.python.org/packages/44/69/522a43a2594cd88e41e3677187a7f311983002c37f64b536c9efa430c25d/ipywidgets-6.0.0.rc2.tar.gz
• ​https://pypi.python.org/packages/98/d9/46c853c43ddf420dd3b8149ace91ef2c8c937e995570f75de837ef28bac0/widgetsnbextension-2.0.0rc1.tar.gz

[vbraun]

Are there any API conflicts? And if yes, how are we dealing with that? Separate @jupyterinteract or what? A global switch?

[jdemeyer]

Replying to vbraun:

Are there any API conflicts?

At least, the API is mostly compatible. I don't know any case where SageNB and Jupyter have significantly different behaviour. For the things that SageNB supports but Jupyter does not, either drop support for that feature or work with upstream to implement the feature in Jupyter. I don't strive to achieve 100% compatibility, I'm happy if it mostly works.

In some cases, the visual presentation might be different (Jupyter using a drop-down box while SageNB uses something like radio buttons). I don't care about that.

One potential issue is that SageNB supports all kinds of numbers (Integer, RealNumber, Rational, ...) for sliders while Jupyter only supports int and float.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​f9e723b

Port SageNB interacts to Jupyter

[kcrisman]

WOW how did I not know about this ticket!

[jdemeyer]

Replying to kcrisman:

WOW how did I not know about this ticket!

I'll let you know when I need testers.

[charpent]

I ​posted a temporary workaround for people wanting interactive graphs in Jupyter notebooks using the Sage kernel.

[jdemeyer]

For the record, I consider this branch mostly feature-complete. It still lacks documentation and tests and the required changes to the ipywidgets package.

I have not decided what to do with the upstream situation. There are 3 options:

• add a whole lot of patches to build/pkgs/ipywidgets
• package the upstream master branch
• wait for ipywidgets-6 to be released

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​8704fc0

Fix selector with labels

[novoselt]

Replying to jdemeyer:

... I don't strive to achieve 100% compatibility, I'm happy if it mostly works.

I say that it is quite important to work towards such a goal for all interfaces: SageNB, Jupyter, SageMathCloud, SageMathCell. It is highly annoying to develop a nice interact and then discover that it does not work with another interface. Perhaps SageNB can be left behind now, but I hope that other 3 can converge to a single documented standard...

[jdemeyer]

Replying to novoselt:

I say that it is quite important to work towards such a goal for all interfaces: SageNB, Jupyter, SageMathCloud, SageMathCell. It is highly annoying to develop a nice interact and then discover that it does not work with another interface. Perhaps SageNB can be left behind now, but I hope that other 3 can converge to a single documented standard...

I agree that we should work towards such a goal but some things are not so easy at the moment. In particular, this patch is re-using the existing widgets from ipywidgets, it doesn't add completely new widgets (it doesn't change any Javascript). For example, ipywidgets doesn't have a slider with rational numbers so we cannot have that. Luckily, this can be emulated by a slider taking an arbitrary list of values. But this trick does not work for a range_slider, so a range_slider with rational numbers is not supported here.

[jdemeyer]

IMHO, sliders behave strangely in SageNB:

@interact
def f(x=(1, 10)):
    print(x, parent(x))

This gives a slider with floating-point values between 1 and 10. In ipywidgets, this would become a slider with integer values between 1 and 10. I think that the ipywidgets convention makes more sense here: the type of the variable should correspond to the type in the interact abbrevation (x=(1, 10) gives integers while x=(1.0, 10.0) gives floats).

On top of this, this gives Python floats while in a Sage context, an element of RR would be more appropriate.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6b0dd4c

Allow unicode labels

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​66decb6

Allow unicode labels

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​cf0621d	Fix Python bug #20108: getcallargs() with func keyword
​41e476d	Port SageNB interacts to Jupyter

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​d8e04b1

Port SageNB interacts to Jupyter

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​6d0845d	u(): assume UTF-8
​3809fff	Upgrade ipywidgets
​19cde93	Port SageNB interacts to Jupyter

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​7d3707a	Upgrade Jupyter packages, add enum34
​bd84e65	u(): assume UTF-8
​1f0caa7	Upgrade ipywidgets
​147b980	Port SageNB interacts to Jupyter

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​4957bbc

Port SageNB interacts to Jupyter

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​9e32c28	Upgrade Jupyter packages, add enum34
​e844814	u(): assume UTF-8
​fe7018b	Upgrade ipywidgets
​f443f89	Port SageNB interacts to Jupyter

[charpent]

Replying to jdemeyer:

Is this upgrade ready for review ? Or do you need any form of help before formal review ?

[jdemeyer]

Essentially yes, it's ready.

I didn't formally set the ticket to needs_review because I'd like to wait until the stable release of ipywidgets is released.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​7d15c46	Upgrade Jupyter packages, add enum34
​343abbf	u(): assume UTF-8
​9598944	Upgrade ipywidgets
​7ce92bb	Port SageNB interacts to Jupyter

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​2b1e9fd	Upgrade ipywidgets
​64ee490	Port SageNB interacts to Jupyter

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​8ec0760	Upgrade ipywidgets
​6e6013a	Port SageNB interacts to Jupyter

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​6890482	Upgrade ipywidgets
​053ad42	Port SageNB interacts to Jupyter

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​2dbd6a5	Upgrade ipywidgets
​30955d2	Port SageNB interacts to Jupyter

[charpent]

starting with a freshly fetched 7.6.beta4, make distclean && make ptestlong passes with one well-known-failure :

----------------------------------------------------------------------
sage -t --long src/sage/homology/simplicial_complex.py  # 1 doctest failed
----------------------------------------------------------------------

which passes when ran standalone.

A couple warnings are repeated in widgetsnbextension-2.0.0rc1.log and ipywidgets-6.0.0.rc2.log :

/usr/local/sage-exp/local/lib/python2.7/site-packages/setuptools/dist.py:285: UserWarning: Normalizing '2.0.0.rc1' to '2.0.0rc1'
warning: manifest_maker: standard file '-c' not found
/usr/local/sage-exp/local/lib/python2.7/site-packages/setuptools/dist.py:285: UserWarning: Normalizing '6.0.0.rc2' to '6.0.0rc2'

which look like innocuous booboos.

==> positive_review

For the record : congratulations for this intricate, lengthy and much expected patch. Thank you very much !

I'd recommend expedited inclusion of this update, which should go a long way making the Jupyter notebook as functional as the old one.

I plan to merge this one in my current working branch (which includes #20523 an its dependencies) and give it a couple "real-life" functional tests. But unless the results turns out to be non- or barely-functional (which would surprise me), further enhancements should go towards a new ticket.

[charpent]

Damn !

from the PREP tutorial "Sage Interact Quickstart" :

sage: @interact
sage: def _(f=input_box(x^2,width=20),
....: color=color_selector(widget='colorpicker', label=""),
....: axes=True,
....: fill=True,
....: zoom=range_slider(-3,3,default=(-3,3))):
....:     show(plot(f,(x,zoom[0], zoom[1]), color=color, axes=axes,fill=fill, figsize=3))

(I added the figsize=3)

gives a static fgure, with no controls. I see that in the terminal windows from where I launched the notebook :

[IPKernelApp] WARNING | The installed widget Javascript is the wrong version. It must satisfy the semver range ~2.1.0.

One hidden dependency ? Something to document (in plain English...) ?

Or something more serious ?

BTW : this works well in the old Sage notebook (same binary, ran from another terminal...).

[jdemeyer]

Replying to charpent:

Damn !

from the PREP tutorial "Sage Interact Quickstart" :

sage: @interact
sage: def _(f=input_box(x^2,width=20),
....: color=color_selector(widget='colorpicker', label=""),
....: axes=True,
....: fill=True,
....: zoom=range_slider(-3,3,default=(-3,3))):
....:     show(plot(f,(x,zoom[0], zoom[1]), color=color, axes=axes,fill=fill, figsize=3))

(I added the figsize=3)

gives a static fgure, with no controls. I see that in the terminal windows from where I launched the notebook :

[IPKernelApp] WARNING | The installed widget Javascript is the wrong version. It must satisfy the semver range ~2.1.0.

One hidden dependency ? Something to document (in plain English...) ?

Your example works for me. My guess is that it's something like a browser cache which might interfere. Can you try clearing the cache and try again?

[charpent]

Replying to jdemeyer:

Replying to charpent:

[ Snip ...]

Your example works for me. My guess is that it's something like a browser cache which might

interfere. Can you try clearing the cache and try again?

Done. Same result.

I might have another problem : when I type "SAGE_BROWSER=chromium sage -n jupyter", Sage still opens a Firefox sheet.

Logging out of this sheet, manually opening Chromium, and pasting the address given in the controlling terminal window allows to open the same sheet in Chrome, with the same result.

Suggestions ?

[charpent]

One more data point : same results (incl. Firefox starting with SAGE_BROWSER="chromium") after rebooting ths whole hawg...

I'm at loss.

[jdemeyer]

Replying to charpent:

Suggestions ?

This is most likely not Sage-related, so please report this issue upstream to ipywidgets: ​https://github.com/ipython/ipywidgets/issues

[charpent]

Replying to jdemeyer:

Replying to charpent:

Suggestions ?

This is most likely not Sage-related, so please report this issue upstream to ipywidgets: ​https://github.com/ipython/ipywidgets/issues

This is now ipywidget's ​issue 1611.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​1046695

Fix exceptions in interacts

[jdemeyer]

Replying to charpent:

[IPKernelApp] WARNING | The installed widget Javascript is the wrong version. It must satisfy the semver range ~2.1.0.

Do you happen to have another installation of ipywidgets/widgetsnbextension which might interfere?

I'm setting this back to needs_review because I cannot reproduce your problem. I hope that other people will try it, so we can see if it works for them.

[charpent]

I tried your patch on two different installations (one physical machine, one pristine VM) where it succeeded. Coming back to my initial test case, I took the speps to remove entierely ipywidgets an widgetsnbextension from the systemwide installs. Theren your patch worked.

I conclude that, somehow, the systemwide installation(s) of Python extensions take precedence over a sage-specific installation. hence a couple questions :

• Is that expected or is that a bug ?
• If a bug, does it deserve a ticket ?
• If expected (not a bug), shouldn't this behavior be documented ?

In any case, This lifts my objections ==> positive_review.

And kudos !

[jdemeyer]

Upstream is aware of issues with conflicting installations. In fact that's why I made the suggestion in 55 to check other installations.

[fbissey]

@ Jeroen: quick question, I notice you have added two patches for ipywidgets. Are those upstreamed? If so can you give some references?

[jdemeyer]

Replying to fbissey:

@ Jeroen: quick question, I notice you have added two patches for ipywidgets.

Actually, there are 3 patches. Two of them (get_interact_value_output.patch and output_exception.patch) should be in ​https://pypi.python.org/pypi/ipywidgets/6.0.0.rc3

The other one (widget_repr.patch) is being discussed at ​https://github.com/ipython/ipywidgets/pull/1031 but upstream isn't very happy about it. However, this one is really only relevant for doctesting, not for runtime functionality.

[fbissey]

Replying to jdemeyer:

Replying to fbissey:

@ Jeroen: quick question, I notice you have added two patches for ipywidgets.

Actually, there are 3 patches. Two of them (get_interact_value_output.patch and output_exception.patch) should be in ​https://pypi.python.org/pypi/ipywidgets/6.0.0.rc3

OK, good!

The other one (widget_repr.patch) is being discussed at ​https://github.com/ipython/ipywidgets/pull/1031 but upstream isn't very happy about it. However, this one is really only relevant for doctesting, not for runtime functionality.

OK, I missed that one completely, I suppose that is why I have

File "/usr/lib64/python2.7/site-packages/sage/repl/ipython_kernel/widgets.py", line 349, in sage.repl.ipython_kernel.widgets.Grid.__init__
Failed example:
    w
Expected:
    Grid(value=[[0, 1], [4, 5]], description=u'2x2 matrix', children=(Label(value=u'2x2 matrix'), VBox(children=(EvalText(value=u'0'), EvalText(value=u'4'))), VBox(children=(EvalText(value=u'1'), EvalText(value=u'5')))))
Got:
    <sage.repl.ipython_kernel.widgets.Grid object at 0x7f8d65145f90>

And indeed it doesn't seem to have an impact on functionality.

[jdemeyer]

Follow-up: #22550, please review

[jason]

I updated the description to reflect recent changes in upstream ipywidgets.

[jdemeyer]

Thanks for the pointers, I will certainly follow up on this (but maybe not right now).