Upgrade R to 3.5.2

[gh-timokau]

R 3.5.1 is out and has changed the dput format (search [1] for dput). That is breaking the sage <-> r conversion in _sage_(), which parses the dput output.

The major difference is that attribute names are now always in a name = value form, instead of the names being seperated into .Names. Concrete example with rs = r.summary(1):

Old:

sage: P.eval('dput(%s)'%rs.name())
'structure(c(1, 1, 1, 1, 1, 1), .Names = c("Min.", "1st Qu.", \n"Median", "Mean", "3rd Qu.", "Max."), class = c("summaryDefault", \n"table"))'

New:

sage: P.eval('dput(%s)'%rs.name())
'structure(c(Min. = 1, `1st Qu.` = 1, Median = 1, Mean = 1, `3rd Qu.` = 1, \nMax. = 1), class = c("summaryDefault", "table"))'

It is possible to pass control = c("showAttributes") (disabling niceNames) to get most of the old behaviour back. Unfortunately that also changes the structure format, which already had the name = value format before, to the .Names format.

Doctest failures with simple upgrade, no changes:

Doctesting 2 files using 4 threads.
sage -t --long /nix/store/zqyp15yakxl49zwd5phjb33fmxid9022-sage-src-8.2/src/sage/stats/r.py
**********************************************************************
File "/nix/store/zqyp15yakxl49zwd5phjb33fmxid9022-sage-src-8.2/src/sage/stats/r.py", line 42, in sage.stats.r.ttest
Failed example:
    a, b = ttest([1,2,3,4,5],[1,2,3,3.5,5.121]); a
Exception raised:
    Traceback (most recent call last):
      File "/nix/store/0ibil6jw4mrqvclmrc504wg0b8zmwvhw-python-2.7.15-env/lib/python2.7/site-packages/sage/doctest/forker.py", line 551, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/nix/store/0ibil6jw4mrqvclmrc504wg0b8zmwvhw-python-2.7.15-env/lib/python2.7/site-packages/sage/doctest/forker.py", line 961, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.stats.r.ttest[0]>", line 1, in <module>
        a, b = ttest([Integer(1),Integer(2),Integer(3),Integer(4),Integer(5)],[Integer(1),Integer(2),Integer(3),RealNumber('3.5'),RealNumber('5.121')]); a
      File "/nix/store/0ibil6jw4mrqvclmrc504wg0b8zmwvhw-python-2.7.15-env/lib/python2.7/site-packages/sage/stats/r.py", line 48, in ttest
        test = myR.t_test(x,y,conf_level = conf_level, **kw)._sage_()
      File "/nix/store/0ibil6jw4mrqvclmrc504wg0b8zmwvhw-python-2.7.15-env/lib/python2.7/site-packages/sage/interfaces/r.py", line 1909, in _sage_
        return eval(exp, globs, globs)
      File "<string>", line 1, in <module>
    NameError: name 'structure' is not defined
**********************************************************************
1 item had failures:
   1 of   2 in sage.stats.r.ttest
    [2 tests, 1 failure, 0.69 s]
sage -t --long /nix/store/zqyp15yakxl49zwd5phjb33fmxid9022-sage-src-8.2/src/sage/interfaces/r.py
**********************************************************************
File "/nix/store/zqyp15yakxl49zwd5phjb33fmxid9022-sage-src-8.2/src/sage/interfaces/r.py", line 1295, in sage.interfaces.r.RElement.tilde
Failed example:
    d = a._sage_()
Exception raised:
    Traceback (most recent call last):
      File "/nix/store/0ibil6jw4mrqvclmrc504wg0b8zmwvhw-python-2.7.15-env/lib/python2.7/site-packages/sage/doctest/forker.py", line 551, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/nix/store/0ibil6jw4mrqvclmrc504wg0b8zmwvhw-python-2.7.15-env/lib/python2.7/site-packages/sage/doctest/forker.py", line 961, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.interfaces.r.RElement.tilde[3]>", line 1, in <module>
        d = a._sage_()
      File "/nix/store/0ibil6jw4mrqvclmrc504wg0b8zmwvhw-python-2.7.15-env/lib/python2.7/site-packages/sage/interfaces/r.py", line 1909, in _sage_
        return eval(exp, globs, globs)
      File "<string>", line 1
        _r_structure(_r_list(structure([1, 2], _Names = ["(Intercept)", "sage5"]), _r_structure([1.40300389255584e-15, -1.89234786867138e-15, -4.90433114572042e-17, 1.6311465870518e-16, 3.7527262886>
                                                                                                                                                                                                      >
    SyntaxError: invalid syntax
**********************************************************************
File "/nix/store/zqyp15yakxl49zwd5phjb33fmxid9022-sage-src-8.2/src/sage/interfaces/r.py", line 1296, in sage.interfaces.r.RElement.tilde
Failed example:
    d['DATA']['coefficients']['DATA'][1]
Exception raised:
    Traceback (most recent call last):
      File "/nix/store/0ibil6jw4mrqvclmrc504wg0b8zmwvhw-python-2.7.15-env/lib/python2.7/site-packages/sage/doctest/forker.py", line 551, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/nix/store/0ibil6jw4mrqvclmrc504wg0b8zmwvhw-python-2.7.15-env/lib/python2.7/site-packages/sage/doctest/forker.py", line 961, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.interfaces.r.RElement.tilde[4]>", line 1, in <module>
        d['DATA']['coefficients']['DATA'][Integer(1)]
    NameError: name 'd' is not defined
**********************************************************************
1 item had failures:
   2 of   6 in sage.interfaces.r.RElement.tilde
    [260 tests, 2 failures, 1.76 s]

Doctest failures with control = c("showAttributes"):

File "/nix/store/zqyp15yakxl49zwd5phjb33fmxid9022-sage-src-8.2/src/sage/interfaces/r.py", line 194, in sage.interfaces.r
Failed example:
    d = rs._sage_()
Exception raised:
    Traceback (most recent call last):
      File "/nix/store/j9c3db3xpl4d81glbpzqq3vyp6nvd3l6-python-2.7.15-env/lib/python2.7/site-packages/sage/doctest/forker.py", line 551, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/nix/store/j9c3db3xpl4d81glbpzqq3vyp6nvd3l6-python-2.7.15-env/lib/python2.7/site-packages/sage/doctest/forker.py", line 961, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.interfaces.r[52]>", line 1, in <module>
        d = rs._sage_()
      File "/nix/store/j9c3db3xpl4d81glbpzqq3vyp6nvd3l6-python-2.7.15-env/lib/python2.7/site-packages/sage/interfaces/r.py", line 1905, in _sage_
        return eval(exp, globs, globs)
      File "<string>", line 1
        _r_structure([Min. = 1, `1st Qu.` = 1.75, Median = 3, Mean = 2.875, `3rd Qu.` = 4, Max_ = 5], _r_class = ["summaryDefault", "table"])
                           ^
    SyntaxError: invalid syntax
**********************************************************************
File "/nix/store/zqyp15yakxl49zwd5phjb33fmxid9022-sage-src-8.2/src/sage/interfaces/r.py", line 1295, in sage.interfaces.r.RElement.tilde
Failed example:
    d = a._sage_()
Exception raised:
    Traceback (most recent call last):
      File "/nix/store/j9c3db3xpl4d81glbpzqq3vyp6nvd3l6-python-2.7.15-env/lib/python2.7/site-packages/sage/doctest/forker.py", line 551, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/nix/store/j9c3db3xpl4d81glbpzqq3vyp6nvd3l6-python-2.7.15-env/lib/python2.7/site-packages/sage/doctest/forker.py", line 961, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.interfaces.r.RElement.tilde[3]>", line 1, in <module>
        d = a._sage_()
      File "/nix/store/j9c3db3xpl4d81glbpzqq3vyp6nvd3l6-python-2.7.15-env/lib/python2.7/site-packages/sage/interfaces/r.py", line 1905, in _sage_
        return eval(exp, globs, globs)
      File "<string>", line 1
        _r_structure(_r_list(coefficients = [`(Intercept)` = 1, sage5 = 2],     residuals = [`1` = 1.40300389255584e-15, `2` = -1.89234786867138e-15,     `3` = -4.90433114572042e-17, `4` = 1.6311465870518e-16, `5` = 3.75272628867564e-16    ], effects = [`(Intercept)` = -15.6524758424985, sage5 = 6.32455532033676,     -8.88178419700125e-16, -1.33226762955019e-15, -1.77635683940025e-15    ], rank = Integer(2), fitted_values = [`1` = 3, `2` = 5, `3` = 7,     `4` = 9, `5` = 11], assign = range(0,2), qr = _r_structure(_r_list(qr = _r_structure([-2.23606797749979,     0.447213595499958, 0.447213595499958, 0.447213595499958,     0.447213595499958, -6.70820393249937, 3.16227766016838, -0.195439507584855,     -0.511667273601693, -0.827895039618531], _Dim = [Integer(5), Integer(2)], _Dimnames = _r_list(        ["1", "2", "3", "4", "5"], ["(Intercept)", "sage5"]), assign = range(0,2)),         qraux = [1.44721359549996, 1.12078825843198], pivot = range(1,3),         tol = 1e-07, rank = Integer(2)), _r_class = "qr"), df_residual = Integer(3),     xlevels = _r_structure(_r_list(), _Names = character(0)), call = "lm(formula = sage24",     terms = "sage3 ~ sage5", model = _r_structure(_r_list(sage3 = [3,     5, 7, 9, 11], sage5 = [1, 2, 3, 4, 5]), terms = "sage3 ~         sage5", row_names = [NA, Integer(5)], _r_class = "data.frame")), _r_class = "lm")
                                                           ^
    SyntaxError: invalid syntax
**********************************************************************
File "/nix/store/zqyp15yakxl49zwd5phjb33fmxid9022-sage-src-8.2/src/sage/interfaces/r.py", line 1808, in sage.interfaces.r.RElement._sage_
Failed example:
    d = rs._sage_()
Exception raised:
    Traceback (most recent call last):
      File "/nix/store/j9c3db3xpl4d81glbpzqq3vyp6nvd3l6-python-2.7.15-env/lib/python2.7/site-packages/sage/doctest/forker.py", line 551, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/nix/store/j9c3db3xpl4d81glbpzqq3vyp6nvd3l6-python-2.7.15-env/lib/python2.7/site-packages/sage/doctest/forker.py", line 961, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.interfaces.r.RElement._sage_[1]>", line 1, in <module>
        d = rs._sage_()
      File "/nix/store/j9c3db3xpl4d81glbpzqq3vyp6nvd3l6-python-2.7.15-env/lib/python2.7/site-packages/sage/interfaces/r.py", line 1905, in _sage_
        return eval(exp, globs, globs)
      File "<string>", line 1
        _r_structure([Min. = 1, `1st Qu.` = 1.75, Median = 3, Mean = 2.875, `3rd Qu.` = 4, Max_ = 5], _r_class = ["summaryDefault", "table"])
                           ^
    SyntaxError: invalid syntax
**********************************************************************
File "/nix/store/zqyp15yakxl49zwd5phjb33fmxid9022-sage-src-8.2/src/sage/stats/r.py", line 42, in sage.stats.r.ttest
Failed example:
    a, b = ttest([1,2,3,4,5],[1,2,3,3.5,5.121]); a
Exception raised:
    Traceback (most recent call last):
      File "/nix/store/j9c3db3xpl4d81glbpzqq3vyp6nvd3l6-python-2.7.15-env/lib/python2.7/site-packages/sage/doctest/forker.py", line 551, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/nix/store/j9c3db3xpl4d81glbpzqq3vyp6nvd3l6-python-2.7.15-env/lib/python2.7/site-packages/sage/doctest/forker.py", line 961, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.stats.r.ttest[0]>", line 1, in <module>
        a, b = ttest([Integer(1),Integer(2),Integer(3),Integer(4),Integer(5)],[Integer(1),Integer(2),Integer(3),RealNumber('3.5'),RealNumber('5.121')]); a
      File "/nix/store/j9c3db3xpl4d81glbpzqq3vyp6nvd3l6-python-2.7.15-env/lib/python2.7/site-packages/sage/stats/r.py", line 48, in ttest
        test = myR.t_test(x,y,conf_level = conf_level, **kw)._sage_()
      File "/nix/store/j9c3db3xpl4d81glbpzqq3vyp6nvd3l6-python-2.7.15-env/lib/python2.7/site-packages/sage/interfaces/r.py", line 1905, in _sage_
        return eval(exp, globs, globs)
      File "<string>", line 1
        _r_structure(_r_list(statistic = [t = 0.0763364058813133], parameter = [df = 7.99838646300247],     p_value = 0.941026372027427, conf_int = _r_structure([-2.21408055475224,     2.36568055475224], conf_level = 0.95), estimate = [`mean of x` = 3,     `mean of y` = 2.9242], null_value = [`difference in means` = 0],     alternative = "two.sided", method = "Welch Two Sample t-test",     data_name = "sage5 and sage11"), _r_class = "htest")
                                            ^
    SyntaxError: invalid syntax

Unfortunately I don't understand the parsing code (and its intentions) well enough to fix this myself.

CC'ing the same people as in the last upgrade.

[1] ​https://stat.ethz.ch/pipermail/r-announce/2018/000628.html

tarball: ​http://cran.r-project.org/src/base/R-3/R-3.5.2.tar.gz

[slelievre]

Should this ticket have #24107 (inclusion of OpenSSL, stage 1) as a dependency, as discussed in #24969 (upgrade to R 3.4.4)?

[gh-timokau]

Yeah, probably. My package manager probably just fetched openSSL, I din't try it with the sage packaging mechanism.

It would still be a great first step for distributions if somebody knew how to fix the dput issue. Currently for example ArchLinux? uses 3.5.0, resulting in failures.

[charpent]

Replying to slelievre:

Should this ticket have #24107 (inclusion of OpenSSL, stage 1) as a dependency, as discussed in #24969 (upgrade to R 3.4.4)?

Yes, indeed. Notwithstanding the dire objections of some, SSL support is now a necessity for R (and it should be...).

I am still searching a way to ensure the possibility of compiling R without SSL support for the purists who whish that, but I have not been successful for now. I thoink I'm ready to throw the towel on this one, and rely on a (standard) openssl package or a systemwide one. stay tuned on #24107...

<ThinkingOutLoud>

For dput change, the only way out is to rewrite the relevant parts of the interface, thus introducing a major incompatibility. This is a problem if we ever want to be able to call a systemwide R, as it has been proposed.

Ideally, version checking should occur at interface initialization, and some function pointer should be initialized according to this check. This pointer should be used everywhere this conversion is needed. Not really easy, but the only way out if we want to be able to use a systemwide R which will not necessarily be >=3.5.0...

A similar initialization is necessary for the (R) library interface. We probably should upgraded Rpy to a 3.5-compatible version. I'm not sure that the dus has already settled on this one...

</ThinkingOutLoud>

Anyway, R 3.5 seems to change a lot of small things. The problem with dput is not the only one, but I am not (yet) at the root of some other problems.

Ideas ? Maybe calling for advice on ​sage-devel ?

[gh-timokau]

For dput change, the only way out is to rewrite the relevant parts of the interface, thus introducing a major incompatibility. This is a problem if we ever want to be able to call a systemwide R, as it has been proposed.

Ideally, version checking should occur at interface initialization, and some function pointer should be initialized according to this check. This pointer should be used everywhere this conversion is needed. Not really easy, but the only way out if we want to be able to use a systemwide R which will not necessarily be >=3.5.0...

Couldn't we simply require the system version of R to be >=3.5.0 in order to be used, in that hypothetical case?

Anyway, R 3.5 seems to change a lot of small things. The problem with dput is not the only one, but I am not (yet) at the root of some other problems.

For me that seemed to be the only error, at least as far as the doctests are concerned.

[gh-timokau]

The latest version is 3.5.1 now.

[slelievre]

Note that a duplicate ticket was opened at #26185.

[infinity0]

Note that that ticket has a patch from me that fixes 6/8 of the test failures, and describes the problem with the remaining 2/8. The parsing code is very very very very brittle and might need to be completely rewritten to work reliably. Or use another method than parsing R's REPL output.

[gh-timokau]

Feel free to use that branch here too and/or amend the description. If you prefer we can also close this ticket and use #26185 instead.

The parsing code is very very very very brittle and might need to be completely rewritten to work reliably

That is what I fear too :/

Or use another method than parsing R's REPL output

That would be best if there is such a method.

[gh-timokau]

I try to fix this in #26596 by replacing the regular expression + dput based interface with rpy2. Help and feedback wanted.

[gh-timokau]

#26596 is ready now. Once that is merged, there is no technical issue with R 3.5 anymore.

[slelievre]

Note: the News section of the ​R-project homepage says R 3.5.2 was released on 2018-12-20.

[dimpase]

perhaps this would also fix #27163

[dimpase]

I'll post a branch in a moment, sorting out patches

[gh-timokau]

Are you aware of the openssl issue?

[dimpase]

What openssl issue?

[dimpase]

New commits:

​37036a4

update are to 3.5.2

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​3c90c03

update R to 3.5.2

[dimpase]

I'm deleting the wishful thinking dependence #24107. We could not wait more years for OpenSSL to be re-licensed.

[gh-timokau]

So what is the solution then? I haven't looked into this much, but my impression was that the R update was blocked by its requirement of openssl and the licensing issues. Can you still ship a newer R version without openssl?

[dimpase]

I don't think anything has changed since R 3.4.4 in this respect. We are not bundling OpenSSL with it, there are no licensing issues to worry about.

[charpent]

Replying to dimpase:

I don't think anything has changed since R 3.4.4 in this respect. We are not bundling OpenSSL with it, there are no licensing issues to worry about.

Therefore, we are now depending on OpenSSL (R is a standard package...).

[dimpase]

I don't think R is the only Sage package that needs OpenSSL or some replacement for it. E.g. curl won't get you things from the web without OpenSSL.

[charpent]

Replying to dimpase:

I don't think R is the only Sage package that needs OpenSSL or some replacement for it. E.g. curl won't get you things from the web without OpenSSL.

Indeed.

But some (quite vocally) argued against depending on OpenSSL ; for now it's just "strongly recommended", not a dependency.

I tried (and failed) to avoid it, but proved that OpenSSL is necessary for R, for some Python and pip functions (among others)... Hence my proposal for integration (which stumbled on the fact that there are no acceptable substitute for some functions), and the wait for the godda OpenSSL relicensing...

[charpent]

On Debian buster (a. k. a. testing), won't give a graphics-able R (see also #27163) :

charpent@zen-book-flip:/usr/local/sage-8$ sage -R

R version 3.5.2 (2018-12-20) -- "Eggshell Igloo"
Copyright (C) 2018 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R est un logiciel libre livré sans AUCUNE GARANTIE.
Vous pouvez le redistribuer sous certaines conditions.
Tapez 'license()' ou 'licence()' pour plus de détails.

R est un projet collaboratif avec de nombreux contributeurs.
Tapez 'contributors()' pour plus d'information et
'citation()' pour la façon de le citer dans les publications.

Tapez 'demo()' pour des démonstrations, 'help()' pour l'aide
en ligne ou 'help.start()' pour obtenir l'aide au format HTML.
Tapez 'q()' pour quitter R.

> capabilities()
       jpeg         png        tiff       tcltk         X11        aqua 
      FALSE       FALSE       FALSE        TRUE       FALSE       FALSE 
   http/ftp     sockets      libxml        fifo      cledit       iconv 
       TRUE        TRUE        TRUE        TRUE        TRUE        TRUE 
        NLS     profmem       cairo         ICU long.double     libcurl 
       TRUE       FALSE       FALSE        TRUE        TRUE        TRUE 
Warning message:
In doTryCatch(return(expr), name, parentenv, handler) :
  impossible de charger l'objet partagé '/usr/local/sage-8/local/lib/R//modules//R_X11.so':
  /usr/lib/x86_64-linux-gnu/libcairo.so.2: undefined symbol: FT_Done_MM_Var
> 

For comparison, on the same machine :

harpent@zen-book-flip:/usr/local/sage-8$ R

R version 3.5.2 (2018-12-20) -- "Eggshell Igloo"
Copyright (C) 2018 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R est un logiciel libre livré sans AUCUNE GARANTIE.
Vous pouvez le redistribuer sous certaines conditions.
Tapez 'license()' ou 'licence()' pour plus de détails.

R est un projet collaboratif avec de nombreux contributeurs.
Tapez 'contributors()' pour plus d'information et
'citation()' pour la façon de le citer dans les publications.

Tapez 'demo()' pour des démonstrations, 'help()' pour l'aide
en ligne ou 'help.start()' pour obtenir l'aide au format HTML.
Tapez 'q()' pour quitter R.

> capabilities()
       jpeg         png        tiff       tcltk         X11        aqua 
       TRUE        TRUE        TRUE        TRUE        TRUE       FALSE 
   http/ftp     sockets      libxml        fifo      cledit       iconv 
       TRUE        TRUE        TRUE        TRUE        TRUE        TRUE 
        NLS     profmem       cairo         ICU long.double     libcurl 
       TRUE        TRUE        TRUE        TRUE        TRUE        TRUE 
> 

[dimpase]

You get the same error on this machine with R 3.4.4. If you can dig up patches that make system's R work on your box, fine, but I have no means to test them.

IMHO this is for another ticket...

[charpent]

Replying to dimpase:

You get the same error on this machine with R 3.4.4. If you can dig up patches that make system's R work on your box, fine, but I have no means to test them.

IMHO this is for another ticket...

That's why I opened #27163...

[charpent]

Replying to charpent:

Replying to dimpase:

You get the same error on this machine with R 3.4.4. If you can dig up patches that make system's R work on your box, fine, but I have no means to test them.

IMHO this is for another ticket...

That's why I opened #27163...

To be fair, the discrepancy between your (positive) results on stretch and my (negative) results on buster show tha the problem is system-dependent. I'll modify #27163 to make that clear.

[dimpase]

how about we going forward with this upgrade? Your problem on buster is pretty much orthogonal to this ticket.

[gh-timokau]

For what its worth, I agree that "just use system openssl" is good enough here. But I'm biased as a packager (who has already switched to the latest R as soon as the switch to rpy2 was merged).

[charpent]

With #27169, passes ptestlong without permanent failure. It also passes its own test suite, up to numerical noise or expected failures (such as Internet tests).

Therefore :

• depends on #27169
• positive review