Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ImportError: failed to map segment from shared object #610

Open
strogdon opened this issue Nov 22, 2020 · 44 comments
Open

ImportError: failed to map segment from shared object #610

strogdon opened this issue Nov 22, 2020 · 44 comments

Comments

@strogdon
Copy link
Contributor

I've never seen this before. It's present with vanilla and also on Prefix. The failure on Prefix is basically the same but with a different library that can't be imported. The following is on Gentoo.

$ sage -t --long /usr/lib/python3.8/site-packages/sage/repl/all.py 
Running doctests with ID 2020-11-21-19-01-03-837ef810.
Using --optional=dochtml,memlimit,sage
Doctesting 1 file.
  ***   Warning: not enough memory, new PARI stack 432537600
Traceback (most recent call last):
  File "/usr/lib/python-exec/python3.8/sage-runtests", line 182, in <module>
    err = DC.run()
  File "/usr/lib/python3.8/site-packages/sage/doctest/control.py", line 1232, in run
    self.run_doctests()
  File "/usr/lib/python3.8/site-packages/sage/doctest/control.py", line 928, in run_doctests
    self.dispatcher = DocTestDispatcher(self)
  File "/usr/lib/python3.8/site-packages/sage/doctest/forker.py", line 1670, in __init__
    init_sage(controller)
  File "/usr/lib/python3.8/site-packages/sage/doctest/forker.py", line 206, in init_sage
    controller.load_environment()
  File "/usr/lib/python3.8/site-packages/sage/doctest/control.py", line 523, in load_environment
    return import_module(self.options.environment)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/lib/python3.8/site-packages/sage/repl/ipython_kernel/all_jupyter.py", line 5, in <module>
    from sage.all_cmdline import *
  File "/usr/lib/python3.8/site-packages/sage/all_cmdline.py", line 26, in <module>
    from sage.all import *
  File "/usr/lib/python3.8/site-packages/sage/all.py", line 138, in <module>
    from sage.rings.all      import *
  File "/usr/lib/python3.8/site-packages/sage/rings/all.py", line 62, in <module>
    from .padics.all import *
  File "/usr/lib/python3.8/site-packages/sage/rings/padics/all.py", line 3, in <module>
    from .factory import Zp, Zq, Zp as pAdicRing, ZpCR, ZpCA, ZpFM, ZpFP, ZpLC, ZpLF, ZqCR, ZqCA, ZqFM, ZqFP #, ZpL, ZqL
  File "/usr/lib/python3.8/site-packages/sage/rings/padics/factory.py", line 57, in <module>
    from .padic_extension_leaves import (EisensteinExtensionFieldCappedRelative,
  File "/usr/lib/python3.8/site-packages/sage/rings/padics/padic_extension_leaves.py", line 56, in <module>
    from .qadic_flint_FM import qAdicFixedModElement
ImportError: /usr/lib/python3.8/site-packages/sage/rings/padics/qadic_flint_FM.cpython-38-x86_64-linux-gnu.so: failed to map segment from shared object

But the following works:

$ sage -t --long --memlimit=0 /usr/lib/python3.8/site-packages/sage/repl/all.py 
Running doctests with ID 2020-11-21-19-01-53-f1c63596.
Using --optional=dochtml,sage
Doctesting 1 file.
sage -t --long --warn-long 223.3 --random-seed=0 /usr/lib/python3.8/site-packages/sage/repl/all.py
    [0 tests, 0.00 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 0.0 seconds
    cpu time: 0.0 seconds
    cumulative wall time: 0.0 seconds

The failure on Prefix does not exhibit the PARI issue, but adding the --memlimit=0 does allow the doctest to pass.
@kiwifb
Copy link
Collaborator

kiwifb commented Nov 22, 2020

Strange stuff, version of pari, cypari2, singular and flint.

@strogdon
Copy link
Contributor Author 

[I] sci-mathematics/pari
     Available versions:  2.11.4(0/6)^t ~2.13.0(0/7)^t {X data doc fltk gmp test threads}
     Installed versions:  2.11.4(0/6)^t(08:37:52 PM 10/03/2020)(X data doc gmp threads -fltk -test)
     Homepage:            https://pari.math.u-bordeaux.fr/
     Description:         Computer-aided number theory C library and tools

[I] sci-mathematics/flint
     Available versions:  2.5.2-r1(0/13) ~2.6.3(0/14)^t {doc gc ntl static-libs test}
     Installed versions:  2.5.2-r1(0/13)(09:31:29 PM 10/20/2020)(ntl -doc -gc -static-libs)
     Homepage:            http://www.flintlib.org/
     Description:         Fast Library for Number Theory

[U] dev-python/cypari2 [1]
     Available versions:  (~)2.1.1 (~)2.1.2 {PYTHON_TARGETS="python3_6 python3_7 python3_8"}
     Installed versions:  2.1.1(09:24:29 PM 10/04/2020)(PYTHON_TARGETS="python3_7 python3_8 -python3_6")
     Homepage:            https://github.com/sagemath/cypari2
     Description:         A Python interface to the number theory library libpari

[U] sci-mathematics/singular
     Available versions:  4.0.2 ~4.0.3 (~)4.1.1_p2-r2[1] (~)4.1.1_p2-r3[1] ~4.1.3_p2[1] ~4.1.3_p2-r1[1] {boost doc emacs examples python +readline static-libs PYTHON_SINGLE_TARGET="+python2_7"}
     Installed versions:  4.1.1_p2-r2[1](01:11:30 PM 05/11/2020)(readline -emacs -examples -python -static-libs PYTHON_SINGLE_TARGET="-python2_7")
     Homepage:            https://www.singular.uni-kl.de/ https://github.com/Singular/Sources
     Description:         Computer algebra system for polynomial computations
[1] "sage-on-gentoo" /var/lib/layman/sage-on-gentoo

This could have been present for some time since I had been doctesting with `--memlimit=0` for other memory limit failures.

@kiwifb
Copy link
Collaborator

kiwifb commented Nov 22, 2020

That's a very weird message too. What does readelf -d /usr/lib/python3.8/site-packages/sage/rings/padics/qadic_flint_FM.cpython-38-x86_64-linux-gnu.so says?

@strogdon
Copy link
Contributor Author 

$ readelf -d /usr/lib/python3.8/site-packages/sage/rings/padics/qadic_flint_FM.cpython-38-x86_64-linux-gnu.so 

Dynamic section at offset 0x60bf0 contains 26 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libflint.so.13]
 0x0000000000000001 (NEEDED)             Shared library: [libgmp.so.10]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000c (INIT)               0x97f0
 0x000000000000000d (FINI)               0x5469c
 0x0000000000000019 (INIT_ARRAY)         0x260be0
 0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x260be8
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x190
 0x0000000000000005 (STRTAB)             0x19e0
 0x0000000000000006 (SYMTAB)             0x360
 0x000000000000000a (STRSZ)              6153 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000003 (PLTGOT)             0x261000
 0x0000000000000002 (PLTRELSZ)           3912 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x88a8
 0x0000000000000007 (RELA)               0x3400
 0x0000000000000008 (RELASZ)             21672 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffe (VERNEED)            0x33d0
 0x000000006fffffff (VERNEEDNUM)         1
 0x000000006ffffff0 (VERSYM)             0x31ea
 0x000000006ffffff9 (RELACOUNT)          833
 0x0000000000000000 (NULL)               0x0

On Prefix the failure is with a different lib

  File "/storage/strogdon/gentoo-rap/usr/lib/python3.8/site-packages/sage/repl/ipython_kernel/all_jupyter.py", line 5, in <module>
    from sage.all_cmdline import *
  File "/storage/strogdon/gentoo-rap/usr/lib/python3.8/site-packages/sage/all_cmdline.py", line 26, in <module>
    from sage.all import *
  File "/storage/strogdon/gentoo-rap/usr/lib/python3.8/site-packages/sage/all.py", line 133, in <module>
    from sage.libs.all       import *
  File "/storage/strogdon/gentoo-rap/usr/lib/python3.8/site-packages/sage/libs/all.py", line 7, in <module>
    import sage.libs.symmetrica.all as symmetrica
  File "/storage/strogdon/gentoo-rap/usr/lib/python3.8/site-packages/sage/libs/symmetrica/all.py", line 4, in <module>
    from .symmetrica import start, end
ImportError: libsymmetrica-2.0.so: failed to map segment from shared object

Upgrading to symmetrica-3.0.1 to see if things change on Prefix.

@kiwifb
Copy link
Collaborator

kiwifb commented Nov 22, 2020

Wondering if this is something like this https://stackoverflow.com/questions/58975839/dopen-failed-to-map-segment-from-shared-object-when-not-running-as-root in which case changing ulimit may do something. The settings for that stuff sits in /etc/security/limits.conf and there is a limits.d/ folder to read more granular settings.

Interesting point that I have seen sometimes when ssh-ing in a box, some extra limits may be applied.

@strogdon
Copy link
Contributor Author

What do you have for

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 30854
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 99
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 30854
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

I changed core file size to unlimited with no change.

@kiwifb
Copy link
Collaborator

kiwifb commented Nov 22, 2020

I have, when ssh-ing

$ ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 127294
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 127294
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

core file size puts limits on the size of coredumps. Leaving it at 0 means no coredump.

@kiwifb
Copy link
Collaborator

kiwifb commented Nov 22, 2020

My max user processes is way bigger than yours.

@strogdon
Copy link
Contributor Author

On prefix I have

max user processes              (-u) 257289

I'm unable to change uname -u on Gentoo (no sshing involved)

ulimit -u 127294
bash: ulimit: max user processes: cannot modify limit: Operation not permitted

@kiwifb
Copy link
Collaborator

kiwifb commented Nov 22, 2020

You may have to be root to change it. Otherwise, you can change things in /etc/security/limits.conf, I don't know if you need a reboot or a new login for it to be effective.

@strogdon
Copy link
Contributor Author

strogdon commented Nov 22, 2020 •
edited

Same failure on Prefix as above comment after upgrading to symmetrica-3.0.1. user processes on Prefix is twice what you have. There must be some component that's doing this.

@dimpase
Copy link

dimpase commented Nov 23, 2020 •
edited

I am seeing this too on vanilla:
https://groups.google.com/g/sage-devel/c/49cYaiuRzWU/m/FR0fQy0RBwAJ

it's Python 3.7 for me, though.
(And building Sage 9.3.beta1 from source, using as many Gentoo system packages as are currently recognised by spkg-configure macros)

For me --memlimit=0 makes a difference in tests too. E.g.

./sage -tp --memlimit=0  src/doc/de/thematische_anleitungen/sage_gymnasium.rst

works, but omitting --memlimit=0 leads to

./sage -tp src/doc/de/thematische_anleitungen/sage_gymnasium.rst >/tmp/x
  ***   Warning: not enough memory, new PARI stack 432537600
Process DocTestWorker-1:
Traceback (most recent call last):
  File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/mnt/opt/Sage/sage-dev/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 2191, in run
    task(self.options, self.outtmpfile, msgpipe, self.result_queue)
  File "/mnt/opt/Sage/sage-dev/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 2541, in __call__
    result_queue.put(result, False)
  File "/usr/lib/python3.7/multiprocessing/queues.py", line 87, in put
    self._start_thread()
  File "/usr/lib/python3.7/multiprocessing/queues.py", line 170, in _start_thread
    self._thread.start()
  File "/usr/lib/python3.7/threading.py", line 852, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread

and in the output I see this ImportError:

File "src/doc/de/thematische_anleitungen/sage_gymnasium.rst", line 615, in doc.de.thematische_anleitungen.sage_gymnasium
Failed example:
    plot(f)
Exception raised:
    Traceback (most recent call last):
      File "/mnt/opt/Sage/sage-dev/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 720, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/mnt/opt/Sage/sage-dev/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1145, in compile_and_execute
        exec(compiled, globs)
      File "<doctest doc.de.thematische_anleitungen.sage_gymnasium[1]>", line 1, in <module>
        plot(f)
      File "/mnt/opt/Sage/sage-dev/local/lib/python3.7/site-packages/sage/misc/decorators.py", line 492, in wrapper
        return func(*args, **options)
      File "/mnt/opt/Sage/sage-dev/local/lib/python3.7/site-packages/sage/plot/plot.py", line 1961, in plot
        G = funcs.plot(*args, **original_opts)
      File "sage/symbolic/expression.pyx", line 12164, in sage.symbolic.expression.Expression.plot (build/cythonized/sage/symbolic/expression.cpp:65514)
        f = self._plot_fast_callable(param)
      File "sage/symbolic/expression.pyx", line 12222, in sage.symbolic.expression.Expression._plot_fast_callable (build/cythonized/sage/symbolic/expression.cpp:66065)
        return fast_callable(self, vars=vars, expect_one_var=True)
      File "sage/ext/fast_callable.pyx", line 487, in sage.ext.fast_callable.fast_callable (build/cythonized/sage/ext/fast_callable.c:5186)
        import sage.ext.interpreters.wrapper_py
    ImportError: /mnt/opt/Sage/sage-dev/local/lib/python3.7/site-packages/sage/ext/interpreters/wrapper_py.cpython-37m-x86_64-linux-gnu.so: failed to map segment from shared object
...

@dimpase
Copy link

dimpase commented Nov 23, 2020

I certainly recall being able to build and run Sage 9.2 on the box just fine. I'm rebuilding 9.2 now to see if it exhibits the same oddity.

@dimpase
Copy link

dimpase commented Nov 23, 2020

I see that Gentoo now provides Pari 2.13. I wonder if it makes a difference.

@dimpase
Copy link

dimpase commented Nov 23, 2020

Sage 9.2 built from source: deja vu, same ImportError. That is, one needs to bisect Gentoo updates rather than Sage updates to get to the root cause, it seems to me.

@dimpase
Copy link

dimpase commented Nov 23, 2020

pari 2.13 does not get installed due to a weird texlive error. No idea why it is trying to do something with pdftex.fmt

make -j4 docpdf 
cd doc && make docpdf
make[1]: Entering directory '/var/tmp/portage/sci-mathematics/pari-2.13.0/work/pari-2.13.0/doc'
rm -f develop.std
rm -f libpari.std
rm -f parallel.std
pdftex develop
pdftex libpari
pdftex parallel
This is pdfTeX, Version 3.14159265-2.6-1.40.21 (TeX Live 2020 Gentoo Linux) (preloaded format=pdftex)
 restricted \write18 enabled.

kpathsea: Running mktexfmt pdftex.fmt
This is pdfTeX, Version 3.14159265-2.6-1.40.21 (TeX Live 2020 Gentoo Linux) (preloaded format=pdftex)
 restricted \write18 enabled.

kpathsea: Running mktexfmt pdftex.fmt
rm -f tutorial.std
pdftex tutorial
This is pdfTeX, Version 3.14159265-2.6-1.40.21 (TeX Live 2020 Gentoo Linux) (preloaded format=pdftex)
 restricted \write18 enabled.

kpathsea: Running mktexfmt pdftex.fmt
This is pdfTeX, Version 3.14159265-2.6-1.40.21 (TeX Live 2020 Gentoo Linux) (preloaded format=pdftex)
 restricted \write18 enabled.

kpathsea: Running mktexfmt pdftex.fmt
mktexfmt: mktexfmt is using the following fmtutil.cnf files (in precedence order):
mktexfmt:   /etc/texmf/web2c/fmtutil.cnf
mktexfmt:   /usr/share/texmf-dist/web2c/fmtutil.cnf
mktexfmt: mktexfmt is using the following fmtutil.cnf file for writing changes:
mktexfmt:   /var/tmp/portage/sci-mathematics/pari-2.13.0/homedir/.texlive/texmf-config/web2c/fmtutil.cnf
mktexfmt: mktexfmt is using the following fmtutil.cnf files (in precedence order):
mktexfmt:   /etc/texmf/web2c/fmtutil.cnf
mktexfmt:   /usr/share/texmf-dist/web2c/fmtutil.cnf
mktexfmt: mktexfmt is using the following fmtutil.cnf file for writing changes:
mktexfmt:   /var/tmp/portage/sci-mathematics/pari-2.13.0/homedir/.texlive/texmf-config/web2c/fmtutil.cnf
mktexfmt: mktexfmt is using the following fmtutil.cnf files (in precedence order):
mktexfmt:   /etc/texmf/web2c/fmtutil.cnf
mktexfmt:   /usr/share/texmf-dist/web2c/fmtutil.cnf
mktexfmt: mktexfmt is using the following fmtutil.cnf file for writing changes:
mktexfmt:   /var/tmp/portage/sci-mathematics/pari-2.13.0/homedir/.texlive/texmf-config/web2c/fmtutil.cnf
mktexfmt [INFO]: mktexfmt [INFO]: writing formats under /var/tmp/portage/sci-mathematics/pari-2.13.0/homedir/.texlive/texmf-var/web2c
writing formats under /var/tmp/portage/sci-mathematics/pari-2.13.0/homedir/.texlive/texmf-var/web2c
mktexfmt [INFO]: writing formats under /var/tmp/portage/sci-mathematics/pari-2.13.0/homedir/.texlive/texmf-var/web2c
mktexfmt [INFO]: did not find entry for byfmt=pdftex, skipped
mktexfmt [INFO]: did not find entry for byfmt=pdftex, skipped
mktexfmt [INFO]: not selected formats: 33
mktexfmt [INFO]: total formats: 33
mktexfmt [INFO]: did not find entry for byfmt=pdftex, skipped
mktexfmt [INFO]: not selected formats: 33
mktexfmt [INFO]: total formats: 33
mktexfmt [INFO]: exiting with status 0
mktexfmt [INFO]: exiting with status 0
mktexfmt [INFO]: not selected formats: 33
mktexfmt [INFO]: total formats: 33
mktexfmt [INFO]: exiting with status 0
I can't find the format file `pdftex.fmt'!
I can't find the format file `pdftex.fmt'!
make[1]: *** [Makefile:62: parallel.pdf] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: *** [Makefile:40: libpari.pdf] Error 1
I can't find the format file `pdftex.fmt'!
make[1]: *** [Makefile:51: develop.pdf] Error 1
mktexfmt: mktexfmt is using the following fmtutil.cnf files (in precedence order):
mktexfmt:   /etc/texmf/web2c/fmtutil.cnf
mktexfmt:   /usr/share/texmf-dist/web2c/fmtutil.cnf
mktexfmt: mktexfmt is using the following fmtutil.cnf file for writing changes:
mktexfmt:   /var/tmp/portage/sci-mathematics/pari-2.13.0/homedir/.texlive/texmf-config/web2c/fmtutil.cnf
mktexfmt [INFO]: writing formats under /var/tmp/portage/sci-mathematics/pari-2.13.0/homedir/.texlive/texmf-var/web2c
mktexfmt [INFO]: did not find entry for byfmt=pdftex, skipped
mktexfmt [INFO]: not selected formats: 33
mktexfmt [INFO]: total formats: 33
mktexfmt [INFO]: exiting with status 0
I can't find the format file `pdftex.fmt'!
make[1]: *** [Makefile:73: tutorial.pdf] Error 1
make[1]: Leaving directory '/var/tmp/portage/sci-mathematics/pari-2.13.0/work/pari-2.13.0/doc'
make: *** [Makefile:54: docpdf] Error 2
 * ERROR: sci-mathematics/pari-2.13.0::gentoo failed (compile phase):
 *   emake failed

@orlitzky
Copy link
Contributor

pari 2.13 does not get installed due to a weird texlive error. No idea why it is trying to do something with pdftex.fmt

Someone else reported a problem with the documentation build in pari a while ago, but neither Francois nor I could reproduce it. If you ever figure out what's going wrong with make docpdf, please let us know so we can fix the ebuild.

@dimpase
Copy link

dimpase commented Nov 23, 2020

pdftex is just broken on my system for no good reason

$ pdftex develop.tex 
This is pdfTeX, Version 3.14159265-2.6-1.40.21 (TeX Live 2020 Gentoo Linux) (preloaded format=pdftex)
 restricted \write18 enabled.

kpathsea: Running mktexfmt pdftex.fmt
mktexfmt: mktexfmt is using the following fmtutil.cnf files (in precedence order):
mktexfmt:   /etc/texmf/web2c/fmtutil.cnf
mktexfmt:   /usr/share/texmf-dist/web2c/fmtutil.cnf
mktexfmt: mktexfmt is using the following fmtutil.cnf file for writing changes:
mktexfmt:   /home/dima/.texlive/texmf-config/web2c/fmtutil.cnf
mktexfmt [INFO]: writing formats under /home/dima/.texlive/texmf-var/web2c
mktexfmt [INFO]: did not find entry for byfmt=pdftex, skipped
mktexfmt [INFO]: not selected formats: 33
mktexfmt [INFO]: total formats: 33
mktexfmt [INFO]: exiting with status 0
I can't find the format file `pdftex.fmt'!

@dimpase
Copy link

dimpase commented Nov 23, 2020

it all works if I use xetex and xelatex instead.

@dimpase
Copy link

dimpase commented Nov 23, 2020

basically you can patch the doc/Makefile in pari to use xe(la)tex and be done with.

@dimpase
Copy link

dimpase commented Nov 23, 2020

https://wiki.gentoo.org/wiki/TeX_Live#eselect_pdftex

makes me wonder whether special steps are needed to enable/disable pdftex

@orlitzky
Copy link
Contributor

https://wiki.gentoo.org/wiki/TeX_Live#eselect_pdftex

makes me wonder whether special steps are needed to enable/disable pdftex

No, that page is just outdated... eselect-pdftex was removed back in August 2020, and was obsolete before then. There are various hacks floating around to fix fmt issues, but I don't really understand this stuff. Regardless.. if you haven't done anything to break your latex installation, pdftex should just work. (I don't think pari's documentation requires xetex, but I could be wrong.) Maybe @Zlogene can suggest something?

@dimpase
Copy link

dimpase commented Nov 23, 2020

I enabled xetex as it has better utf-8 support, I
don't recall how; perhaps they don't function together?

@dimpase
Copy link

dimpase commented Nov 23, 2020

OK, I reinstalled dev-texlive/texlive-basic (the package that contains pdftex.fmt) and now pari 2.13 builds.

@strogdon
Copy link
Contributor Author

Doctesting vanilla 9.3.beta1 using all available Gentoo components as

sage -tp 5 --long --all --memlimit=0

gives the failures

sage -t --long --warn-long 223.1 --random-seed=0 src/sage/doctest/test.py  # 29 doctests failed
sage -t --long --warn-long 223.1 --random-seed=0 src/sage/tests/cmdline.py  # 7 doctests failed

The failures in src/sage/doctest/test.py are all memory-related failures involving subprocess.call. The failures in src/sage/tests/cmdline.py are perhaps formatting issues?

@dimpase
Copy link

dimpase commented Nov 23, 2020

upgrade to Pari 2.13 does not change the picture.

@dimpase
Copy link

dimpase commented Nov 23, 2020

in the 1st coredump in src/sage/doctest/test.py I see openblas, so it could be the Gentoo openblas upgrade to 3.12 (Sage is still on 3.9) that is responsible for this mess.

@strogdon
Copy link
Contributor Author

On gentoo prefix, where I have symbols compiled in, I have

    Cython backtrace
    ----------------
    #0  0x00007f98d795e4e0 in __GI___wait4 () at /storage/strogdon/gentoo-rap/var/tmp/portage/sys-libs/glibc-2.32-r2/work/glibc-2.32/posix/../sysdeps/unix/sysv/linux/wait4.c:30
    #1  0x00007f98d2417ab1 in print_enhanced_backtrace () at /storage/strogdon/gentoo-rap/var/tmp/portage/dev-python/cysignals-1.10.2/work/cysignals-1.10.2/build/src/cysignals/implementation.c:563
    #2  0x00007f98d2417c4d in sigdie () at /storage/strogdon/gentoo-rap/var/tmp/portage/dev-python/cysignals-1.10.2/work/cysignals-1.10.2/build/src/cysignals/implementation.c:589
    #3  0x00007f98d2416fcb in sigdie_for_sig () at /storage/strogdon/gentoo-rap/var/tmp/portage/dev-python/cysignals-1.10.2/work/cysignals-1.10.2/build/src/cysignals/implementation.c:164
    #4  0x00007f98d2417172 in cysigs_signal_handler () at /storage/strogdon/gentoo-rap/var/tmp/portage/dev-python/cysignals-1.10.2/work/cysignals-1.10.2/build/src/cysignals/implementation.c:262
    #5  0x00007f98d821f5b0 in __restore_rt ()
    #6  0x0000000000000000 in ?? ()
    #7  0x00007f98c67a4b32 in blas_memory_alloc () at /storage/strogdon/gentoo-rap/var/tmp/portage/sci-libs/openblas-0.3.12-r1/work/OpenBLAS-0.3.12/driver/others/memory.c:2793
    #8  0x00007f98c67a5678 in blas_thread_init () at /storage/strogdon/gentoo-rap/var/tmp/portage/sci-libs/openblas-0.3.12-r1/work/OpenBLAS-0.3.12/driver/others/blas_server_omp.c:130
    #9  0x00007f98c67a4f22 in gotoblas_init () at /storage/strogdon/gentoo-rap/var/tmp/portage/sci-libs/openblas-0.3.12-r1/work/OpenBLAS-0.3.12/driver/others/memory.c:3123
    #10 0x00007f98d8212cc0 in call_init () at /storage/strogdon/gentoo-rap/var/tmp/portage/sys-libs/glibc-2.32-r2/work/glibc-2.32/elf/dl-init.c:74
    #11 0x00007f98d8212e13 in call_init () at /storage/strogdon/gentoo-rap/var/tmp/portage/sys-libs/glibc-2.32-r2/work/glibc-2.32/elf/dl-init.c:37
    #12 0x00007f98d8212db0 in _dl_init () at /storage/strogdon/gentoo-rap/var/tmp/portage/sys-libs/glibc-2.32-r2/work/glibc-2.32/elf/dl-init.c:121
    #13 0x00007f98d79c5f20 in __GI__dl_catch_exception () at /storage/strogdon/gentoo-rap/var/tmp/portage/sys-libs/glibc-2.32-r2/work/glibc-2.32/elf/dl-error-skeleton.c:182
    #14 0x00007f98d8216d10 in dl_open_worker () at /storage/strogdon/gentoo-rap/var/tmp/portage/sys-libs/glibc-2.32-r2/work/glibc-2.32/elf/dl-open.c:783
    #15 0x00007f98d79c5f20 in __GI__dl_catch_exception () at /storage/strogdon/gentoo-rap/var/tmp/portage/sys-libs/glibc-2.32-r2/work/glibc-2.32/elf/dl-error-skeleton.c:208

So maybe openblas?

@strogdon
Copy link
Contributor Author

In Gentoo prefix this

sage -t --long ~/usr/lib/python3.8/site-packages/sage/doctest/test.py

now passes with openblas-0.3.9-r1 where I has 29 failures with openblas-0.3.12-1

@dimpase
Copy link

dimpase commented Nov 23, 2020

I can confirm that going back to openblas-0.3.9-r1 in my "vanilla" case fixes the problem.

@kiwifb
Copy link
Collaborator

kiwifb commented Nov 23, 2020

OK, so openblas has historically been playing with cgroups for a while to deal with resource issues. May be something has been switched on. Do both of you, have openmp enabled for openblas?

@kiwifb
Copy link
Collaborator

kiwifb commented Nov 23, 2020

The documentation issue with pari seems to be a migration problem between versions of texlive. Some rebuilding may need to be enforced.

@strogdon
Copy link
Contributor Author

My openblas

[U] sci-libs/openblas
     Available versions:  0.3.9-r1^t 0.3.10^t 0.3.12-r1^t {dynamic eselect-ldso index-64bit openmp pthread +relapack test}
     Installed versions:  0.3.9-r1^t(12:36:42 PM 11/23/2020)(eselect-ldso openmp -dynamic -index-64bit -pthread -test)
     Homepage:            http://xianyi.github.com/OpenBLAS/
     Description:         Optimized BLAS library based on GotoBLAS2

@kiwifb
Copy link
Collaborator

kiwifb commented Nov 23, 2020

OK, that may cause some rebuild. Can I ask you guys to try pthread instead of opemp and whether that behaves better.

@dimpase
Copy link

dimpase commented Nov 23, 2020

this seems to be platform-dependent; on Homebrew one can use its OpenBLAS 3.12 with openmp in Sage just fine.

@kiwifb
Copy link
Collaborator

kiwifb commented Nov 23, 2020

But did you try with 0.3.12 with pthread instead of openmp?

@strogdon
Copy link
Contributor Author

strogdon commented Nov 23, 2020 •
edited

All tests passed for

sage -t --long ~/usr/lib/python3.8/site-packages/sage/doctest/test.py

in prefix with sci-libs/openblas -openmp pthread. The doctest failed miserably with the useflags switched. This was with 0.3.12

@kiwifb
Copy link
Collaborator

kiwifb commented Nov 23, 2020

There may be something subtle at play because of ldso switching. Or something in Gentoo patch set. That would explain that homebrew does work. Although I guess homebrew could have a critical patch :)

@dimpase
Copy link

dimpase commented Nov 23, 2020

https://github.com/Homebrew/homebrew-core/blob/master/Formula/openblas.rb

no patches as far as I can see

@strogdon
Copy link
Contributor Author

strogdon commented Nov 23, 2020 •
edited

openblas-0.3.12-r1 with USE=-openmp pthread on "vanilla"

sage -t --long --warn-long 222.9 --random-seed=0 src/sage/doctest/test.py
**********************************************************************
File "src/sage/doctest/test.py", line 521, in sage.doctest.test
Failed example:
    if system() == "Linux":
        P = subprocess.Popen(["sage", "-t", "--warn-long", "0", "--random-seed=0", "--memlimit=2000", "memlimit.rst"], stdout=subprocess.PIPE, **kwds)
        out, err = P.communicate()
        ok = ("MemoryError: failed to allocate" in bytes_to_str(out))
Expected nothing
Got:
    Process DocTestWorker-1:
    Traceback (most recent call last):
      File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
        self.run()
      File "/local/sage-git/sage/local/lib/python3.8/site-packages/sage/doctest/forker.py", line 2191, in run
        task(self.options, self.outtmpfile, msgpipe, self.result_queue)
      File "/local/sage-git/sage/local/lib/python3.8/site-packages/sage/doctest/forker.py", line 2541, in __call__
        result_queue.put(result, False)
      File "/usr/lib/python3.8/multiprocessing/queues.py", line 88, in put
        self._start_thread()
      File "/usr/lib/python3.8/multiprocessing/queues.py", line 173, in _start_thread
        self._thread.start()
      File "/usr/lib/python3.8/threading.py", line 852, in start
        _start_new_thread(self._bootstrap, ())
    RuntimeError: can't start new thread
**********************************************************************
1 item had failures:
   1 of  58 in sage.doctest.test
    [57 tests, 1 failure, 175.42 s]

So, maybe there is a hardware/platform component. I believe if the memlimit were increased in the doctest things would be fine.

@strogdon
Copy link
Contributor Author

strogdon commented Nov 23, 2020 via email

@dimpase
Copy link

dimpase commented Nov 23, 2020

Do you have libgomp linked into your build of openblas on homebrew?

yes. It comes from gcc 10, by the way.

I wonder whether your Gentoo tries come with gcc 10, or a "normal" gcc 8 from the system?
(it's gcc 8 in my case).

@kiwifb
Copy link
Collaborator

kiwifb commented Nov 23, 2020

I am all on gcc 10 personally. I usually upgrade early to figure problems out.

@strogdon
Copy link
Contributor Author

On Gentoo, I'm using gcc-9.3.0-r1 and on gentoo Prefix gcc-10.2.0-r3.

This was referenced Dec 20, 2020
Closed
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@kiwifb @strogdon @dimpase @orlitzky