curious doctest failure #685

strogdon · 2022-03-16T23:15:42Z

On vanilla (9.6.beta5) I have

sage -t --long --warn-long 107.7 --random-seed=251680023171002497689895805913335591705 src/sage/structure/coerce_actions.pyx
**********************************************************************
File "src/sage/structure/coerce_actions.pyx", line 773, in sage.structure.coerce_actions.IntegerMulAction._act_
Failed example:
    alarm(0.1); (2^(10^7)) * P
Expected:
    Traceback (most recent call last):
    ...
    AlarmInterrupt
Got:
    (0 : 1 : 0)

which is resolved by https://trac.sagemath.org/ticket/33496. However, the doctest passes on s-o-g as is. Is my machine slower when running s-o-g or something else? I did notice that the vanilla coerce_actions.cpython-310-x86_64-linux-gnu.so was built with threads. The s-o-g build does not use threads.

The text was updated successfully, but these errors were encountered:

kiwifb · 2022-03-16T23:19:09Z

Threading has a lot to answer for, especially in transient failures. In this case, I would be most interested to figure why one has threads and not the other.

strogdon · 2022-03-16T23:23:31Z

For info,
Vanilla:

ldd -r venv/lib/python3.10/site-packages/sage/structure/coerce_actions.cpython-310-x86_64-linux-gnu.so 
        linux-vdso.so.1 (0x00007fff42d72000)
        libgmp.so.10 => /usr/lib64/libgmp.so.10 (0x00007f308e020000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f308e000000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f308de32000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f308e0e3000)

Gentoo:

ldd -r /usr/lib/python3.10/site-packages/sage/structure/coerce_actions.cpython-310-x86_64-linux-gnu.so 
        linux-vdso.so.1 (0x00007ffe2178c000)
        libgmp.so.10 => /usr/lib64/libgmp.so.10 (0x00007febe3ed2000)
        libc.so.6 => /lib64/libc.so.6 (0x00007febe3d04000)
        /lib64/ld-linux-x86-64.so.2 (0x00007febe3f95000)

kiwifb · 2022-03-16T23:25:48Z

Very strange. I was expecting some other libraries to be involved. readelf -d of the same?

strogdon · 2022-03-16T23:32:46Z

There are other Python symbols that are undefined - I'm assuming they're picked up at runtime?

Vanilla:

Dynamic section at offset 0x28cd0 contains 27 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libgmp.so.10]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000001d (RUNPATH)            Library runpath: [/local/sage-git/sage/local/lib]
 0x000000000000000c (INIT)               0x5000
 0x000000000000000d (FINI)               0x22274
 0x0000000000000019 (INIT_ARRAY)         0x29cc0
 0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x29cc8
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x200
 0x0000000000000005 (STRTAB)             0xfb8
 0x0000000000000006 (SYMTAB)             0x280
 0x000000000000000a (STRSZ)              3028 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000003 (PLTGOT)             0x2a000
 0x0000000000000002 (PLTRELSZ)           2376 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x41e0
 0x0000000000000007 (RELA)               0x1cd8
 0x0000000000000008 (RELASZ)             9480 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffe (VERNEED)            0x1ca8
 0x000000006fffffff (VERNEEDNUM)         1
 0x000000006ffffff0 (VERSYM)             0x1b8c
 0x000000006ffffff9 (RELACOUNT)          355
 0x0000000000000000 (NULL)               0x0

Gentoo:

Dynamic section at offset 0x28cf0 contains 25 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libgmp.so.10]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000c (INIT)               0x5000
 0x000000000000000d (FINI)               0x22478
 0x0000000000000019 (INIT_ARRAY)         0x29ce0
 0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x29ce8
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x200
 0x0000000000000005 (STRTAB)             0xfb8
 0x0000000000000006 (SYMTAB)             0x280
 0x000000000000000a (STRSZ)              2981 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000003 (PLTGOT)             0x2a000
 0x0000000000000002 (PLTRELSZ)           2376 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x41b0
 0x0000000000000007 (RELA)               0x1ca8
 0x0000000000000008 (RELASZ)             9480 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffe (VERNEED)            0x1c78
 0x000000006fffffff (VERNEEDNUM)         1
 0x000000006ffffff0 (VERSYM)             0x1b5e
 0x000000006ffffff9 (RELACOUNT)          355
 0x0000000000000000 (NULL)               0x0

kiwifb · 2022-03-16T23:34:55Z

I think the linking to pthread is bogus and comes from not using --as-needed during linking. That being said, just loading the library may have some impact.

strogdon · 2022-03-16T23:37:38Z

However when I build vanilla I have exported

export MAKE="make -j9" && export LDFLAGS="-Wl,-O1 -Wl,--as-needed"

kiwifb · 2022-03-16T23:37:44Z

And yes, python symbols are provided by python when it loads the module. It would be an issue if something other than python (or cpython more precisely) was loading them, which is an issue with some gap libraries.

kiwifb · 2022-03-16T23:55:40Z

I have run out of ideas there. The next step would be linking flags coming from python. But both are using the same python as far as I can see. It also suggests that the Gentoo version is underlinked with regards to pthreads, but it would show in the output from ldd -r.

Would have to inspect the linking line in the log for the vanilla version. I can use my own copy for Gentoo.

strogdon · 2022-03-17T00:34:46Z

I must say the build log is not very informative. All I have is

[sagelib-9.6.beta5] [ 43/531] Cythonizing sage/structure/coerce_actions.pyx

I can't find any linking info.

kiwifb · 2022-03-17T00:35:59Z

Darn, same as gentoo log. We have to find a way to restore verbosity.

kiwifb · 2022-03-17T01:30:25Z

That's funny, I get the failure when I do testing from inside the ebuild (something I am currently working on thanks to new bits of distutils-r1 see 1468853) and therefore test on the original source file but not when I do a separate doctest with the installed sage.

strogdon · 2022-03-18T17:05:06Z

I do have an old s-o-g build log from about a year ago (May 2021) in Prefix

[ 62/526] x86_64-pc-linux-gnu-gcc -O2 -pipe -march=native -O2 -pipe -fPIC -I./sage/cpython -I./sage/rings -I/storage/strogdon/gentoo-rap/usr/lib/python3.9/site-packages/cysignals -I/storage/strogdon/gentoo-rap/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9 -I/storage/strogdon/gentoo-rap/usr/include/python3.9 -I/storage/strogdon/gentoo-rap/usr/lib/python3.9/site-packages/numpy/core/include -I/storage/strogdon/gentoo-rap/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized -I/storage/strogdon/gentoo-rap/usr/include/python3.9 -c /storage/strogdon/gentoo-rap/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/structure/coerce_actions.c -o /storage/strogdon/gentoo-rap/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/temp.linux-x86_64-3.9/storage/strogdon/gentoo-rap/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/structure/coerce_actions.o -fno-strict-aliasing -DCYTHON_CLINE_IN_TRACEBACK=1 -std=c99
x86_64-pc-linux-gnu-gcc -shared -Wl,-O1 -Wl,--as-needed -O2 -pipe -march=native -O2 -pipe /storage/strogdon/gentoo-rap/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/temp.linux-x86_64-3.9/storage/strogdon/gentoo-rap/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/cythonized/sage/structure/coerce_actions.o -L/storage/strogdon/gentoo-rap/usr/lib64 -lgmp -o /storage/strogdon/gentoo-rap/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_9/build/lib/sage/structure/coerce_actions.cpython-39-x86_64-linux-gnu.so -lpari

The build failed for some reason.

strogdon · 2022-03-19T18:57:33Z

dev-python/memory_allocator also does not have linking info when building. There is only

[1/2] Cythonizing memory_allocator/memory_allocator.pyx
[2/2] Cythonizing memory_allocator/test.pyx

Perhaps this would be easier to debug since there is only one setup.py. I'm guessing this is a cythonize thing. But how to turn on the linking info?

strogdon · 2022-03-21T21:33:30Z

A poor mans pthread_find.sh

#! /bin/bash

#for i in `find venv/lib/python3.10/site-packages/sage/ -name "*gnu.so"`
for i in `find /usr/lib/python3.10/site-packages/sage/ -name "*gnu.so"`
do
#       if [[ -n `ldd -r $i | grep pthread` ]]
        if [[ -n `readelf -d $i | grep pthread` ]]
        then
                echo $i
#               ldd -r $i | grep pthread
#               readelf -d $i | grep pthread
        fi
done

On vanilla I did a make distclean and rebuilt sage. pthread_find.sh | wc -l gives 237 libraries that are linked against libpthread and of those 237 libraries 155 indicate libpthread is NEEDED. On Gentoo I get that 80 libraries are linked against libpthread and of those libraries only 2 indicate libpthread is NEEDED.

kiwifb · 2022-03-21T21:35:58Z

Yes, I am not convinced at all that -Wl,--as-needed makes it through on vanilla. But we cannot see because nothing is verbose.

kiwifb · 2022-03-22T01:17:01Z

I have verbosity. sage_setup needs to be patched. There are probably other place to put it but it didn't work from sage's main setup.py file. I had to plumb it in sage_setup/command/sage_build_ext.py and add

log.set_verbosity(1)

in the function build_extension (note that extension is singular, there is another methods build_extensions plural). And after that I have glorious verbosity with the prefix INFO:.

strogdon · 2022-03-22T01:27:57Z

Good news. I also added

diff --git a/src/sage_setup/command/sage_build_cython.py b/src/sage_setup/command/sage_build_cython.py
index 5c384f6dba..99421f32b6 100644
--- a/src/sage_setup/command/sage_build_cython.py
+++ b/src/sage_setup/command/sage_build_cython.py
@@ -43,7 +43,7 @@ if ntl_prefix:
 # setting in Cython which causes some overhead every time an exception
 # is raised).
 extra_compile_args = ["-fno-strict-aliasing", "-DCYTHON_CLINE_IN_TRACEBACK=1"]
-extra_link_args = [ ]
+extra_link_args = ["-Wl,-O1 -Wl,--as-needed"]
 
 DEVEL = False
 if DEVEL:

with no change to pthread linking.

kiwifb · 2022-03-22T01:48:16Z

Really not sure where it come from and whether it is really missing in s-o-g. I am having random compile error on my machine these days :( the pattern makes me suspect faulty RAM. I'll have to figure how to test it.

strogdon · 2022-03-22T02:05:31Z

Well, I get verbose output on vanilla to a point until in building sagelib I see

Traceback (most recent call last):
  File "/local/sage-git/sage/pkgs/sagemath-standard/setup.py", line 101, in <module>
    code = setup(
  File "/local/sage-git/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python3.10/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.10/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.10/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/local/sage-git/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/sage_setup/command/sage_build.py", line 37, in run
    build.run(self)
  File "/usr/lib/python3.10/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/usr/lib/python3.10/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.10/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/local/sage-git/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/sage_setup/command/sage_build_ext.py", line 26, in run
    build_ext.run(self)
  File "/local/sage-git/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 79, in run
    _build_ext.run(self)
  File "/local/sage-git/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
    _build_ext.build_ext.run(self)
  File "/usr/lib/python3.10/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/local/sage-git/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/sage_setup/command/sage_build_ext.py", line 86, in build_extensions
    execute_list_of_commands(compile_commands)
  File "/local/sage-git/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/sage_setup/run_parallel.py", line 133, in execute_list_of_commands
    execute_list_of_commands_in_parallel(command_list, nthreads)
  File "/local/sage-git/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/sage_setup/run_parallel.py", line 81, in execute_list_of_commands_in_parallel
    result = pool.map_async(apply_func_progress, command_list, 1).get(99999)
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 771, in get
    raise self._value
AttributeError: module 'distutils.log' has no attribute 'set_verbose'

The structure/coerce_actions.pyx did compile

[346/531] gcc -g -O2 -fPIC -I./sage/cpython -I/local/sage-git/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/cysignals -I./sage/rings -I/local/sage-git/sage/pkgs/sagemath-standard -I/local/sage-git/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/numpy/core/include -I/usr/include/python3.10 -Ibuild/cythonized -I/local/sage-git/sage/local/var/lib/sage/venv-python3.10/include -I/usr/include/python3.10 -c build/cythonized/sage/structure/coerce_actions.c -o build/temp.linux-x86_64-3.10/build/cythonized/sage/structure/coerce_actions.o -fno-strict-
aliasing -DCYTHON_CLINE_IN_TRACEBACK=1 -std=c99

and

[351/531] gcc -g -O2 -fPIC -I./sage/cpython -I/local/sage-git/sage/pkgs/sagemath-standard -I/local/sage-git/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/numpy/core/include -I/usr/include/python3.10 -Ibuild/cythonized -I/local/sage-git/sage/local/var/lib/sage/venv-python3.10/include -I/usr/include/python3.10 -c build/cythonized/sage/structure/list_clone_timings_cy.c -o build/temp.linux-x86_64-3.10/build/cythonized/sage/structure/list_clone_timings_cy.o -fno-strict-aliasing -DCYTHON_CLINE_IN_TRACEBACK=1 -std=c99
x86_64-pc-linux-gnu-gcc -pthread -shared -Wl,-rpath-link,/local/sage-git/sage/local/lib -L/local/sage-git/sage/local/lib -Wl,-rpath,/local/sage-git/sage/local/lib -Wl,-rpath-link,/local/sage-git/sage/local/lib -L/local/sage-git/sage/local/lib -Wl,-rpath,/local/sage-git/sage/local/lib -Wl,-O1 -Wl,--as-needed -g -O2 build/temp.linux-x86_64-3.10/build/cythonized/sage/structure/coerce_actions.o -L/usr/lib64 -lgmp -o build/lib.linux-x86_64-3.10/sage/structure/coerce_actions.cpython-310-x86_64-linux-gnu.so -lpari

The as-needed is there as well as -pthread.

strogdon · 2022-03-22T02:13:00Z

The [351/531] should not be in the above post. The subsequent line is OK.

strogdon · 2022-03-22T02:21:36Z

Trying again, I had log.set_verbose(1) instead of log.set_verbosity(1).

strogdon · 2022-03-22T03:37:43Z

Success! The above compile/linking is correct for vanilla. However, I see no INFO: on vanilla. On Gentoo

INFO: x86_64-pc-linux-gnu-gcc -shared -Wl,-O1 -Wl,--as-needed -march=native -O2 -pipe /var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_10/temp.linux-x86_64-3.10/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_10/cythonized/sage/structure/coerce_actions.o -L/usr/lib64 -lgmp -o /var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_10/lib/sage/structure/coerce_actions.cpython-310-x86_64-linux-gnu.so -lpari

There is no -pthread and line that compiles coerce_actions.c -> coerce_actions.o does not appear in the build.log. It also appears that the vanilla build skips the Gentoo CFLAGS, i.e.

COMMON_FLAGS="-march=native -O2 -pipe"
CFLAGS="${COMMON_FLAGS}"

strogdon · 2022-03-22T04:58:39Z

Perhaps the INFO: is because on vanilla setuptools-59.8.0 is used but on Gentoo I'm using
setuptools-60.9.2.

strogdon · 2022-03-22T18:03:33Z

The INFO: tag doesn't seem to originate with setuptools. I Installed the vanilla version on Gentoo and the tag is still present. There are the tags INFO: C compiler:, INFO: compile options:, extra options: and INFO: x86_64-pc-linux-gnu-gcc:. It's not clear, since the build is in parallel, which tags go together. Unless some of the tags do not change. The INFO: tag associated with creating shared objects does appear to be self-contained.

strogdon · 2022-03-27T18:30:31Z

Vanilla 9.6.beta6 is available. I have some odd verbose results. Now this is an incremental build from beta5. I no longer get verbose results like

[346/531] gcc -g -O2 -fPIC -I./sage/cpython -I/local/sage-git/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/cysignals -I./sage/rings -I/local/sage-git/sage/pkgs/sagemath-standard -I/local/sage-git/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/numpy/core/include -I/usr/include/python3.10 -Ibuild/cythonized -I/local/sage-git/sage/local/var/lib/sage/venv-python3.10/include -I/usr/include/python3.10 -c build/cythonized/sage/structure/coerce_actions.c -o build/temp.linux-x86_64-3.10/build/cythonized/sage/structure/coerce_actions.o -fno-strict-
aliasing -DCYTHON_CLINE_IN_TRACEBACK=1 -std=c99

and

x86_64-pc-linux-gnu-gcc -pthread -shared -Wl,-rpath-link,/local/sage-git/sage/local/lib -L/local/sage-git/sage/local/lib -Wl,-rpath,/local/sage-git/sage/local/lib -Wl,-rpath-link,/local/sage-git/sage/local/lib -L/local/sage-git/sage/local/lib -Wl,-rpath,/local/sage-git/sage/local/lib -Wl,-O1 -Wl,--as-needed -g -O2 build/temp.linux-x86_64-3.10/build/cythonized/sage/structure/coerce_actions.o -L/usr/lib64 -lgmp -o build/lib.linux-x86_64-3.10/sage/structure/coerce_actions.cpython-310-x86_64-linux-gnu.so -lpari

but results similar to Gentoo output without the INFO: tag, i.e.

[68/95] C compiler: gcc -g -O2 -fPIC

creating build/temp.linux-x86_64-3.10/build/cythonized/sage/structure
compile options: '-I./sage/cpython -I./sage/rings -I/local/sage-git/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/cysignals -I/local/sage-git/sage/pkgs/sagemath-standard -I/local/sage-git/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/numpy/core/include -I/usr/include/python3.10 -Ibuild/cythonized -I/local/sage-git/sage/local/var/lib/sage/venv-python3.10/include -I/usr/include/python3.10 -c'
extra options: '-fno-strict-aliasing -DCYTHON_CLINE_IN_TRACEBACK=1 -std=c99'
gcc: build/cythonized/sage/structure/coerce_actions.c

and

x86_64-pc-linux-gnu-gcc -pthread -shared -Wl,-rpath-link,/local/sage-git/sage/local/lib -L/local/sage-git/sage/local/lib -Wl,-rpath,/local/sage-git/sage/local/lib -Wl,-rpath-link,/local/sage-git/sage/local/lib -L/local/sage-git/sage/local/lib -Wl,-rpath,/local/sage-git/sage/local/lib -Wl,-O1 -Wl,--as-needed -g -O2 build/temp.linux-x86_64-3.10/build/cythonized/sage/structure/coerce_actions.o -L/usr/lib64 -lgmp -o build/lib.linux-x86_64-3.10/sage/structure/coerce_actions.cpython-310-x86_64-linux-gnu.so -lpari

Not sure what is controlling this different output. The only new pkgs are pythran-0.11.0 and scipy-1.7.3.

kiwifb · 2022-03-27T19:42:40Z

I just merged the vbraun branch into master. s-o-g is verbose from now on. Anyone with FEATURES=test is in for a shock.

kiwifb · 2022-06-21T00:32:06Z

For the record verbosity is off again. My current patch appears to be useless now. Not sure what caused verbosity to switch off.

strogdon · 2022-06-21T06:08:27Z

Do you know whether this started with 9.7.beta3?

kiwifb · 2022-06-21T07:53:19Z

No this happened earlier, I just didn't stop to put a note about it. I may have noticed around 9.7.beta1.

strogdon · 2022-06-26T03:50:12Z

ebuild sage-9999.ebuild prepare, then from the portage work folder, edit src/sage_setup/command/sage_build_ext.py and apply your sage_setup-9.6-verbosity.patch and then ebuild sage-9999.ebuild compile gives verbose output. So, perhaps not looking for the system sage_setup?

kiwifb · 2022-06-26T04:27:33Z

Let me check what I do with sage_setup in the sage ebuild these days...

kiwifb · 2022-06-26T04:30:46Z

Indeed, for a while I was moving sage_setup out of the way so it wouldn't be used. But it caused issue with testing. Good spotting. I will just chuck it out as I have decided on a different testing strategy for now.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

curious doctest failure #685

curious doctest failure #685

strogdon commented Mar 16, 2022

kiwifb commented Mar 16, 2022

strogdon commented Mar 16, 2022

kiwifb commented Mar 16, 2022

strogdon commented Mar 16, 2022

kiwifb commented Mar 16, 2022

strogdon commented Mar 16, 2022

kiwifb commented Mar 16, 2022

kiwifb commented Mar 16, 2022

strogdon commented Mar 17, 2022

kiwifb commented Mar 17, 2022

kiwifb commented Mar 17, 2022

strogdon commented Mar 18, 2022

strogdon commented Mar 19, 2022

strogdon commented Mar 21, 2022

kiwifb commented Mar 21, 2022

kiwifb commented Mar 22, 2022

strogdon commented Mar 22, 2022

kiwifb commented Mar 22, 2022

strogdon commented Mar 22, 2022

strogdon commented Mar 22, 2022

strogdon commented Mar 22, 2022

strogdon commented Mar 22, 2022 •

edited

strogdon commented Mar 22, 2022 •

edited

strogdon commented Mar 22, 2022 •

edited

strogdon commented Mar 27, 2022

kiwifb commented Mar 27, 2022

kiwifb commented Jun 21, 2022

strogdon commented Jun 21, 2022

kiwifb commented Jun 21, 2022

strogdon commented Jun 26, 2022

kiwifb commented Jun 26, 2022

kiwifb commented Jun 26, 2022

curious doctest failure #685

curious doctest failure #685

Comments

strogdon commented Mar 16, 2022

kiwifb commented Mar 16, 2022

strogdon commented Mar 16, 2022

kiwifb commented Mar 16, 2022

strogdon commented Mar 16, 2022

kiwifb commented Mar 16, 2022

strogdon commented Mar 16, 2022

kiwifb commented Mar 16, 2022

kiwifb commented Mar 16, 2022

strogdon commented Mar 17, 2022

kiwifb commented Mar 17, 2022

kiwifb commented Mar 17, 2022

strogdon commented Mar 18, 2022

strogdon commented Mar 19, 2022

strogdon commented Mar 21, 2022

kiwifb commented Mar 21, 2022

kiwifb commented Mar 22, 2022

strogdon commented Mar 22, 2022

kiwifb commented Mar 22, 2022

strogdon commented Mar 22, 2022

strogdon commented Mar 22, 2022

strogdon commented Mar 22, 2022

strogdon commented Mar 22, 2022 • edited

strogdon commented Mar 22, 2022 • edited

strogdon commented Mar 22, 2022 • edited

strogdon commented Mar 27, 2022

kiwifb commented Mar 27, 2022

kiwifb commented Jun 21, 2022

strogdon commented Jun 21, 2022

kiwifb commented Jun 21, 2022

strogdon commented Jun 26, 2022

kiwifb commented Jun 26, 2022

kiwifb commented Jun 26, 2022

strogdon commented Mar 22, 2022 •

edited

strogdon commented Mar 22, 2022 •

edited

strogdon commented Mar 22, 2022 •

edited