NumPy 1.21 support #7483

stuartarchibald · 2021-10-14T08:37:50Z

This PR:

adds NumPy 1.21 to the public CI build matrix. NumPy 1.17 is dropped (supported ended on July 26th, 2021 as per NEP=029 https://numpy.org/neps/nep-0029-deprecation_policy.html).
Fixes up conda recipes, setup.py, __init__.py and docs for 1.21.
Pulls in the ufunc C code patch (df8442f) from BUG: Fastcall fix ufunc methods (NumPy 1.21 support) #7449 (with thanks to @seberg).

As title.

As title. Also updates python version that was incorrect

As title.

This is a start to support NumPy 1.21 use of FASTCALL methods in ufuncs. There are probably better ways to achieve this support. NumPy further uses `tp_vectorcall_offset` on newer versions. This PR does not add support for this (meaning that a DUfunc will be unnecessarily slow on NumPy >=1.21 when kwargs are used).

gmarkall · 2021-10-14T09:38:05Z

Will this close #7176?

stuartarchibald · 2021-10-14T09:39:36Z

Will this close #7176?

Not sure yet. I've got this patch running all tests locally to see if there's any more failures (there's usually a few floating point issues with a NumPy upgrade but we might be lucky!).

stuartarchibald · 2021-10-14T10:24:58Z

Testing locally on linux x86_64:

Run 1:

..Fatal Python error: Aborted

Current thread 0x00007f4c4cf88740 (most recent call first):
  File "<path>/numba/numba/tests/support.py", line 785 in tearDown
  File "<pypath>lib/python3.9/unittest/case.py", line 553 in _callTearDown
  File "<pypath>lib/python3.9/unittest/case.py", line 595 in run
  File "<pypath>lib/python3.9/unittest/case.py", line 651 in __call__
  File "<path>/numba/numba/testing/main.py", line 664 in __call__
  File "<pypath>lib/python3.9/multiprocessing/pool.py", line 125 in worker
  File "<pypath>lib/python3.9/multiprocessing/process.py", line 108 in run
  File "<pypath>lib/python3.9/multiprocessing/process.py", line 315 in _bootstrap
  File "<pypath>lib/python3.9/multiprocessing/popen_fork.py", line 71 in _launch
  File "<pypath>lib/python3.9/multiprocessing/popen_fork.py", line 19 in __init__
  File "<pypath>lib/python3.9/multiprocessing/context.py", line 277 in _Popen
  File "<pypath>lib/python3.9/multiprocessing/process.py", line 121 in start
  File "<pypath>lib/python3.9/multiprocessing/pool.py", line 326 in _repopulate_pool_static
  File "<pypath>lib/python3.9/multiprocessing/pool.py", line 303 in _repopulate_pool
  File "<pypath>lib/python3.9/multiprocessing/pool.py", line 212 in __init__
  File "<pypath>lib/python3.9/multiprocessing/context.py", line 119 in Pool
  File "<path>/numba/numba/testing/main.py", line 741 in _run_inner
  File "<pypath>lib/python3.9/unittest/runner.py", line 176 in run
  File "<path>/numba/numba/testing/main.py", line 795 in run
  File "<pypath>lib/python3.9/unittest/main.py", line 271 in runTests
  File "<path>/numba/numba/testing/main.py", line 325 in run_tests_real
  File "<path>/numba/numba/testing/main.py", line 340 in runTests
  File "<pypath>lib/python3.9/unittest/main.py", line 101 in __init__
  File "<path>/numba/numba/testing/main.py", line 168 in __init__
  File "<path>/numba/numba/testing/__init__.py", line 54 in run_tests
  File "<path>/numba/numba/testing/_runtests.py", line 25 in _main
  File "<path>/numba/numba/runtests.py", line 9 in <module>
  File "<pypath>lib/python3.9/runpy.py", line 87 in _run_code
  File "<pypath>lib/python3.9/runpy.py", line 213 in run_module
  File "<path>/numba/./runtests.py", line 16 in <module>
....Traceback (most recent call last):
  File "<path>/numba/./runtests.py", line 16, in <module>
    runpy.run_module('numba.runtests', run_name='__main__')
  File "<pypath>lib/python3.9/runpy.py", line 213, in run_module
    return _run_code(code, {}, init_globals, run_name, mod_spec)
  File "<pypath>lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "<path>/numba/numba/runtests.py", line 9, in <module>
    sys.exit(0 if _main(sys.argv) else 1)
  File "<path>/numba/numba/testing/_runtests.py", line 25, in _main
    return run_tests(argv, defaultTest='numba.tests',
  File "<path>/numba/numba/testing/__init__.py", line 54, in run_tests
    prog = NumbaTestProgram(argv=argv,
  File "<path>/numba/numba/testing/main.py", line 168, in __init__
    super(NumbaTestProgram, self).__init__(*args, **kwargs)
  File "<pypath>lib/python3.9/unittest/main.py", line 101, in __init__
    self.runTests()
  File "<path>/numba/numba/testing/main.py", line 340, in runTests
    run_tests_real()
  File "<path>/numba/numba/testing/main.py", line 325, in run_tests_real
    super(NumbaTestProgram, self).runTests()
  File "<pypath>lib/python3.9/unittest/main.py", line 271, in runTests
    self.result = testRunner.run(self.test)
  File "<path>/numba/numba/testing/main.py", line 795, in run
    return super(ParallelTestRunner, self).run(self._run_inner)
  File "<pypath>lib/python3.9/unittest/runner.py", line 176, in run
    test(result)
  File "<path>/numba/numba/testing/main.py", line 743, in _run_inner
    self._run_parallel_tests(result, pool, child_runner, tests)
  File "<path>/numba/numba/testing/main.py", line 779, in _run_parallel_tests
    raise e
  File "<path>/numba/numba/testing/main.py", line 770, in _run_parallel_tests
    child_result = it.__next__(self.timeout)
  File "<pypath>lib/python3.9/multiprocessing/pool.py", line 865, in next
    raise TimeoutError from None
multiprocessing.context.TimeoutError: Tests didn't finish before timeout (or crashed):
- 'numba.tests.test_ctypes.TestCTypesUseCases.test_storing_voidptr_to_int_array'

Run 2:

======================================================================
ERROR: test_ufunc_attributes (numba.tests.npyufunc.test_ufunc.TestUFuncs)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "<path>/numba/numba/tests/npyufunc/test_ufunc.py", line 74, in test_ufunc_attributes
    self._test_ufunc_attributes(v, a[0], b[0])
  File "<path>/numba/numba/tests/npyufunc/test_ufunc.py", line 58, in _test_ufunc_attributes
    self.assertPreciseEqual(ufunc_reduce(ufunc, a), np.sum(a), msg=info)
  File "<path>/numba/numba/tests/npyufunc/test_ufunc.py", line 35, in ufunc_reduce
    arg = ufunc.reduce(arg)
TypeError: reduce() got an unexpected keyword argument '<class 'tuple'>'

----------------------------------------------------------------------
Ran 10848 tests in 1983.202s

FAILED (errors=1, skipped=729, expected failures=14)

stuartarchibald · 2021-10-14T11:08:12Z

Run 3:

Fatal Python error: Aborted

Current thread 0x00007f9c41256740 (most recent call first):
  File "<path>/numba/numba/tests/support.py", line 785 in tearDown
  File "<pypath>/lib/python3.9/unittest/case.py", line 553 in _callTearDown
  File "<pypath>/lib/python3.9/unittest/case.py", line 595 in run
  File "<pypath>/lib/python3.9/unittest/case.py", line 651 in __call__
  File "<path>/numba/numba/testing/main.py", line 664 in __call__
  File "<pypath>/lib/python3.9/multiprocessing/pool.py", line 125 in worker
  File "<pypath>/lib/python3.9/multiprocessing/process.py", line 108 in run
  File "<pypath>/lib/python3.9/multiprocessing/process.py", line 315 in _bootstrap
  File "<pypath>/lib/python3.9/multiprocessing/popen_fork.py", line 71 in _launch
  File "<pypath>/lib/python3.9/multiprocessing/popen_fork.py", line 19 in __init__
  File "<pypath>/lib/python3.9/multiprocessing/context.py", line 277 in _Popen
  File "<pypath>/lib/python3.9/multiprocessing/process.py", line 121 in start
  File "<pypath>/lib/python3.9/multiprocessing/pool.py", line 326 in _repopulate_pool_static
  File "<pypath>/lib/python3.9/multiprocessing/pool.py", line 303 in _repopulate_pool
  File "<pypath>/lib/python3.9/multiprocessing/pool.py", line 212 in __init__
  File "<pypath>/lib/python3.9/multiprocessing/context.py", line 119 in Pool
  File "<path>/numba/numba/testing/main.py", line 741 in _run_inner
  File "<pypath>/lib/python3.9/unittest/runner.py", line 176 in run
  File "<path>/numba/numba/testing/main.py", line 795 in run
  File "<pypath>/lib/python3.9/unittest/main.py", line 271 in runTests
  File "<path>/numba/numba/testing/main.py", line 325 in run_tests_real
  File "<path>/numba/numba/testing/main.py", line 340 in runTests
  File "<pypath>/lib/python3.9/unittest/main.py", line 101 in __init__
  File "<path>/numba/numba/testing/main.py", line 168 in __init__
  File "<path>/numba/numba/testing/__init__.py", line 54 in run_tests
  File "<path>/numba/numba/testing/_runtests.py", line 25 in _main
  File "<path>/numba/numba/runtests.py", line 9 in <module>
  File "<pypath>/lib/python3.9/runpy.py", line 87 in _run_code
  File "<pypath>/lib/python3.9/runpy.py", line 213 in run_module
  File "<path>/numba/./runtests.py", line 16 in <module>
<path>/numba/numba/core/ir_utils.py:2141: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'x' of function 'TestListComprehension.test_bulk_use_cases.<locals>.list25'.

For more information visit https://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "numba/tests/test_comprehension.py", line 189:

        def list25(x):
        ^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
....Traceback (most recent call last):
  File "<path>/numba/./runtests.py", line 16, in <module>
    runpy.run_module('numba.runtests', run_name='__main__')
  File "<pypath>/lib/python3.9/runpy.py", line 213, in run_module
    return _run_code(code, {}, init_globals, run_name, mod_spec)
  File "<pypath>/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "<path>/numba/numba/runtests.py", line 9, in <module>
    sys.exit(0 if _main(sys.argv) else 1)
  File "<path>/numba/numba/testing/_runtests.py", line 25, in _main
    return run_tests(argv, defaultTest='numba.tests',
  File "<path>/numba/numba/testing/__init__.py", line 54, in run_tests
    prog = NumbaTestProgram(argv=argv,
  File "<path>/numba/numba/testing/main.py", line 168, in __init__
    super(NumbaTestProgram, self).__init__(*args, **kwargs)
  File "<pypath>/lib/python3.9/unittest/main.py", line 101, in __init__
    self.runTests()
  File "<path>/numba/numba/testing/main.py", line 340, in runTests
    run_tests_real()
  File "<path>/numba/numba/testing/main.py", line 325, in run_tests_real
    super(NumbaTestProgram, self).runTests()
  File "<pypath>/lib/python3.9/unittest/main.py", line 271, in runTests
    self.result = testRunner.run(self.test)
  File "<path>/numba/numba/testing/main.py", line 795, in run
    return super(ParallelTestRunner, self).run(self._run_inner)
  File "<pypath>/lib/python3.9/unittest/runner.py", line 176, in run
    test(result)
  File "<path>/numba/numba/testing/main.py", line 743, in _run_inner
    self._run_parallel_tests(result, pool, child_runner, tests)
  File "<path>/numba/numba/testing/main.py", line 779, in _run_parallel_tests
    raise e
  File "<path>/numba/numba/testing/main.py", line 770, in _run_parallel_tests
    child_result = it.__next__(self.timeout)
  File "<pypath>/lib/python3.9/multiprocessing/pool.py", line 865, in next
    raise TimeoutError from None
multiprocessing.context.TimeoutError: Tests didn't finish before timeout (or crashed):
- 'numba.tests.test_ctypes.TestCTypesUseCases.test_storing_voidptr_to_int_array'

gmarkall · 2021-10-14T12:30:46Z

Results on my linux-64 system:

======================================================================
ERROR: test_ufunc_attributes (numba.tests.npyufunc.test_ufunc.TestUFuncs)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gmarkall/numbadev/numba/numba/tests/npyufunc/test_ufunc.py", line 74, in test_ufunc_attributes
    self._test_ufunc_attributes(v, a[0], b[0])
  File "/home/gmarkall/numbadev/numba/numba/tests/npyufunc/test_ufunc.py", line 58, in _test_ufunc_attributes
    self.assertPreciseEqual(ufunc_reduce(ufunc, a), np.sum(a), msg=info)
  File "/home/gmarkall/numbadev/numba/numba/tests/npyufunc/test_ufunc.py", line 35, in ufunc_reduce
    arg = ufunc.reduce(arg)
TypeError: reduce() got an unexpected keyword argument '<class 'tuple'>'

======================================================================
FAIL: test_check_objmode_cache_ndarray (numba.tests.test_extending.TestCachingOverloadObjmode)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gmarkall/numbadev/numba/numba/tests/test_extending.py", line 1860, in test_check_objmode_cache_ndarray
    self.assertEqual(res['exitcode'], 0)
AssertionError: 1 != 0

----------------------------------------------------------------------
Ran 10848 tests in 1442.926s

I didn't have scipy installed so some BLAS tests have not run. Also I saw messages about not having enough cores for some tests (I have 6 physical, 12 HT). Going to install SciPy and do another run.

stuartarchibald · 2021-10-14T13:07:09Z

@gmarkall thanks for trying. I'm surprised there's tests that need more than 6 cores, maybe it's a bug in a test/some logic, if you get a moment and can identify the culprit that'd be very helpful, thanks!

gmarkall · 2021-10-14T14:16:20Z

This time with SciPy installed:

======================================================================
ERROR: test_ufunc_attributes (numba.tests.npyufunc.test_ufunc.TestUFuncs)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gmarkall/numbadev/numba/numba/tests/npyufunc/test_ufunc.py", line 74, in test_ufunc_attributes
    self._test_ufunc_attributes(v, a[0], b[0])
  File "/home/gmarkall/numbadev/numba/numba/tests/npyufunc/test_ufunc.py", line 58, in _test_ufunc_attributes
    self.assertPreciseEqual(ufunc_reduce(ufunc, a), np.sum(a), msg=info)
  File "/home/gmarkall/numbadev/numba/numba/tests/npyufunc/test_ufunc.py", line 35, in ufunc_reduce
    arg = ufunc.reduce(arg)
TypeError: reduce() got an unexpected keyword argument '<class 'tuple'>'

----------------------------------------------------------------------
Ran 10848 tests in 1530.389s

FAILED (errors=1, skipped=633, expected failures=14)

gmarkall · 2021-10-14T14:32:03Z

@gmarkall thanks for trying. I'm surprised there's tests that need more than 6 cores, maybe it's a bug in a test/some logic, if you get a moment and can identify the culprit that'd be very helpful, thanks!

An example:

test_threadmask_across_fork_workqueue_1_threads (numba.tests.test_num_threads.TestNumThreadsBackends) ... skipped 'Not enough CPU cores'

which comes from:

numba/numba/tests/test_num_threads.py

Line 525 in 679ade8

@unittest.skipIf(config.NUMBA_NUM_THREADS < 2, "Not enough CPU cores")

somehow config.NUMBA_NUM_THREADS is getting set to 1 or less in a subprocess.

seberg · 2021-10-14T14:51:32Z

Is this easy to run locally? I am confused why TypeError: reduce() got an unexpected keyword argument '<class 'tuple'>' appears here but CI is happy. It is not that FASTCALL and normal calling convention are mixed up for some reason?

stuartarchibald · 2021-10-14T15:26:41Z

Is this easy to run locally? I am confused why TypeError: reduce() got an unexpected keyword argument '<class 'tuple'>' appears here but CI is happy. It is not that FASTCALL and normal calling convention are mixed up for some reason?

@seberg Thanks for taking a look. It should just be building Numba from this branch and then from the Numba repo root:

./runtests.py numba.tests.npyufunc.test_ufunc.TestUFuncs.test_ufunc_attributes

I presume you've got a local build working already so as to create the original 1.21 PR?

RE:

CI is happy.

Numba's public CI is a bit unusual: https://numba.readthedocs.io/en/stable/developer/contributing.html#continuous-integration-testing, there's a possibility that it just wasn't hit by the right combination of tests/packages/systems/pythons.

seberg · 2021-10-14T16:04:19Z

No, to be honest, I trusted the CI to run it... I will try to have a look later today probably.

stuartarchibald · 2021-10-14T16:14:59Z

Thanks @seberg! Development builds (of e.g. llvmlite) are in the conda channel numba/label/dev.

seberg · 2021-10-14T22:28:55Z

Trivial error, my strcmp call is obvious nonsense and has inverted behaviour.

numba/np/ufunc/_internal.c

As title.

stuartarchibald · 2021-10-15T16:12:11Z

First local test run against 4df2640 passed.

stuartarchibald · 2021-10-15T17:40:35Z

Second local test run against 4df2640 passed.

sklam · 2021-10-21T15:02:42Z

I think all that is left here is a build farm run?

esc · 2021-11-22T18:31:02Z

Build numba_smoketest_cpu_yaml_65 has started

sklam

The changes on NaT looks good as well.

Once NumPy stabilizes on the behavior for NaT comparison, we can come back to update the test. But for now, the changes here is avoiding tests that contains NaT in median.

sklam · 2021-11-22T18:58:27Z

Note: once this passes on the farm, it can be merged.

esc · 2021-11-22T22:42:22Z

Everything not Python 3.10 passed. So we will merge this tonight.

esc · 2021-11-22T22:42:54Z

@sklam @seberg @stuartarchibald @gmarkall thank you all!

esc · 2021-11-22T22:45:22Z

Everything not Python 3.10 passed. So we will merge this tonight.

With the exception of the ppcle64 build, due to missing Numpy 1.21 -- as I just double checked -- @sklam still good to go?

seberg · 2021-11-23T17:38:12Z

One thing to note, is that on Python 3.8+, NumPy also uses tp_vectorcall, since I had first thought this might be backported too, I wanted to be a bit minimal and did not include that.

If keywords are being used, numba will have some extra overhad for e.g. ufunc(x, out=y), implementing fastcall could be nice.

christian-cahig · 2021-12-08T15:37:51Z

Hi, is this supported in Numba 0.54.1?

seibert · 2021-12-08T15:40:10Z

This will be released in Numba 0.55.0, coming soon. (Edit: I should say, a release candidate will be coming soon. Due to the magnitude of changes in 0.55, we're going to give people a long RC period to test it before a final release in January 2022.)

kyoungseoun-chung · 2021-12-08T15:40:14Z

As far as I know, it doesn't.

I think you should install numba by cloning the repository, otherwise, we need to wait for the new version release...

seberg · 2021-12-08T15:43:44Z

I expect the patch applies cleanly to 0.54.x, although I guess that won't help much...

seibert · 2021-12-08T15:47:05Z

If you are using conda, we also automatically build packages whenever a PR is merged, which you can install:

conda install -c numba/label/dev numba=0.55.0dev0

(Edit: Note that while these packages have to pass our full test suite, they don't have any of the community testing that comes with an official release. So be careful. :) )

christian-cahig · 2021-12-11T05:32:10Z

If you are using conda, we also automatically build packages whenever a PR is merged, which you can install:
conda install -c numba/label/dev numba=0.55.0dev0
(Edit: Note that while these packages have to pass our full test suite, they don't have any of the community testing that comes with an official release. So be careful. :) )

Thanks, @seibert!

stuartarchibald requested review from esc and sklam as code owners October 14, 2021 08:37

stuartarchibald added 5 commits October 14, 2021 09:51

Update build matrix for NumPy 1.21 as per NEP-029.

3a2c504

As title.

Fix up conda recipe

f7d4a6c

As title. Also updates python version that was incorrect

Fix setup.py for NumPy minimum 1.18

c3824b4

As title.

Fix __init__ catching unsupported NumPy

d4c2bfb

Fix up docs for np1.120.

4fd6ef6

As title.

stuartarchibald force-pushed the fix/7176 branch from 278f003 to 4fd6ef6 Compare October 14, 2021 08:51

stuartarchibald added the 2 - In Progress label Oct 14, 2021

stuartarchibald added this to the Numba 0.55 RC milestone Oct 14, 2021

seberg reviewed Oct 15, 2021

View reviewed changes

numba/np/ufunc/_internal.c Outdated Show resolved Hide resolved

Add suggested fix for detecting ufunc.at

4df2640

As title.

moble mentioned this pull request Oct 21, 2021

Bug in test_algebra.py: unexpected keyword argument 'extra_arg' moble/quaternionic#26

Closed

stuartarchibald force-pushed the fix/7176 branch from 283bb98 to 635e056 Compare November 16, 2021 12:09

sklam self-assigned this Nov 18, 2021

sklam approved these changes Nov 22, 2021

View reviewed changes

sklam added 4 - Waiting on reviewer Waiting for reviewer to respond to author and removed 3 - Ready for Review labels Nov 22, 2021

sklam added 4 - Waiting on CI Review etc done, waiting for CI to finish and removed 4 - Waiting on reviewer Waiting for reviewer to respond to author labels Nov 22, 2021

sklam added 5 - Ready to merge Review and testing done, is ready to merge and removed Pending BuildFarm For PRs that have been reviewed but pending a push through our buildfarm 4 - Waiting on CI Review etc done, waiting for CI to finish labels Nov 22, 2021

sklam merged commit fc280ec into numba:master Nov 22, 2021

This was referenced Nov 23, 2021

BUG: Fastcall fix ufunc methods (NumPy 1.21 support) #7449

Closed

numpy 1.21 ufunc reduce segfault #7423

Closed

Ufunc method incompatibility with Numpy 1.21 #7175

Closed

NumPy 1.21 support tracker #7176

Closed

cmeyer mentioned this pull request Nov 28, 2021

Interoperability issue with hyperspy and numba nion-software/nionswift#768

Closed

noah-peeters mentioned this pull request Dec 19, 2021

Bump numpy from 1.20.3 to 1.21.4 noah-peeters/ChimpStackr#2

Closed

trimeta mentioned this pull request Jan 12, 2022

Accelerate release of 0.55 to no longer depend on versions of numpy with buffer overflow #7731

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NumPy 1.21 support #7483

NumPy 1.21 support #7483

stuartarchibald commented Oct 14, 2021 •

edited

gmarkall commented Oct 14, 2021

stuartarchibald commented Oct 14, 2021

stuartarchibald commented Oct 14, 2021

stuartarchibald commented Oct 14, 2021

gmarkall commented Oct 14, 2021

stuartarchibald commented Oct 14, 2021

gmarkall commented Oct 14, 2021

gmarkall commented Oct 14, 2021

seberg commented Oct 14, 2021

stuartarchibald commented Oct 14, 2021

seberg commented Oct 14, 2021

stuartarchibald commented Oct 14, 2021

seberg commented Oct 14, 2021

stuartarchibald commented Oct 15, 2021

stuartarchibald commented Oct 15, 2021

sklam commented Oct 21, 2021

esc commented Nov 22, 2021

sklam left a comment

sklam commented Nov 22, 2021

esc commented Nov 22, 2021

esc commented Nov 22, 2021

esc commented Nov 22, 2021

seberg commented Nov 23, 2021

christian-cahig commented Dec 8, 2021

seibert commented Dec 8, 2021 •

edited

kyoungseoun-chung commented Dec 8, 2021

seberg commented Dec 8, 2021

seibert commented Dec 8, 2021 •

edited

christian-cahig commented Dec 11, 2021

NumPy 1.21 support #7483

NumPy 1.21 support #7483

Conversation

stuartarchibald commented Oct 14, 2021 • edited

gmarkall commented Oct 14, 2021

stuartarchibald commented Oct 14, 2021

stuartarchibald commented Oct 14, 2021

stuartarchibald commented Oct 14, 2021

gmarkall commented Oct 14, 2021

stuartarchibald commented Oct 14, 2021

gmarkall commented Oct 14, 2021

gmarkall commented Oct 14, 2021

seberg commented Oct 14, 2021

stuartarchibald commented Oct 14, 2021

seberg commented Oct 14, 2021

stuartarchibald commented Oct 14, 2021

seberg commented Oct 14, 2021

stuartarchibald commented Oct 15, 2021

stuartarchibald commented Oct 15, 2021

sklam commented Oct 21, 2021

esc commented Nov 22, 2021

sklam left a comment

Choose a reason for hiding this comment

sklam commented Nov 22, 2021

esc commented Nov 22, 2021

esc commented Nov 22, 2021

esc commented Nov 22, 2021

seberg commented Nov 23, 2021

christian-cahig commented Dec 8, 2021

seibert commented Dec 8, 2021 • edited

kyoungseoun-chung commented Dec 8, 2021

seberg commented Dec 8, 2021

seibert commented Dec 8, 2021 • edited

christian-cahig commented Dec 11, 2021

stuartarchibald commented Oct 14, 2021 •

edited

seibert commented Dec 8, 2021 •

edited

seibert commented Dec 8, 2021 •

edited