spacy[cuda100] installs cupy-cuda100 8.0.0b1, which has a breaking change

[tommilligan]

How to reproduce the behaviour

$ pip install spacy[cuda100]==2.2.3
...
$ pip freeze | grep -E '(spacy|cupy)'
cupy-cuda100==8.0.0b1
spacy==2.2.3

See the release notes of cupy-cuda100==8.0.0b1 here: https://github.com/cupy/cupy/releases/tag/v8.0.0b1

Changes without compatibility

• Removed cupy.scatter_add, which had been deprecated since CuPy v4. Use cupyx.scatter_add instead.

Runtime Error

This results in the following runtime error in my application, but I'm sure would be similarly picked up by your unit tests:

  File "nn_parser.pyx", line 462, in spacy.syntax.nn_parser.Parser.update
  File "_parser_model.pyx", line 246, in spacy.syntax._parser_model.ParserModel.begin_update.finish_parser_update
  File "_parser_model.pyx", line 389, in spacy.syntax._parser_model.ParserStepModel.make_updates
  File "ops.pyx", line 1087, in thinc.neural.ops.CupyOps.scatter_add
AttributeError: module 'cupy' has no attribute 'scatter_add'

Your Environment

• Operating System: Ubuntu 18.04
• Python Version Used: 3.6.9
• spaCy Version Used: 2.2.3
• Environment Information: CUDA version 10.0.103

[adrianeboyd]

Thanks for the report! We should restrict it (at least temporarily) to <8.0.0, I think.

[tommilligan]

I agree that seems sensible for now. I've submitted #5194 which implements your suggestion.

[svlandeg]

oh, actually I have a local fix to use cupyx instead, I'll submit it to thinc (I should have done that few days ago, sorry!)

[ravi0912]

@svlandeg Can you post the fix in this thread because I'm also facing the same issue.

[svlandeg]

The PR mentioned just above your comment has the fix, cf https://github.com/explosion/thinc/pull/330/files (the fix will be in spaCy from 3.0 onwards (thinc >= v8.0.0a3))

The other option, as originally posted, is to restrict your cupy-cuda to < 8.0.0

[ivangru90]

Hi all,

I wanted to create new issues regarding this but luckly i found this which is about same topic. I have same problem when i run spacy training on gpu. I already had installed nvidia drivers and cuda 10.0. What i did to start training spacy on gpu is:
$ export CUDA_HOME=/usr/local/cuda-10.0/
$ pip install thinc_gpu_ops thinc[cuda100]
$ pip install spacy[cuda100]

Everything passed well during installations. In my code i added this:

import thinc_gpu_ops
thinc_gpu_ops.AVAILABLE
import spacy
spacy.require_gpu()

rest of code is just example from spacy documentation for training. It is working on cpu.

---

This is what i am getting when i start training on gpu:

AttributeError Traceback (most recent call last)
in

in train_spacy(data, iterations)

~/anaconda3/lib/python3.7/site-packages/spacy/language.py in begin_training(self, get_gold_tuples, sgd, component_cfg, **cfg)
627 pipeline=self.pipeline,
628 sgd=self._optimizer,
--> 629 **kwargs
630 )
631 return self._optimizer

nn_parser.pyx in spacy.syntax.nn_parser.Parser.begin_training()

_parser_model.pyx in spacy.syntax._parser_model.ParserModel.begin_training()

~/anaconda3/lib/python3.7/site-packages/thinc/check.py in checked_function(wrapped, instance, args, kwargs)
154 raise ExpectedTypeError(check, ["Callable"])
155 check(arg_id, fix_args, kwargs)
--> 156 return wrapped(*args, **kwargs)
157
158 def arg_check_adder(func):

~/anaconda3/lib/python3.7/site-packages/thinc/neural/_classes/model.py in begin_training(self, train_X, train_y, **trainer_cfg)
120 def begin_training(self, train_X, train_y=None, **trainer_cfg):
121 for hook in self.on_data_hooks:
--> 122 hook(self, train_X, train_y)
123 return self.Trainer(self, **trainer_cfg)
124

~/anaconda3/lib/python3.7/site-packages/spacy/_ml.py in (model, X, y)
149
150 @describe.on_data(
--> 151 _set_dimensions_if_needed, lambda model, X, y: model.init_weights(model)
152 )
153 @describe.attributes(

~/anaconda3/lib/python3.7/site-packages/spacy/_ml.py in init_weights(model)
268 t_i = 0
269 for t_i in range(t_max):
--> 270 acts1 = predict(ids, tokvecs)
271 var = model.ops.xp.var(acts1)
272 mean = model.ops.xp.mean(acts1)

~/anaconda3/lib/python3.7/site-packages/spacy/_ml.py in predict(ids, tokvecs)
254 (hiddens.shape[0] * model.nF, model.nO * model.nP)
255 )
--> 256 model.ops.scatter_add(vectors, ids.flatten(), hiddens)
257 vectors = vectors.reshape((vectors.shape[0], model.nO, model.nP))
258 vectors += model.b

ops.pyx in thinc.neural.ops.CupyOps.scatter_add()

AttributeError: module 'cupy' has no attribute 'scatter_add'

How i can fix this, what should i do?

Environment:

• Operating System: Linux Ubunutu 18LTS
• Python Version Used: 3.7
• spaCy Version Used: 2.2.4
• Environment Information: CUDA version 10.0.0

Thank you,
Ivan

[tommilligan]

Hi Ivan - I think the easiest way for you to fix this would be to run:

pip install -U 'cupy-cuda100==7.3.0'

which should downgrade your installed cupy package to a compatible version. Let me know if it doesn't work.

[ivangru90]

Thank you @tommilligan now it is working!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.