Skip to content
This repository has been archived by the owner on Dec 31, 2023. It is now read-only.

How to configure the parameters of the main thread in the case of multiple workers? #76

Open
gclm opened this issue Dec 1, 2021 · 2 comments

Comments

@gclm
Copy link
Contributor

gclm commented Dec 1, 2021

At present, I want to run a torch program with multiple workers, but GPU CUDA needs to configure compatible parameters in the main thread. At present, the relevant configuration documents are not found in the documents

@ahopkins
Copy link
Member

ahopkins commented Dec 1, 2021

I am not entirely sure what you are trying to achieve. Maybe main_process_start?

https://sanicframework.org/en/guide/basics/listeners.html

@gclm
Copy link
Contributor Author

gclm commented Dec 2, 2021

no way. This seems to be configured every time the worker is created. At present, an error is still reported.
config code:

@app.listener("main_process_start")
async def init_processor(app, loop):
    mp.set_start_method('spawn')
    app.processor = Processor(MODEL_WEIGHTS, MODEL_LABELS)

error message:

[2021-12-02 16:36:56 +0800] [64614] [ERROR] Exception occurred while handling uri: 'http://localhost:8000/predict/file'
Traceback (most recent call last):
  File "handle_request", line 83, in handle_request
    class Sanic(BaseSanic, metaclass=TouchUpMeta):
  File "/opt/projects/ai-server/server.py", line 29, in predict_file
    result = app.processor.process(img)
  File "/opt/projects/ai-server/models/Detect.py", line 82, in process
    img = self.pre_processor(image)
  File "/opt/projects/ai-server/models/Detect.py", line 74, in pre_processor
    img = torch.from_numpy(img).to(self.device)
  File "/home/breakfox/anaconda3/envs/ai-server/lib/python3.9/site-packages/torch/cuda/__init__.py", line 204, in _lazy_init
    raise RuntimeError(
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
Exception occurred while handling uri: 'http://localhost:8000/predict/file'
Traceback (most recent call last):
  File "handle_request", line 83, in handle_request
    class Sanic(BaseSanic, metaclass=TouchUpMeta):
  File "/opt/projects/ai-server/server.py", line 29, in predict_file
    result = app.processor.process(img)
  File "/opt/projects/ai-server/models/Detect.py", line 82, in process
    img = self.pre_processor(image)
  File "/opt/projects/ai-server/models/Detect.py", line 74, in pre_processor
    img = torch.from_numpy(img).to(self.device)
  File "/home/breakfox/anaconda3/envs/ai-server/lib/python3.9/site-packages/torch/cuda/__init__.py", line 204, in _lazy_init
    raise RuntimeError(
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method

error message docs: https://discuss.pytorch.org/t/not-using-multiprocessing-but-getting-cuda-error-re-forked-subprocess/54610
How to configure it in this case?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants