Skip to content
This repository has been archived by the owner on Feb 15, 2023. It is now read-only.

Serving on GPU doesn't work locally. #15

Open
tchaton opened this issue Nov 21, 2022 · 1 comment
Open

Serving on GPU doesn't work locally. #15

tchaton opened this issue Nov 21, 2022 · 1 comment

Comments

@tchaton
Copy link
Contributor

tchaton commented Nov 21, 2022

When running the dreambooth.py component locally, it fails when serving the model locally.

/home/thomas/Dreambooth_app/.venv/lib/python3.8/site-packages/torch/utils/checkpoint.py:25: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
  warnings.warn("None of the inputs have requires_grad=True. Gradients will be None")
100%|█████████████████████████████████████████████████████████████████████| 51/51 [00:14<00:00,  3.60it/s]
Dreambooth finetuning is done!
INFO: Received SIGTERM signal. Gracefully terminating finetuner.ws.0...
{'pretrained_model_name_or_path': './model.pt'}
Process Process-2:
Traceback (most recent call last):
  File "/home/thomas/.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/home/thomas/.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/thomas/lightning/src/lightning/app/utilities/proxies.py", line 418, in __call__
    raise e
  File "/home/thomas/lightning/src/lightning/app/utilities/proxies.py", line 401, in __call__
    self.run_once()
  File "/home/thomas/lightning/src/lightning/app/utilities/proxies.py", line 549, in run_once
    self.work.on_exception(e)
  File "/home/thomas/lightning/src/lightning/app/core/work.py", line 564, in on_exception
    raise exception
  File "/home/thomas/lightning/src/lightning/app/utilities/proxies.py", line 514, in run_once
    ret = self.run_executor_cls(self.work, work_run, self.delta_queue)(*args, **kwargs)
  File "/home/thomas/lightning/src/lightning/app/utilities/proxies.py", line 350, in __call__
    return self.work_run(*args, **kwargs)
  File "/home/thomas/lightning/src/lightning/app/components/serve/python_server.py", line 212, in run
    self.setup()
  File "/home/thomas/Dreambooth_app/lightning_diffusion/diffusion_serve.py", line 44, in setup
    self._parent_flow.setup(*args, **kwargs)
  File "dreambooth_component.py", line 9, in setup
    self.model = StableDiffusionPipeline.from_pretrained(
  File "/home/thomas/Dreambooth_app/.venv/lib/python3.8/site-packages/diffusers/pipeline_utils.py", line 220, in to
    module.to(torch_device)
  File "/home/thomas/Dreambooth_app/.venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 927, in to
    return self._apply(convert)
  File "/home/thomas/Dreambooth_app/.venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 579, in _apply
    module._apply(fn)
  File "/home/thomas/Dreambooth_app/.venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 579, in _apply
    module._apply(fn)
  File "/home/thomas/Dreambooth_app/.venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 579, in _apply
    module._apply(fn)
  File "/home/thomas/Dreambooth_app/.venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 602, in _apply
    param_applied = fn(param)
  File "/home/thomas/Dreambooth_app/.venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 925, in convert
    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
  File "/home/thomas/Dreambooth_app/.venv/lib/python3.8/site-packages/torch/cuda/__init__.py", line 207, in _lazy_init
    raise RuntimeError(
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
@tchaton
Copy link
Contributor Author

tchaton commented Nov 21, 2022

Related to Lightning-AI/pytorch-lightning#15699

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant