AttributeError: Can't get attribute 'NewsSummaryDataset' on <module '__main__' (built-in)>

[gokul427]

Getting below error when I fit the model, please help. I think due to below error, sanity validation step hangs.
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\gokul\anaconda3\envs\py38torch\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Users\gokul\anaconda3\envs\py38torch\lib\multiprocessing\spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
AttributeError: Can't get attribute 'NewsSummaryDataset' on <module 'main' (built-in)>
notebook:
https://drive.google.com/file/d/1S4CycFAd8i5uuOCvunLbZRdHUlbKu840/view?usp=sharing
data:
https://drive.google.com/file/d/11WKKEcZta26EVDCz7x-KjchxZrL7oPNC/view?usp=sharing
cuda 11.6
pytorch-lightning==1.7.7
torch==1.12.1
transformers==4.23.1

[akihironitta]

To debug your issue, I'd start by running a simpler script:
https://github.com/Lightning-AI/lightning/tree/c39c8eb2e4f49e35ef38a2ae6de2765ec71623db/examples/pl_bug_report

[rwoodard-prog]

I think this has to do with how Jupyter handles passing objects to workers. The bug report script works fine with multiple workers but the notebook fails in the way described.

I am testing by adding num_workers=2 to the train DataLoader:

Script:

$ diff ~/Documents/bug_report_model.py ./bug_report_model.py
47c47
<     train_data = DataLoader(RandomDataset(32, 64), batch_size=2)
---
>     train_data = DataLoader(RandomDataset(32, 64), batch_size=2, num_workers=2)

$ python ./bug_report_model.py
GPU available: True (mps), used: False
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
...
────────
       Test metric             DataLoader 0
────────
        test_loss           0.7913365364074707
────────

Notebook:

diff ~/Documents/bug_report_model.ipynb.txt notebooks/pytorch_lightning_boring_model_test.ipynb
191c191
<         "    train_data = DataLoader(RandomDataset(32, 64), batch_size=2)\n",
---
>         "    train_data = DataLoader(RandomDataset(32, 64), batch_size=2, num_workers=2)\n",

Notebook crashes with:

run()
...
RuntimeError: DataLoader worker (pid 72196) exited unexpectedly with exit
code 1. Details are lost due to multiprocessing. Rerunning with
num_workers=0 may give better error trace.
...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/homebrew/Caskroom/miniconda/base/envs/db_rt_11.3_LTS_ML_sorta/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/db_rt_11.3_LTS_ML_sorta/lib/python3.9/multiprocessing/spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
AttributeError: Can't get attribute 'RandomDataset' on <module '__main__' (built-in)>

This type of multiprocessing pickling affects Jupyter and I will dig a bit for the fix as I can. Just wanted to share this info here in case someone has a fix.

[rwoodard-prog]

And a quick and dirty fix is to move the offending RandomDataset into a separate file and import it like so:

Notebook, cell 3:

# Moved to boring.py
# class RandomDataset(Dataset):
#     def __init__(self, size, num_samples):
# ...
from boring import RandomDataset

Contents of boring.py:

import torch


class RandomDataset(torch.utils.data.Dataset):
    def __init__(self, size, num_samples):
        self.len = num_samples
        self.data = torch.randn(num_samples, size)

    def __getitem__(self, index):
        return self.data[index]

    def __len__(self):
        return self.len

Seems to work for me. Gets around the way Jupyter handles multiprocessing. Some extra links for reading about this:

https://stackoverflow.com/a/59569742/17637057

https://stackoverflow.com/questions/52265120/python-multiprocessing-pool-attributeerror

https://discuss.pytorch.org/t/distributeddataparallel-on-terminal-vs-jupyter-notebook/101404

https://pytorch-lightning.readthedocs.io/en/stable/accelerators/gpu_intermediate.html#distributed-data-parallel-in-notebooks

[bala1802]

wow! this worked :) thanks

[Codernob]

you saved my life. I'm forever indebted to you.

[ImRIzo]

it works !!!

[3zhang]

Is there a fix now? I also encountered this issue on Spyder. The above fix didn't work either. It gave the following error:
ModuleNotFoundError: No module named 'ts_dataset'

[mikbuch]

Just removing workers (num_workers argument) worked as a quick fix for me with Jupyter Lab