Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

can't install pytorch 2.2 when install.cache is on #2614

Closed
1 task done
fancyerii opened this issue Feb 4, 2024 · 18 comments
Closed
1 task done

can't install pytorch 2.2 when install.cache is on #2614

fancyerii opened this issue Feb 4, 2024 · 18 comments
Labels
🤔 not enough info Requires more information to clarify the issue

Comments

@fancyerii
Copy link

fancyerii commented Feb 4, 2024 •
edited

  • I have searched the issue tracker and believe that this is not a duplicate.

Make sure you run commands with -v flag before pasting the output.

Steps to reproduce

pdm add torch torchaudio torchvision numpy
pdm shell(I follow Looking for pdm shell? in https://pdm-project.org/2.12/usage/venv/#activate-a-virtualenv)
$ python
Python 3.9.18 (main, Feb  2 2024, 13:57:14) 
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> print(torch.__version__)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'torch' has no attribute '__version__'

Actual behavior

Expected behavior

it works with cache.

Environment Information

# Paste the output of `pdm info && pdm info --env` below:

$ pdm info && pdm info --env
PDM version:
  2.12.3
Python Interpreter:
  /nas/lili/codes/pt/my-project/.venv/bin/python (3.9)
Project Root:
  /nas/lili/codes/pt/my-project
Local Packages:
  
{
  "implementation_name": "cpython",
  "implementation_version": "3.9.18",
  "os_name": "posix",
  "platform_machine": "x86_64",
  "platform_release": "4.15.0-213-generic",
  "platform_system": "Linux",
  "platform_version": "#224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023",
  "python_full_version": "3.9.18",
  "platform_python_implementation": "CPython",
  "python_version": "3.9",
  "sys_platform": "linux"
}

I also tried symlink_individual and hardlink. none of them worked. I even tried "pth" in some old issue but still not work.
I search related issue such as #1732 and can't find solution.
When I set install.cache=False, it works. But that's not I wanted.
@fancyerii fancyerii added the 🐛 bug Something isn't working label Feb 4, 2024
@fancyerii fancyerii mentioned this issue Feb 4, 2024
Closed
1 task
@fancyerii
Copy link
Author

fancyerii commented Feb 4, 2024 •
edited

when I view the torch installation directory, there isn't any file here:

tree
...
│       ├── opinfo
│       │   └── definitions
│       ├── optests
│       └── test_module
├── utils
│   ├── backcompat
│   ├── benchmark
│   │   ├── examples
│   │   ├── op_fuzzers
│   │   └── utils
│   │       └── valgrind_wrapper
│   ├── bottleneck
│   ├── data
│   │   ├── datapipes
│   │   │   ├── dataframe
│   │   │   ├── iter
│   │   │   ├── map
│   │   │   └── utils
│   │   └── _utils
│   ├── hipify
│   ├── jit
│   ├── model_dump
│   ├── _sympy
│   ├── tensorboard
│   └── viz
└── _vendor
    ├── packaging
    │   └── __pycache__
    └── __pycache__

416 directories, 0 files

There isn't any py file here. As comparasion. Here is my installation by poetry:

│   │   ├── _utils.py
│   │   └── writer.py
│   ├── throughput_benchmark.py
│   ├── _traceback.py
│   ├── _triton.py
│   ├── viz
│   │   ├── _cycles.py
│   │   └── __init__.py
│   ├── weak.py
│   └── _zip.py
├── _utils_internal.py
├── _utils.py
├── _vendor
│   ├── __init__.py
│   ├── packaging
│   │   ├── __init__.py
│   │   ├── __pycache__
│   │   │   ├── __init__.cpython-39.pyc
│   │   │   ├── _structures.cpython-39.pyc
│   │   │   └── version.cpython-39.pyc
│   │   ├── _structures.py
│   │   └── version.py
│   └── __pycache__
│       └── __init__.cpython-39.pyc
├── version.py
├── _VF.py
├── _VF.pyi
├── _vmap_internals.py
└── _weights_only_unpickler.py

582 directories, 11072 files

@fancyerii
Copy link
Author

when I installed without cache. the torch directory is correctly installed.

tree .venv/lib/python3.9/site-packages/torch
.....
│   │   ├── _pytorch_graph.py
│   │   ├── summary.py
│   │   ├── _utils.py
│   │   └── writer.py
│   ├── throughput_benchmark.py
│   ├── _traceback.py
│   ├── _triton.py
│   ├── viz
│   │   ├── _cycles.py
│   │   └── __init__.py
│   ├── weak.py
│   └── _zip.py
├── _utils_internal.py
├── _utils.py
├── _vendor
│   ├── __init__.py
│   ├── packaging
│   │   ├── __init__.py
│   │   ├── __pycache__
│   │   │   ├── __init__.cpython-39.pyc
│   │   │   ├── _structures.cpython-39.pyc
│   │   │   └── version.cpython-39.pyc
│   │   ├── _structures.py
│   │   └── version.py
│   └── __pycache__
│       └── __init__.cpython-39.pyc
├── version.py
├── _VF.py
├── _VF.pyi
├── _vmap_internals.py
└── _weights_only_unpickler.py

537 directories, 10714 files

@frostming
Copy link
Collaborator

frostming commented Feb 4, 2024 •
edited

Can't reproduce. Maybe related to nix. Can you look at what the links point to by ls -l .venv/lib/python3.10/site-packages/

@frostming frostming added 🤔 not enough info Requires more information to clarify the issue and removed 🐛 bug Something isn't working labels Feb 4, 2024
@fancyerii
Copy link
Author

@frostming

$ ls -l .venv/lib/python3.9/site-packages/
total 20
lrwxrwxrwx 1 ubuntu ubuntu   74 Feb  4 14:12 certifi -> /home/ubuntu/.cache/pdm/packages/certifi-2024.2.2-py3-none-any/lib/certifi
drwxrwxr-x 2 ubuntu ubuntu  123 Feb  4 14:12 certifi-2024.2.2.dist-info
lrwxrwxrwx 1 ubuntu ubuntu  133 Feb  4 14:12 charset_normalizer -> /home/ubuntu/.cache/pdm/packages/charset_normalizer-3.3.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/charset_normalizer
drwxrwxr-x 2 ubuntu ubuntu  150 Feb  4 14:12 charset_normalizer-3.3.2.dist-info
lrwxrwxrwx 1 ubuntu ubuntu   74 Feb  4 14:12 filelock -> /home/ubuntu/.cache/pdm/packages/filelock-3.13.1-py3-none-any/lib/filelock
drwxrwxr-x 3 ubuntu ubuntu  100 Feb  4 14:12 filelock-3.13.1.dist-info
lrwxrwxrwx 1 ubuntu ubuntu   73 Feb  4 14:12 fsspec -> /home/ubuntu/.cache/pdm/packages/fsspec-2023.12.2-py3-none-any/lib/fsspec
drwxrwxr-x 2 ubuntu ubuntu  123 Feb  4 14:12 fsspec-2023.12.2.dist-info
lrwxrwxrwx 1 ubuntu ubuntu   86 Feb  4 14:13 functorch -> /home/ubuntu/.cache/pdm/packages/torch-2.2.0-cp39-cp39-manylinux1_x86_64/lib/functorch
lrwxrwxrwx 1 ubuntu ubuntu   63 Feb  4 14:12 idna -> /home/ubuntu/.cache/pdm/packages/idna-3.6-py3-none-any/lib/idna
drwxrwxr-x 2 ubuntu ubuntu  102 Feb  4 14:12 idna-3.6.dist-info
lrwxrwxrwx 1 ubuntu ubuntu   69 Feb  4 14:12 jinja2 -> /home/ubuntu/.cache/pdm/packages/Jinja2-3.1.3-py3-none-any/lib/jinja2
drwxrwxr-x 2 ubuntu ubuntu  154 Feb  4 14:12 Jinja2-3.1.3.dist-info
lrwxrwxrwx 1 ubuntu ubuntu  117 Feb  4 14:12 markupsafe -> /home/ubuntu/.cache/pdm/packages/MarkupSafe-2.1.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/markupsafe
drwxrwxr-x 2 ubuntu ubuntu  127 Feb  4 14:12 MarkupSafe-2.1.5.dist-info
lrwxrwxrwx 1 ubuntu ubuntu   69 Feb  4 14:12 mpmath -> /home/ubuntu/.cache/pdm/packages/mpmath-1.3.0-py3-none-any/lib/mpmath
drwxrwxr-x 2 ubuntu ubuntu  123 Feb  4 14:12 mpmath-1.3.0.dist-info
lrwxrwxrwx 1 ubuntu ubuntu   73 Feb  4 14:12 networkx -> /home/ubuntu/.cache/pdm/packages/networkx-3.2.1-py3-none-any/lib/networkx
drwxrwxr-x 2 ubuntu ubuntu  154 Feb  4 14:12 networkx-3.2.1.dist-info
lrwxrwxrwx 1 ubuntu ubuntu  108 Feb  4 14:12 numpy -> /home/ubuntu/.cache/pdm/packages/numpy-1.26.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/numpy
drwxrwxr-x 2 ubuntu ubuntu  130 Feb  4 14:12 numpy-1.26.3.dist-info
lrwxrwxrwx 1 ubuntu ubuntu  113 Feb  4 14:12 numpy.libs -> /home/ubuntu/.cache/pdm/packages/numpy-1.26.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/numpy.libs
lrwxrwxrwx 1 ubuntu ubuntu   97 Feb  4 14:13 nvidia -> /home/ubuntu/.cache/pdm/packages/nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64/lib/nvidia
drwxrwxr-x 2 ubuntu ubuntu  127 Feb  4 14:13 nvidia_cublas_cu12-12.1.3.1.dist-info
drwxrwxr-x 2 ubuntu ubuntu  127 Feb  4 14:12 nvidia_cuda_cupti_cu12-12.1.105.dist-info
drwxrwxr-x 2 ubuntu ubuntu  127 Feb  4 14:12 nvidia_cuda_nvrtc_cu12-12.1.105.dist-info
drwxrwxr-x 2 ubuntu ubuntu  127 Feb  4 14:12 nvidia_cuda_runtime_cu12-12.1.105.dist-info
drwxrwxr-x 2 ubuntu ubuntu  127 Feb  4 14:13 nvidia_cudnn_cu12-8.9.2.26.dist-info
drwxrwxr-x 2 ubuntu ubuntu  127 Feb  4 14:12 nvidia_cufft_cu12-11.0.2.54.dist-info
drwxrwxr-x 2 ubuntu ubuntu  127 Feb  4 14:12 nvidia_curand_cu12-10.3.2.106.dist-info
drwxrwxr-x 2 ubuntu ubuntu  127 Feb  4 14:12 nvidia_cusolver_cu12-11.4.5.107.dist-info
drwxrwxr-x 2 ubuntu ubuntu  127 Feb  4 14:13 nvidia_cusparse_cu12-12.1.0.106.dist-info
drwxrwxr-x 2 ubuntu ubuntu  127 Feb  4 14:13 nvidia_nccl_cu12-2.19.3.dist-info
drwxrwxr-x 2 ubuntu ubuntu  127 Feb  4 14:12 nvidia_nvjitlink_cu12-12.3.101.dist-info
drwxrwxr-x 2 ubuntu ubuntu  127 Feb  4 14:12 nvidia_nvtx_cu12-12.1.105.dist-info
lrwxrwxrwx 1 ubuntu ubuntu  107 Feb  4 14:12 PIL -> /home/ubuntu/.cache/pdm/packages/pillow-10.2.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/PIL
drwxrwxr-x 2 ubuntu ubuntu  142 Feb  4 14:12 pillow-10.2.0.dist-info
lrwxrwxrwx 1 ubuntu ubuntu  115 Feb  4 14:12 pillow.libs -> /home/ubuntu/.cache/pdm/packages/pillow-10.2.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/pillow.libs
drwxrwxr-x 2 ubuntu ubuntu   47 Feb  4 14:12 __pycache__
lrwxrwxrwx 1 ubuntu ubuntu   74 Feb  4 14:12 requests -> /home/ubuntu/.cache/pdm/packages/requests-2.31.0-py3-none-any/lib/requests
drwxrwxr-x 2 ubuntu ubuntu  123 Feb  4 14:12 requests-2.31.0.dist-info
lrwxrwxrwx 1 ubuntu ubuntu   66 Feb  4 14:12 sympy -> /home/ubuntu/.cache/pdm/packages/sympy-1.12-py3-none-any/lib/sympy
drwxrwxr-x 2 ubuntu ubuntu 4096 Feb  4 14:12 sympy-1.12.dist-info
lrwxrwxrwx 1 ubuntu ubuntu   82 Feb  4 14:13 torch -> /home/ubuntu/.cache/pdm/packages/torch-2.2.0-cp39-cp39-manylinux1_x86_64/lib/torch
drwxrwxr-x 2 ubuntu ubuntu 4096 Feb  4 14:13 torch-2.2.0.dist-info
lrwxrwxrwx 1 ubuntu ubuntu   92 Feb  4 14:13 torchaudio -> /home/ubuntu/.cache/pdm/packages/torchaudio-2.2.0-cp39-cp39-manylinux1_x86_64/lib/torchaudio
drwxrwxr-x 2 ubuntu ubuntu  123 Feb  4 14:13 torchaudio-2.2.0.dist-info
lrwxrwxrwx 1 ubuntu ubuntu   85 Feb  4 14:13 torchgen -> /home/ubuntu/.cache/pdm/packages/torch-2.2.0-cp39-cp39-manylinux1_x86_64/lib/torchgen
lrwxrwxrwx 1 ubuntu ubuntu   95 Feb  4 14:13 torchvision -> /home/ubuntu/.cache/pdm/packages/torchvision-0.17.0-cp39-cp39-manylinux1_x86_64/lib/torchvision
drwxrwxr-x 2 ubuntu ubuntu  123 Feb  4 14:13 torchvision-0.17.0.dist-info
lrwxrwxrwx 1 ubuntu ubuntu  100 Feb  4 14:13 torchvision.libs -> /home/ubuntu/.cache/pdm/packages/torchvision-0.17.0-cp39-cp39-manylinux1_x86_64/lib/torchvision.libs
lrwxrwxrwx 1 ubuntu ubuntu   87 Feb  4 14:13 torio -> /home/ubuntu/.cache/pdm/packages/torchaudio-2.2.0-cp39-cp39-manylinux1_x86_64/lib/torio
lrwxrwxrwx 1 ubuntu ubuntu  109 Feb  4 14:13 triton -> /home/ubuntu/.cache/pdm/packages/triton-2.2.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/triton
drwxrwxr-x 2 ubuntu ubuntu  105 Feb  4 14:13 triton-2.2.0.dist-info
drwxrwxr-x 2 ubuntu ubuntu   99 Feb  4 14:13 typing_extensions-4.9.0.dist-info
lrwxrwxrwx 1 ubuntu ubuntu   71 Feb  4 14:13 urllib3 -> /home/ubuntu/.cache/pdm/packages/urllib3-2.2.0-py3-none-any/lib/urllib3
drwxrwxr-x 3 ubuntu ubuntu  100 Feb  4 14:13 urllib3-2.2.0.dist-info
-rw-rw-r-- 1 ubuntu ubuntu   18 Feb  4 14:12 _virtualenv.pth
-rw-rw-r-- 1 ubuntu ubuntu 4329 Feb  4 14:12 _virtualenv.py

@frostming
Copy link
Collaborator

frostming commented Feb 4, 2024 •
edited

so let's look at one link for example, does /home/ubuntu/.cache/pdm/packages/torch-2.2.0-cp39-cp39-manylinux1_x86_64/lib/torch contain any files? The tree command may not follow the symlinks, so what you saw may not be wrong.

@fancyerii
Copy link
Author

@frostming

ubuntu@VM-4-12-ubuntu:~/.cache/pdm/packages/torch-2.2.0-cp39-cp39-manylinux1_x86_64/lib/torch$ ls
amp       _C          _decomp        _export     _higher_order_ops  _library  multiprocessing  package        _refs        testing
ao        compiler    _dispatch      fft         include            linalg    nested           _prims         share        utils
autograd  contrib     distributed    func        _inductor          _logging  nn               _prims_common  signal       _vendor
_awaits   cpu         distributions  _functorch  jit                masked    _numpy           profiler       sparse
backends  cuda        _dynamo        futures     _lazy              monitor   onnx             __pycache__    special
bin       _custom_op  export         fx          lib                mps       optim            quantization   _subclasses

there is no file in this directory. And I also tried poetry:

(ft-py3.9) ubuntu@VM-4-12-ubuntu:~/.cache/pypoetry/virtualenvs/ft-zSqjAXBp-py3.9/lib/python3.9/site-packages/torch$ ls
amp                                distributed        lib                        package                _streambase.py
ao                                 distributions      _library                   _prims                 _subclasses
_appdirs.py                        _dynamo            library.py                 _prims_common          _tensor_docs.py
autograd                           export             linalg                     profiler               _tensor.py
_awaits                            _export            _linalg_utils.py           __pycache__            _tensor_str.py
backends                           fft                _lobpcg.py                 _python_dispatcher.py  testing
bin                                func               _logging                   py.typed               _torch_docs.py
_C                                 functional.py      _lowrank.py                quantization           torch_version.py
_C.cpython-39-x86_64-linux-gnu.so  _functorch         masked                     quasirandom.py         types.py
_classes.py                        __future__.py      _meta_registrations.py     random.py              utils
_compile.py                        futures            monitor                    _refs                  _utils_internal.py
compiler                           fx                 mps                        return_types.py        _utils.py
__config__.py                      _guards.py         multiprocessing            return_types.pyi       _vendor
contrib                            _higher_order_ops  _namedtensor_internals.py  serialization.py       version.py
cpu                                hub.py             nested                     share                  _VF.py
cuda                               include            nn                         signal                 _VF.pyi
_custom_op                         _inductor          _numpy                     _sources.py            _vmap_internals.py
_custom_ops.py                     __init__.py        onnx                       sparse                 _weights_only_unpickler.py
_decomp                            jit                _ops.py                    special
_deploy.py                         _jit_internal.py   optim                      _storage_docs.py
_dispatch                          _lazy              overrides.py               storage.py

And I also check disk usage:

ubuntu@VM-4-12-ubuntu:~/.cache/pdm/packages/torch-2.2.0-cp39-cp39-manylinux1_x86_64/lib$ du -sh torch
2.0M    torch

(ft-py3.9) ubuntu@VM-4-12-ubuntu:~/.cache/pypoetry/virtualenvs/ft-zSqjAXBp-py3.9/lib/python3.9/site-packages$ du -sh torch
1.5G    torch

@frostming
Copy link
Collaborator

Clear caches by pdm cache clear packages and run pdm sync -r to see if the issue still exists

@fancyerii
Copy link
Author

@frostming

ubuntu@VM-4-12-ubuntu:/nas/lili/codes/pt/my-project$ pdm cache clear packages
No files need to be removed
ubuntu@VM-4-12-ubuntu:/nas/lili/codes/pt/my-project$ pdm sync -r
Synchronizing working set with resolved packages: 0 to add, 31 to update, 0 to remove

  ✔ Update filelock 3.13.1 -> 3.13.1 successful
  ✔ Update idna 3.6 -> 3.6 successful
  ✔ Update certifi 2024.2.2 -> 2024.2.2 successful
  ✔ Update jinja2 3.1.3 -> 3.1.3 successful
  ✔ Update fsspec 2023.12.2 -> 2023.12.2 successful
  ✔ Update mpmath 1.3.0 -> 1.3.0 successful
  ✔ Update markupsafe 2.1.5 -> 2.1.5 successful
  ✔ Update charset-normalizer 3.3.2 -> 3.3.2 successful
  ✔ Update nvidia-cuda-runtime-cu12 12.1.105 -> 12.1.105 successful
  ✔ Update nvidia-cuda-cupti-cu12 12.1.105 -> 12.1.105 successful
  ✔ Update networkx 3.2.1 -> 3.2.1 successful
  ✔ Update nvidia-cuda-nvrtc-cu12 12.1.105 -> 12.1.105 successful
  ✔ Update numpy 1.26.3 -> 1.26.3 successful
  ✔ Update nvidia-curand-cu12 10.3.2.106 -> 10.3.2.106 successful
  ✔ Update nvidia-nvtx-cu12 12.1.105 -> 12.1.105 successful
  ✔ Update nvidia-nvjitlink-cu12 12.3.101 -> 12.3.101 successful
  ✔ Update pillow 10.2.0 -> 10.2.0 successful
  ✔ Update requests 2.31.0 -> 2.31.0 successful
  ✔ Update nvidia-cufft-cu12 11.0.2.54 -> 11.0.2.54 successful
  ✔ Update nvidia-cusolver-cu12 11.4.5.107 -> 11.4.5.107 successful
  ✔ Update torchaudio 2.2.0 -> 2.2.0 successful
  ✔ Update sympy 1.12 -> 1.12 successful
  ✔ Update typing-extensions 4.9.0 -> 4.9.0 successful
  ✔ Update torchvision 0.17.0 -> 0.17.0 successful
  ✔ Update urllib3 2.2.0 -> 2.2.0 successful
  ✔ Update nvidia-nccl-cu12 2.19.3 -> 2.19.3 successful
  ✔ Update nvidia-cusparse-cu12 12.1.0.106 -> 12.1.0.106 successful 
  ✔ Update triton 2.2.0 -> 2.2.0 successful
  ✔ Update nvidia-cublas-cu12 12.1.3.1 -> 12.1.3.1 successful
  ✔ Update nvidia-cudnn-cu12 8.9.2.26 -> 8.9.2.26 successful
  ✔ Update torch 2.2.0 -> 2.2.0 successful

🎉 All complete!

ubuntu@VM-4-12-ubuntu:/nas/lili/codes/pt/my-project$ pdm shell
(my-project-3.9) ubuntu@VM-4-12-ubuntu:/nas/lili/codes/pt/my-project$ python
Python 3.9.18 (main, Feb  2 2024, 13:57:14) 
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/nas/lili/codes/pt/my-project/.venv/lib/python3.9/site-packages/torch/__init__.py", line 237, in <module>
    from torch._C import *  # noqa: F403
ImportError: libcupti.so.12: cannot open shared object file: No such file or directory
>>>

@fancyerii
Copy link
Author 

(my-project-3.9) ubuntu@VM-4-12-ubuntu:/nas/lili/codes/pt/my-project/.venv/lib/python3.9/site-packages$ 
 tree torch
....
│   │   ├── _cycles.py
│   │   └── __init__.py
│   ├── weak.py
│   └── _zip.py
├── _utils_internal.py
├── _utils.py
├── _vendor
│   ├── __init__.py
│   ├── packaging
│   │   ├── __init__.py
│   │   ├── __pycache__
│   │   │   ├── __init__.cpython-39.pyc
│   │   │   ├── _structures.cpython-39.pyc
│   │   │   └── version.cpython-39.pyc
│   │   ├── _structures.py
│   │   └── version.py
│   └── __pycache__
│       └── __init__.cpython-39.pyc
├── version.py
├── _VF.py
├── _VF.pyi
├── _vmap_internals.py
└── _weights_only_unpickler.py

416 directories, 10159 files

there are py files now. But another import problem occurs.

@frostming
Copy link
Collaborator

Change the cache method:

pdm config -l install.cache_method individual_symlink
pdm sync -r

@frostming
Copy link
Collaborator

And don't compare with poetry, it doesn't support install cache at all.

@fancyerii
Copy link
Author

@frostming

(my-project-3.9) ubuntu@VM-4-12-ubuntu:/nas/lili/codes/pt/my-project$ python
Python 3.9.18 (main, Feb  2 2024, 13:57:14) 
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> print(torch.__version__)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'torch' has no attribute '__version__'
>>> 

pdm config
...
global_project.fallback_verbose = True
global_project.path = /home/ubuntu/.config/pdm/global-project
global_project.user_site = False
install.cache = False
install.cache_method = symlink
install.parallel = True
pypi.ignore_stored_index = False
pypi.json_api = False
pypi.url = https://pypi.org/simple
pypi.verify_ssl = True
python.providers = ['venv', 'path', 'asdf', 'pyenv', 'rye', 'winreg', 'macos']
python.use_pyenv = True
python.use_venv = True
strategy.inherit_metadata = True
strategy.resolve_max_rounds = 10000
strategy.save = minimum
strategy.update = reuse
theme.error = red
theme.info = blue
theme.primary = cyan
theme.req = bold green
theme.success = green
theme.warning = yellow
venv.backend = virtualenv
venv.in_project = True
venv.location = /home/ubuntu/.local/share/pdm/venvs
venv.prompt = {project_name}-{python_version}
venv.with_pip = False

Home configuration (/home/ubuntu/.config/pdm/config.toml):
install.cache = True

Project configuration (/nas/lili/codes/pt/my-project/pdm.toml):
install.cache_method = individual_symlink

(my-project-3.9) ubuntu@VM-4-12-ubuntu:/nas/lili/codes/pt/my-project/.venv/lib/python3.9/site-packages/torch$ ll
total 412
drwxrwxr-x 58 ubuntu ubuntu 4096 Feb  4 19:33 ./
drwxrwxr-x  6 ubuntu ubuntu 4096 Feb  4 12:27 ../
drwxrwxr-x  2 ubuntu ubuntu 4096 Feb  4 19:33 amp/
drwxrwxr-x  6 ubuntu ubuntu 4096 Feb  4 19:33 ao/
lrwxrwxrwx  1 ubuntu ubuntu   94 Feb  4 19:33 _appdirs.py -> /home/ubuntu/.cache/pdm/packages/torch-2.2.0-cp39-cp39-manylinux1_x86_64/lib/torch/_appdirs.py                                                                                                                               
drwxrwxr-x  3 ubuntu ubuntu 4096 Feb  4 19:33 autograd/
drwxrwxr-x  2 ubuntu ubuntu 4096 Feb  4 19:33 _awaits/
drwxrwxr-x 15 ubuntu ubuntu 4096 Feb  4 19:33 backends/
drwxrwxr-x  2 ubuntu ubuntu 4096 Feb  4 19:33 bin/
drwxrwxr-x  2 ubuntu ubuntu 4096 Feb  4 19:33 _C/
lrwxrwxrwx  1 ubuntu ubuntu  116 Feb  4 19:33 _C.cpython-39-x86_64-linux-gnu.so -> /home/ubuntu/.cache/pdm/packages/torch-2.2.0-cp39-cp39-manylinux1_x86_64/lib/torch/_C.cpython-39-x86_64-linux-gnu.so
lrwxrwxrwx  1 ubuntu ubuntu   94 Feb  4 19:33 _classes.py -> /home/ubuntu/.cache/pdm/packages/torch-2.2.0-cp39-cp39-manylinux1_x86_64/lib/torch/_classes.py                                                                                                                               
lrwxrwxrwx  1 ubuntu ubuntu   94 Feb  4 19:33 _compile.py -> /home/ubuntu/.cache/pdm/packages/torch-2.2.0-cp39-cp39-manylinux1_x86_64/lib/torch/_compile.py                                                                                                                               
drwxrwxr-x  2 ubuntu ubuntu 4096 Feb  4 19:33 compiler/
lrwxrwxrwx  1 ubuntu ubuntu   96 Feb  4 19:33 __config__.py -> /home/ubuntu/.cache/pdm/packages/torch-2.2.0-cp39-cp39-manylinux1_x86_64/lib/torch/__config__.py                                                                                                                           
drwxrwxr-x  2 ubuntu ubuntu 4096 Feb  4 19:33 contrib/

@fancyerii
Copy link
Author

@frostming I found the file in /home/ubuntu/.cache/pdm link to itself.

(my-project-3.9) ubuntu@VM-4-12-ubuntu:~$ ll /home/ubuntu/.cache/pdm/packages/torch-2.2.0-cp39-cp39-manylinux1_x86_64/lib/torch/version.py
lrwxrwxrwx 1 ubuntu ubuntu 93 Feb  4 19:33 /home/ubuntu/.cache/pdm/packages/torch-2.2.0-cp39-cp39-manylinux1_x86_64/lib/torch/version.py -> /home/ubuntu/.cache/pdm/packages/torch-2.2.0-cp39-cp39-manylinux1_x86_64/lib/torch/version.py

@frostming
Copy link
Collaborator

I found the file in /home/ubuntu/.cache/pdm link to itself.

Is it the only one? Manually fix it yourself and it should work.

@fancyerii
Copy link
Author

I just check one. and I have clear all cache with pdm cache clear and remove .venv and rerun pdm sync -r. I can see it download packages from internet.

This time, it become empty again and don't link to anything.

ubuntu@VM-4-12-ubuntu:/nas/lili/codes/pt/my-project/.venv/lib/python3.9/site-packages/torch$ tree 
│   │       └── valgrind_wrapper
│   ├── bottleneck
│   ├── data
│   │   ├── datapipes
│   │   │   ├── dataframe
│   │   │   ├── iter
│   │   │   ├── map
│   │   │   └── utils
│   │   └── _utils
│   ├── hipify
│   ├── jit
│   ├── model_dump
│   ├── _sympy
│   ├── tensorboard
│   └── viz
└── _vendor
    └── packaging

413 directories, 0 files

And I check cached packages. it's also empty(only a file named referrers)

~/.cache/pdm/packages/torch-2.2.0-cp39-cp39-manylinux1_x86_64$ tree
│   │   │   │   │   ├── map
│   │   │   │   │   └── utils
│   │   │   │   └── _utils
│   │   │   ├── hipify
│   │   │   ├── jit
│   │   │   ├── model_dump
│   │   │   ├── _sympy
│   │   │   ├── tensorboard
│   │   │   └── viz
│   │   └── _vendor
│   │       └── packaging
│   ├── torch-2.2.0.dist-info
│   └── torchgen
│       ├── api
│       │   └── types
│       ├── dest
│       ├── executorch
│       │   └── api
│       │       └── types
│       ├── operator_versions
│       ├── packaged
│       │   ├── ATen
│       │   │   ├── native
│       │   │   └── templates
│       │   └── autograd
│       │       └── templates
│       ├── selective_build
│       └── static_runtime
└── referrers

443 directories, 1 file

@frostming
Copy link
Collaborator

frostming commented Feb 5, 2024 •
edited

It seems like the symlinks are followed when deleting the links from the current project. But unfortunately I still can't reproduce after experimenting with different cache methods. Does it work with other simpler packages? Try click

@fancyerii
Copy link
Author

It seems like the symlinks are followed when deleting the links from the current project. But unfortunately I still can't reproduce after experimenting with different cache methods. Does it work with other simpler packages? Try click

click is correctly installed with individual_symlink. It's correctedly imported:

(testcache-3.9) ubuntu@VM-4-12-ubuntu:/nas/lili/codes/pt/testcache$ python
Python 3.9.18 (main, Feb  2 2024, 13:57:14)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import click
>>> print(click.__file__)
/nas/lili/codes/pt/testcache/.venv/lib/python3.9/site-packages/click/__init__.py

And goto the .venv I can see it's linked to global cache:

(testcache-3.9) ubuntu@VM-4-12-ubuntu:/nas/lili/codes/pt/testcache/.venv/lib/python3.9/site-packages/click$ ll
total 8
drwxrwxr-x 3 ubuntu ubuntu 4096 Feb  5 10:48 ./
drwxrwxr-x 5 ubuntu ubuntu  131 Feb  5 10:48 ../
lrwxrwxrwx 1 ubuntu ubuntu   78 Feb  5 10:48 _compat.py -> /home/ubuntu/.cache/pdm/packages/click-8.1.7-py3-none-any/lib/click/_compat.py
lrwxrwxrwx 1 ubuntu ubuntu   75 Feb  5 10:48 core.py -> /home/ubuntu/.cache/pdm/packages/click-8.1.7-py3-none-any/lib/click/core.py
lrwxrwxrwx 1 ubuntu ubuntu   81 Feb  5 10:48 decorators.py -> /home/ubuntu/.cache/pdm/packages/click-8.1.7-py3-none-any/lib/click/decorators.py
lrwxrwxrwx 1 ubuntu ubuntu   81 Feb  5 10:48 exceptions.py -> /home/ubuntu/.cache/pdm/packages/click-8.1.7-py3-none-any/lib/click/exceptions.py
lrwxrwxrwx 1 ubuntu ubuntu   81 Feb  5 10:48 formatting.py -> /home/ubuntu/.cache/pdm/packages/click-8.1.7-py3-none-any/lib/click/formatting.py
lrwxrwxrwx 1 ubuntu ubuntu   78 Feb  5 10:48 globals.py -> /home/ubuntu/.cache/pdm/packages/click-8.1.7-py3-none-any/lib/click/globals.py
lrwxrwxrwx 1 ubuntu ubuntu   79 Feb  5 10:48 __init__.py -> /home/ubuntu/.cache/pdm/packages/click-8.1.7-py3-none-any/lib/click/__init__.py
lrwxrwxrwx 1 ubuntu ubuntu   77 Feb  5 10:48 parser.py -> /home/ubuntu/.cache/pdm/packages/click-8.1.7-py3-none-any/lib/click/parser.py
drwxrwxr-x 2 ubuntu ubuntu 4096 Feb  5 10:48 __pycache__/
lrwxrwxrwx 1 ubuntu ubuntu   76 Feb  5 10:48 py.typed -> /home/ubuntu/.cache/pdm/packages/click-8.1.7-py3-none-any/lib/click/py.typed
lrwxrwxrwx 1 ubuntu ubuntu   87 Feb  5 10:48 shell_completion.py -> /home/ubuntu/.cache/pdm/packages/click-8.1.7-py3-none-any/lib/click/shell_completion.py
lrwxrwxrwx 1 ubuntu ubuntu   83 Feb  5 10:48 _termui_impl.py -> /home/ubuntu/.cache/pdm/packages/click-8.1.7-py3-none-any/lib/click/_termui_impl.py
lrwxrwxrwx 1 ubuntu ubuntu   77 Feb  5 10:48 termui.py -> /home/ubuntu/.cache/pdm/packages/click-8.1.7-py3-none-any/lib/click/termui.py
lrwxrwxrwx 1 ubuntu ubuntu   78 Feb  5 10:48 testing.py -> /home/ubuntu/.cache/pdm/packages/click-8.1.7-py3-none-any/lib/click/testing.py
lrwxrwxrwx 1 ubuntu ubuntu   80 Feb  5 10:48 _textwrap.py -> /home/ubuntu/.cache/pdm/packages/click-8.1.7-py3-none-any/lib/click/_textwrap.py
lrwxrwxrwx 1 ubuntu ubuntu   76 Feb  5 10:48 types.py -> /home/ubuntu/.cache/pdm/packages/click-8.1.7-py3-none-any/lib/click/types.py
lrwxrwxrwx 1 ubuntu ubuntu   76 Feb  5 10:48 utils.py -> /home/ubuntu/.cache/pdm/packages/click-8.1.7-py3-none-any/lib/click/utils.py
lrwxrwxrwx 1 ubuntu ubuntu   82 Feb  5 10:48 _winconsole.py -> /home/ubuntu/.cache/pdm/packages/click-8.1.7-py3-none-any/lib/click/_winconsole.py

@fancyerii
Copy link
Author

I found the problem. the project is in a nfs, while the cache dir is not. So the system can't create symlink correctly. I have change the config like this:

cache_dir = /nas/lili/codes/pt/.cache/pdm
global_project.path = /nas/lili/codes/pt/.config/pdm/global-project
install.cache = True
install.cache_method = symlink_individual
venv.location = /nas/lili/codes/pt/.local/share/pdm/venvs

And it works now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
🤔 not enough info Requires more information to clarify the issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@fancyerii @frostming