Triton installation not found. #125093

mfbalin · 2024-04-27T01:22:27Z

🐛 Describe the bug

Using torch.compile with a GNN model on the 20240426 nightly build installed via pip raises Triton installation not found error.

Error logs

Traceback (most recent call last):
File "/home/mfbalin/dgl-1/examples/sampling/graphbolt/lightning/../pyg/node_classification_advanced.py", line 446, in
main()
File "/home/mfbalin/dgl-1/examples/sampling/graphbolt/lightning/../pyg/node_classification_advanced.py", line 428, in main
train(train_dataloader, valid_dataloader, num_classes, model, args.device)
File "/home/mfbalin/dgl-1/examples/sampling/graphbolt/lightning/../pyg/node_classification_advanced.py", line 236, in train
out = model(minibatch.sampled_subgraphs, node_features)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 403, in _fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 977, in catch_errors
return callback(frame, cache_entry, hooks, frame_state, skip=1)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 411, in _convert_frame_assert
return _compile(
^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_utils_internal.py", line 70, in wrapper_function
return function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 700, in _compile
guarded_code = compile_inner(code, one_graph, hooks, transform)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/utils.py", line 268, in time_wrapper
r = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 568, in compile_inner
out_code = transform_code_object(code, transform)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/bytecode_transformation.py", line 1116, in transform_code_object
transformations(instructions, code_options)
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 173, in _fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 515, in transform
tracer.run()
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/symbolic_convert.py", line 2241, in run
super().run()
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/symbolic_convert.py", line 875, in run
while self.step():
^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/symbolic_convert.py", line 790, in step
self.dispatch_table[inst.opcode](self, inst)
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/symbolic_convert.py", line 2398, in RETURN_VALUE
self._return(inst)
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/symbolic_convert.py", line 2383, in _return
self.output.compile_subgraph(
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/output_graph.py", line 1068, in compile_subgraph
self.compile_and_call_fx_graph(tx, list(reversed(stack_values)), root)
File "/usr/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/output_graph.py", line 1285, in compile_and_call_fx_graph
compiled_fn = self.call_user_compiler(gm)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/utils.py", line 268, in time_wrapper
r = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/output_graph.py", line 1376, in call_user_compiler
raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/output_graph.py", line 1357, in call_user_compiler
compiled_fn = compiler_fn(gm, self.example_inputs())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/dynamo/repro/after_dynamo.py", line 127, in debug_wrapper
compiled_gm = compiler_fn(gm, example_inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/dynamo/repro/after_dynamo.py", line 127, in debug_wrapper
compiled_gm = compiler_fn(gm, example_inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/init.py", line 1742, in call
return compile_fx(model, inputs, config_patches=self.config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 1450, in compile_fx
return aot_autograd(
^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/backends/common.py", line 65, in compiler_fn
cg = aot_module_simplified(gm, example_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_functorch/aot_autograd.py", line 958, in aot_module_simplified
compiled_fn = create_aot_dispatcher_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/utils.py", line 268, in time_wrapper
r = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_functorch/aot_autograd.py", line 685, in create_aot_dispatcher_function
compiled_fn = compiler_fn(
^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 470, in aot_wrapper_dedupe
return compiler_fn(flat_fn, leaf_flat_args, aot_config, fw_metadata=fw_metadata)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 672, in aot_wrapper_synthetic_base
return compiler_fn(flat_fn, flat_args, aot_config, fw_metadata=fw_metadata)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py", line 434, in aot_dispatch_autograd
compiled_fw_func = aot_config.fw_compiler(fw_module, adjusted_flat_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/utils.py", line 268, in time_wrapper
r = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 1354, in fw_compiler_base
return inner_compile(
^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/repro/after_aot.py", line 83, in debug_wrapper
inner_compiled_fn = compiler_fn(gm, example_inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_inductor/debug.py", line 304, in inner
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/utils.py", line 268, in time_wrapper
r = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 482, in compile_fx_inner
compiled_graph = fx_codegen_and_compile(
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 775, in fx_codegen_and_compile
compiled_fn = graph.compile_to_fn()
^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_inductor/graph.py", line 1622, in compile_to_fn
return self.compile_to_module().call
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/utils.py", line 268, in time_wrapper
r = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_inductor/graph.py", line 1565, in compile_to_module
self.codegen_with_cpp_wrapper() if self.cpp_wrapper else self.codegen()
^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_inductor/graph.py", line 1517, in codegen
self.scheduler = Scheduler(self.buffers)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/utils.py", line 268, in time_wrapper
r = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_inductor/scheduler.py", line 1279, in init
self.nodes = [self.create_scheduler_node(n) for n in nodes]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_inductor/scheduler.py", line 1371, in create_scheduler_node
return SchedulerNode(self, node)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_inductor/scheduler.py", line 700, in init
self._compute_attrs()
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_inductor/scheduler.py", line 711, in _compute_attrs
group_fn = self.scheduler.get_backend(self.node.get_device()).group_fn
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_inductor/scheduler.py", line 2436, in get_backend
self.backends[device] = self.create_backend(device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_inductor/scheduler.py", line 2428, in create_backend
raise RuntimeError(
torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
RuntimeError: Cannot find a working triton installation. More information on installing Triton can be found at https://github.com/openai/triton

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information

You can suppress this exception and fall back to eager by setting:
import torch._dynamo
torch._dynamo.config.suppress_errors = True

Minified repro

python ../pyg/node_classification_advanced.py --torch-compile

Build DGL from source

https://github.com/dmlc/dgl/blob/master/examples/sampling/graphbolt/pyg/node_classification_advanced.py

Versions

Collecting environment information...
PyTorch version: 2.4.0.dev20240426+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 24.04 LTS (x86_64)
GCC version: (Ubuntu 13.2.0-23ubuntu4) 13.2.0
Clang version: Could not collect
CMake version: version 3.28.3
Libc version: glibc-2.39

Python version: 3.12.3 (main, Apr 10 2024, 05:33:47) [GCC 13.2.0] (64-bit runtime)
Python platform: Linux-5.15.146.1-microsoft-standard-WSL2-x86_64-with-glibc2.39
Is CUDA available: True
CUDA runtime version: 12.4.131
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 4090
Nvidia driver version: 552.22
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 48 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Vendor ID: AuthenticAMD
Model name: AMD Ryzen 9 5950X 16-Core Processor
CPU family: 25
Model: 33
Thread(s) per core: 2
Core(s) per socket: 16
Socket(s): 1
Stepping: 2
BogoMIPS: 6800.06
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr arat npt nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmload umip vaes vpclmulqdq rdpid fsrm
Virtualization: AMD-V
Hypervisor vendor: Microsoft
Virtualization type: full
L1d cache: 512 KiB (16 instances)
L1i cache: 512 KiB (16 instances)
L2 cache: 8 MiB (16 instances)
L3 cache: 32 MiB (1 instance)
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Mitigation; safe RET, no microcode
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected

Versions of relevant libraries:
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.26.4
[pip3] torch==2.4.0.dev20240426+cu121
[pip3] torch_geometric==2.5.3
[pip3] torchaudio==2.2.0.dev20240426+cu121
[pip3] torchdata==0.7.1
[pip3] torchmetrics==1.3.2
[pip3] torchvision==0.19.0.dev20240426+cu121
[conda] Could not collect

cc @seemethere @malfet @osalpekar @atalman @ezyang @msaroufim @bdhirsh @anijain2305 @chauhang

The text was updated successfully, but these errors were encountered:

jbschlosser · 2024-04-29T16:00:31Z

Have you tried installing Triton via pip? e.g. for CUDA 11.7:

pip install torchtriton --extra-index-url "https://download.pytorch.org/whl/nightly/cu117"

(from the docs here)

Leaving open because I believe Triton is intended to be included in our binaries.

mfbalin · 2024-05-06T01:40:05Z

Have you tried installing Triton via pip? e.g. for CUDA 11.7:
pip install torchtriton --extra-index-url "https://download.pytorch.org/whl/nightly/cu117"
(from the docs here)

Leaving open because I believe Triton is intended to be included in our binaries.

pip install torchtriton --extra-index-url "https://download.pytorch.org/whl/nightly/cu121"
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/nightly/cu121
ERROR: Could not find a version that satisfies the requirement torchtriton (from versions: none)
ERROR: No matching distribution found for torchtriton

mfbalin · 2024-05-06T01:46:00Z

Could this be due to me using Python 3.12? Ubuntu 24.04 ships with it.

mfbalin · 2024-05-07T03:46:04Z

#120233 is the main issue thread I think.

williamwen42 · 2024-05-08T17:48:33Z

Yeah, we're not packaging triton with the 3.12 binaries yet - this is WIP

mfbalin added the oncall: pt2 label Apr 27, 2024

jbschlosser added module: binaries Anything related to official binaries that we release to users module: build Build system issues triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module labels Apr 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Triton installation not found. #125093

Triton installation not found. #125093

mfbalin commented Apr 27, 2024 •

edited by pytorch-bot bot

jbschlosser commented Apr 29, 2024

mfbalin commented May 6, 2024

mfbalin commented May 6, 2024

mfbalin commented May 7, 2024

williamwen42 commented May 8, 2024

Triton installation not found. #125093

Triton installation not found. #125093

Comments

mfbalin commented Apr 27, 2024 • edited by pytorch-bot bot

🐛 Describe the bug

Error logs

Minified repro

Versions

jbschlosser commented Apr 29, 2024

mfbalin commented May 6, 2024

mfbalin commented May 6, 2024

mfbalin commented May 7, 2024

williamwen42 commented May 8, 2024

mfbalin commented Apr 27, 2024 •

edited by pytorch-bot bot