You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When using the PyTorch Profiler with TensorBoard, the generated trace files are too large (e.g., 1 ~2 GB for just 10 steps), causing TensorBoard to crash or hang.
To reproduce
Steps to reproduce the behavior:
Set up a PyTorch Lightning Trainer with the following profiler configuration:
Run the training for a few steps.
The produced trace file size becomes excessively large.
Attempt to open with TensorBoard.
TensorBoard crashes or becomes unresponsive when viewing in the trace or memory tab.
Expected behavior
The trace file should be of manageable size, or there should be a method to limit or chunk the file size to prevent such issues. Additionally, TensorBoard should be able to handle large trace files more gracefully.
I have tried 1) Disabled profile_memory and 2) Reducing active steps in the profiler schedule. However, it seems like the trace file is always more than 1GB, which I can't view on tensorbaord. Can someone suggest some alternatives for profiling ?
Given the challenges with the current profiler, I am looking for alternative methods or tools to view the profile my PyTorch Lightning training. Suggestions or recommendations would be highly appreciated.
The text was updated successfully, but these errors were encountered:
Hi @kvignesh1420@idontkonwher@UTokyoChenYe ,I haven't found a good solution yet. My current approach involves minimizing the file size as much as possible and reducing the number of active/warm-up steps. Alternatively, I opt for using a simpler profiler for performance profiling.
When using the PyTorch Profiler with TensorBoard, the generated trace files are too large (e.g., 1 ~2 GB for just 10 steps), causing TensorBoard to crash or hang.
To reproduce
Steps to reproduce the behavior:
Run the training for a few steps.
The produced trace file size becomes excessively large.
Attempt to open with TensorBoard.
TensorBoard crashes or becomes unresponsive when viewing in the trace or memory tab.
Expected behavior
The trace file should be of manageable size, or there should be a method to limit or chunk the file size to prevent such issues. Additionally, TensorBoard should be able to handle large trace files more gracefully.
Environment:
PyTorch Lightning Version: 1.9.0
Python version: 3.9.18
I have tried 1) Disabled profile_memory and 2) Reducing active steps in the profiler schedule. However, it seems like the trace file is always more than 1GB, which I can't view on tensorbaord. Can someone suggest some alternatives for profiling ?
Given the challenges with the current profiler, I am looking for alternative methods or tools to view the profile my PyTorch Lightning training. Suggestions or recommendations would be highly appreciated.
The text was updated successfully, but these errors were encountered: