Profiler only generates xplane.pb (Tensorflow 2.11 & 2.12)

[HLneoh]

System information

Ubuntu 22.04

Virtual env #1

Virtual env #2

Description

Profiler only generates xplane.pb. Missing plugin files for Overview Page, Input Pipeline Analyzer, TensorFlow Stats, Trace Viewer, GPU Kernel Stats, Memory Profile Tool and Pod Viewer.

Results

Warning obtained as below when viewing the overview page.
No step marker observed and hence the step time is unknown. This may happen if (1) training steps are not instrumented (e.g., if you are not using Keras) or (2) the profiling duration is shorter than the step time. For (1), you need to add step instrumentation; for (2), you may try to profile longer.

[zzzaries]

Hi HLneoh, just want to check that if the tools (eg. overview page, tensorflow stats..etc) still works for you? As if they still works fine for you, this should be something expected. Because our recent change removed dependency on those materialized JSON tool files, but the data will be generated from xplane.pb on the fly directly.

[martintx]

Hi @HLneoh, I have an environment that is similar to your env2, but it seems that due to issues with protobuf, the profile page is not displayed at all. It would be greatly appreciated if you could share your protobuf version with me, tks!

[pritamdodeja]

This might have to do with location of profile data being different starting in 2.12 (can't test 2.11 because of cuda dependency). I was able to get this to work by copying the data manually with tensorflow v2.13.