Is it possible to calculate the number of operations per inference?

[Apisteftos]

Hello to everyone,

I need to calculate the throughput (TOPs) of an onnx model, but firstly I need to calculate the amount of the operations during the inference. Is any tool available and if not how can I estimate number of operations during the inference? I suppose that number is not static but dynamically during the inference. How is that possible?

[justinchuby]

Sounds like this may be relevant: https://github.com/ThanatosShinji/onnx-tool, however, the true number of operations depend on the runtime implementation. For example, onnxruntime does additional optimization on the graph before executing it.

[Apisteftos]

I did profiling with NVIDIA GPU A100 and used the pretrained models from onnx zoo, like faster RCNN. I am getting in the node 486 and 490 convolutional layers smaller runtime, which they should have longer runtime than other smaller convolution layers with smaller kernel filters, input and output feature map. I estimated the MACs according to onnx-tool and math background and in these two convolution layers I am getting almost 600 TFLOPS, over the Peak TF32 which is normally for FP32 operations 156TFLOPS according to the specs of the GPU. So why the throughput in the GPU is over the limits at this two nodes? Is any special operation?

[justinchuby]

It’s one possibility that the runtime optimized and simplified the graph for execution. So there aren’t as many operations done.

[Apisteftos]

Yes bus these nodes, layers should not extend the peak values of the GPU.