Regression on thunder/benchmarks/targets.py

[IvanYashchuk]

🐛 Bug

Changes added in b1f4470 (#279) introduced a regression on the benchmarks, for example,

pytest thunder/benchmarks/targets.py -vs -k 'test_nanogpt_gpt2_fwd[thunder]'

has a median time of about 4.3ms before the commit and 6.15ms with b1f4470.

Many other benchmark cases from benchmarks/targets.py are affected by the change.

git revert b1f447022b0732e83c11661c30746568280834f7 on the current main restores the performance.

@t-vi, do you know what could be causing the slowdown?

cc @apaz-cli @crcrpar

[t-vi]

For my own reference, on my reference machine:

b1f447022b0732e83c11661c30746568280834f7^
test_nanogpt_gpt2_fwd[thunder]     13.0015  13.6725  13.1576  0.2891  13.0281  0.2124       1;1  76.0015       5           1

b1f447022b0732e83c11661c30746568280834f7
test_nanogpt_gpt2_fwd[thunder]     14.7374  15.5216  14.9018  0.3466  14.7441  0.2112       1;1  67.1059       5           1

4663b87e67955edec371c6f37a763a7ec358d835 (current main)
test_nanogpt_gpt2_fwd[thunder]     14.7783  15.2227  14.8889  0.1873  14.8131  0.1166       1;1  67.1642       5           1

pre-populating overrides will recover the majority of the slowdown
test_nanogpt_gpt2_fwd[thunder]     13.7017  14.1694  13.8310  0.1938  13.7548  0.1872       1;0  72.3014       5           1

caching submodules (in addition to pre-populating overrides) will actually speed up things a tiny bit
test_nanogpt_gpt2_fwd[thunder]     12.8832  13.1997  12.9568  0.1361  12.8997  0.0922       1;1  77.1794       5           1

[t-vi]

Has been reverted, so I'm reopening.