Address MDN performance issue

[atopal]

Summary
time to interactive examples for 90th percentile has regressed by .3s some time after February 7.

Steps To Reproduce (STR)
[see 90th percentile data](MDN time to interactive examples for 90th percentile has regressed by .3s)

Actual behavior
90th percentile is at 3.7s

Expected behavior
90th percentile should be at 3.4s

Additional context
Is there anything else we should know?

[callahad]

Here are the deployments that happened around the time that this issue cropped up:

• 2020-01-31 0a74227
• 2020-02-06 f48f3fd
• 2020-02-11 c41c71b
• 2020-02-12 9aea0ed
• 2020-02-13 7778679
• 2020-02-20 79abfd3

The issue definitely was not happening prior to February 6th, and definitely was happening by February 13th. If the regression was caused by a change to our code, it must have been a change in this range: f48f3fd...7778679

[callahad]

That range covers the following individual commits. I've struck through ones that absolutely could not have contributed to this regression.

• f48f3fd update sub-modules (update sub-modules #6431)
• 669974e Make BCD Signaling section look more like designs (Make BCD Signaling section look more like designs  #6382)
• 4e7db43 Fix dropdown menu position on RTL languages (Fix dropdown menu position on RTL languages  #6426)
• d477a40 Add stripe subscription to user profile edit view (Add stripe subscription to user profile edit view #6269)
• 37b3e54 Fix deprecation warnings for logger.warn() (Fix deprecation warnings for logger.warn() #6434)
• 727f0d9 Update pytest (3.1.3 -> 3.6.4) (Update pytest (3.1.3 -> 3.6.4) #6433)
• 53b7513 add black as dev dependency + routine updates (add black as dev dependency #6436)
• 37b5907 Update pytest-django (3.1.2 -> 3.4.8) (Update pytest-django (3.1.2 -> 3.4.8) #6438)
• 5c5646f Fix two deprecation warnings (Fix two deprecation warnings #6439)
• 345c541 Use Black to format Python code (Use Black to format Python code #6447)
• 566a333 Straightforward Python dependency updates (Straightforward Python dependency updates #6456)
• 43113ad Update coverage (4.5.4 -> 5.0.3) (Update coverage (4.5.4 -> 5.0.3) #6458)
• 0a9abdc Align Youtube video embeds with content (Align Youtube video embeds with content #6452)
• 1b1bf28 Straightforward Python dependency updates (Straightforward Python dependency updates #6459)
• 22761e2 Upgrade pytest (3.10.1 -> 4.6.9) (Upgrade pytest (3.10.1 -> 4.6.9) #6460)
• 11a2a36 Remove unused custom pytest markers (Remove unused custom pytest markers #6461)
• 2ccdedd don't index MDN/Doc_status (don't index MDN/Doc_status #6448)
• c41c71b update sub-modules (update sub-modules #6465)
• e444e9c Include Django migrations in linting (Include Django migrations in linting #6469)
• 669ec0a Switch to async GA tag (Switch to async GA tag #6463)
• 82bffcd Add unified git-style diff feature to revise script (Add unified git-style diff feature to revise script #6466)
• f93f134 Update pytest and pytest-django (Update pytest and pytest-django #6467)
• c1d4fc2 Remove deprecated wiki test (Remove deprecated wiki test #6476)
• ec74256 Remove jQueryUI, interest and expertise fields - Profile cleanup (Remove jQueryUI, interest and expertise fields - Profile cleanup #6446)
• 9aea0ed link wiki's sign-in to read-only site (link wiki's sign-in to read-only site #6435)
• a0a9b25 minimum 10 characters on bcd signal explanation (minimum 10 characters on bcd signal explanation #6143)
• 3df115a ability to edit whose parent is deleted (ability to edit whose parent is deleted #6455)
• a53fd4b Small Travis improvements (Small Travis improvements #6477)
• 2cae6b7 Update to the latest Node LTS (12.13.1 -> 12.16.0) (Update to the latest Node LTS (12.13.1 -> 12.16.0) #6490)
• 282bdb3 don't search on user tags (don't search on user tags #6489)
• 16529cb Fix .editorconfig (Fix .editorconfig #6484)
• a0918bb Do not show avatar transition on signup (Do not show avatar transition on signup #6487)
• 7778679 update sub-modules (update sub-modules #6493)

Of particular note, we know that 669ec0a (Switch to async GA tag) cannot be at issue because we reverted it and re-deployed last Wednesday, and the performance regression is still present in our data.

[callahad]

That leaves us with the much more manageable list of the following 9 commits that we can't trivially exclude:

• 669974e Make BCD Signaling section look more like designs (Make BCD Signaling section look more like designs  #6382)
• 4e7db43 Fix dropdown menu position on RTL languages (Fix dropdown menu position on RTL languages  #6426)
• d477a40 Add stripe subscription to user profile edit view (Add stripe subscription to user profile edit view #6269)
• 2ccdedd don't index MDN/Doc_status (don't index MDN/Doc_status #6448)
• 82bffcd Add unified git-style diff feature to revise script (Add unified git-style diff feature to revise script #6466)
• ec74256 Remove jQueryUI, interest and expertise fields - Profile cleanup (Remove jQueryUI, interest and expertise fields - Profile cleanup #6446)
• 9aea0ed link wiki's sign-in to read-only site (link wiki's sign-in to read-only site #6435)
• a0a9b25 minimum 10 characters on bcd signal explanation (minimum 10 characters on bcd signal explanation #6143)
• 3df115a ability to edit whose parent is deleted (ability to edit whose parent is deleted #6455)

[atopal]

Confirming that yes, #6601 didn't have a performance impact. Thanks Dan for divining into this and making this issue much more actionable.

What's the suggestion for next steps here? Do we pick the things that that are easily revertible and try them first? We need at least 1 full day of data to verify the impact.

[callahad]

There's a pretty good chance we could exclude half of the commits above, too, but they just require a slightly closer read than that first pass :)

I think the next step is to have two devs hop on Zoom, synchronously read those commits, and come up with a prioritized list of ones we should try reverting.

Unfortunately, this is unlikely to happen this week: Django 2 has been merged and we need to get it deployed ASAP so we're ready for the security patch that's landing Wednesday morning. And then we'll probably need a few days to see if Django 2 itself changes our baseline performance before we can start testing those commits 🙁

[atopal]

That schedule works for me. It's important we find the issue and fix it, so we don't end up compounding performance regressions, but it's not time sensitive.

[callahad]

Gregor and I spent half an hour reading each commit line-by-line, and there's nothing that we can conceive of impacting frontend performance in any way.

The Stripe (d477a40) and jQuery UI (ec74256) commits were the only ones to really touch the frontend in any meaningful way, but both are constrained to the Wiki.

At this point, we think the regression is unlikely to have been caused by code changes in Kuma.

[callahad]

I don't know where else to look, but perhaps existing Waffle flags or settings were changed in the Django Admin, maybe there was a new deploy of the Interactive Examples service? Or something else entirely?

[atopal]

Hmm, we've had something in the soapbox for the last 2 weeks roughly. Maybe that?

[callahad]

Could be! Is that scheduled to end any time soon?

[callahad]

NOTE: This is measuring Time to Interactive Examples, a custom metric that we track, not the more generic "TTI" metric in, say, Lighthouse.

[tobinmori]

@atopal will check on soapbox and turn it off wed 3/11.

[atopal]

okay, we have an alternative explanation: Corona