Research SLO1 problems

[majamassarini]

We still need an high SLO1 threshold (25 seconds or more) because of some spikes even though most of the times we can give to the user the first status check update in a couple of seconds.

It is not clear where we spend all the time; we should understand it and see if we can further improve and lower the threshold again.
Some suggestions are:

• we are waiting for the token from tokman (that maybe is waiting for an answer from github).
• we are waiting for the handler pre-check, as an example for the copr handler in the pre-check we call the copr service and this can slow us down.

[lsm5]

@majamassarini containers/buildah#5388 looks like another issue where packit tasks haven't shown up in at least a minute after pushing to the PR. There are no changes to the packit config but in cases where there are and this issue occurs, the end user is left wondering whatsup.

[lsm5]

containers/crun#1425 another one where it's been 42 mins since commit push and no packit tasks appeared.

EDIT: correct PR link.

EDIT 2: This may have been because of issues during redeployment of packit service. A new push triggered packit tasks instantaneously.

[nforro]

Could be also related to packit/deployment#548.

[lachmanfrantisek]

Take a look at #1954

[lbarcziova]

There were few improvements implemented, see, so for now I am putting this back to backlog and if we start seeing issues again, let's revisit.