[RFE] Cross repo pass/fail reporting

[NickLaMuro]

A comment from @lpichler

ManageIQ/manageiq#20776 (comment)

Tests are 🟢 ManageIQ/manageiq-cross_repo-tests#220

Got me thinking: "This would be excellent if the bot could do this for us!"

Approach 1

On new commits to a cross_repo build, we check on the status of a cross repo build and report back when it has completed (pass/fail), and delay said job if it hasn't completed.

This is probably the simplest solution, but also the biggest hack.

Approach 2

Add some form of web hook to the travis build to inform the associated PR of a build pass/failure. Not sure of the possibility of this, so this would require more research. At a minimum, we probably could be able to send a web request to the bot, or enable a direct status post from the cross_repo travis run on the success of the run.

[Fryguy]

Sounds kind of similar to #451 but specific to cross repo?

[NickLaMuro]

@Fryguy Bit different.

The idea for that one is that it is notifying gitter that there is a failure in one of the main branches, and takes a guess at maybe what PR caused it.

In this case, we are looking a particular cross_repo pull request ("run"), and reporting back to the PR that might have issued it. Similar code might be able to be shared, but the target we are watching and receiver of the notification are much more focused.

---

This is more for a reviewer of a particular PR knows that the cross repo tests that were run have completed as expected.

While mostly saving a quick jumping through a link, there is a scenario where this makes it much more useful:

Say, a given PR is open and has a cross-repo build associated with it, and the cross repo build happens and the bot pings on the calling PR that it passes. After some code review, the PR is rebased (or even just some extra fixes are added), but said changes would have actually broken the cross_repo build, a reviewer would at least be able to see that the confirmation of a working happened before the new changes were introduced.

A bit contrived, but I don't think far fetched either.

[Fryguy]

Oh ok...I was thinking more the tech needed to "monitor" would be the same as they both would watch Travis, though one is watching Travis runs for PRs, and the other for primary branches.

[agrare]

Maybe dreaming...but what about a Github Checks entry for the cross-repo test run instead of just a comment?

[NickLaMuro]

@agrare kinda what I lumped into “webhook” above, though I am admittedly unsure of the capabilities of that system. Specifically, unsure if you can dynamically add checks to a PR, since I am not sure we want to make every PR require a cross repo test.

[agrare]

👍 to not require for every PR, I'm assuming miq_bot could post to some API to add a check after starting the cross-repo-test but you're right that is an assumption

[NickLaMuro]

@agrare yeah, and it might be something where this is a feature that really might only need to be coded on the manageiq-cross_repo end, and there is nothing that the bot actually has to do to support this. But for now, figured I would at least open the idea up here to see what is possible.

Probably deserves a help wanted tag, though, since I don't plan on addressing it myself at he moment.

@miq-bot add_label help wanted

[Fryguy]

Now that we have our bot "public", we have a few more of these options for callbacks...we could even do webhooks instead of notifications for other things ;)

[kbrock]

whoa - we have a travis status and a code climate status - are you suggesting we could have a cross repo status?

[NickLaMuro]

@kbrock suggesting as a possibility, yes.