-
Notifications
You must be signed in to change notification settings - Fork 430
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Conformance] Allow for reporting results before fully achieving conformance #3036
Comments
I think that it might be helpful, for this and other unforeseen problems with conformance reports, to have a spot in the conformance suite and the resultant output that allows the suite to say "There are issues with this report, that causes it to not be conformant for some reason that's not covered by the suite". I think that, in general, we should encourage implementations to submit partial conformance results, or other results that say "Our implementation is working on conformance but it's not done yet". This allows for:
Using an override feature like this would require:
In terms of more concrete changes required we would need to add:
Again, just a quick proposal for discussion, not intended to be binding. |
After discussing #3021 and #3025, I agree with you it makes sense to allow implementations to submit partial reports, for all the good reasons listed by @youngnick as well. When it comes to the report API, we already have the gateway-api/conformance/reports/v1.0.0/azure-application-gateway-for-containers/v1.0.0-report.yaml Lines 12 to 18 in a416c22
If we need to include some information on the profile to allow implementations to say "I am not supporting this or that because the suite lacks this functionality that I need", we have the gateway-api/conformance/apis/v1/profilereport.go Lines 26 to 28 in 7352409
For what concerns us allowing implementations to submit partial results, what's currently missing is the explicit allowance to do so in our docs, as we are currently saying that partial reports are not allowed: gateway-api/conformance/reports/README.md Lines 123 to 127 in 7352409
One thing I think we should pay attention to is that we should require implementations to always write the current status of their support level in the README. If there are any workarounds the projects need to use because of implementation/suite limitations, they need to be listed here, along with the related issues. |
What I meant was that, in the case that there's something that we don't catch in the suite that renders the result not complete (like the use of annotations in #3021 as part of their suite, which means that the reproduction instructions are not runnable), there needs to be a way in the conformance YAML for implementation maintainers to say "although we're passing conformance, something else is wrong, and the report doesn't count". I don't think this is the last time this will happen. |
What would you like to be added:
A way to display "in progress" or "partial" conformance.
Why this is needed:
In #3021 and #3025 we ran into situations where an implementation was not able to support a core feature in a conformance profile. Despite that meaning that the implementation is not conformant yet, it is likely still helpful for the broader ecosystem if they are able to report their current status, especially given the incoming work to display results in #2874. This could also be helpful for other implementations to report their current status even if they haven't reached 100% support of core features yet.
Note: This is meant to be the start of a discussion, this is not ready to be worked on yet.
The text was updated successfully, but these errors were encountered: