Clear one mapped task's state via the REST API

[uranusjr]

First part of #24699 (the other would be to fix the web UI). This adds the ability to clear one specific mapped task’s state via the REST API.

This is a bit more complicated than I’d desired. Currently the clearTaskInstance endpoint accepts a task_ids field that takes a list of strings to filter out specific tasks. To make this support map indexes, I first tried the followings:

{
    "task_ids": [
        "task_1",  // Clearing the entire task, multiple tis if mapped.
        ["task_2", 1],  // Clearing one specific ti.
    ],
}

But this does not work since it requires using the new prefixItems key to specify an array as tuple, which was added in Open API 3.1, but we’re still on 3.0.

Next I tried

{
    "task_ids": [
        "task_1",
        {"task_id": "task_2", "map_index": 1},
    ],
}

I don’t like duplicating task_id inside the nested object, but even this does not work, since Open API 3.0 also forbids heterogeneous arrays.

So in the end I added a new field tasks, that only accepts the object form, and merge task_ids and tasks in the backend instead. In the future, once we are able to use heterogeneous arrays, we could merge the two fields back into one property to clean up the interface. For now though, this is what we need to do.

Example request now:

{
    ...,
    "task_ids": ["task_1", ...],
    "tasks": [
        {"task_id": "task_2", "map_index": 1},
        {"task_id": "task_3"},  // Allowed; same as adding task_3 to task_ids.
        ...,
    ]
}

[ashb]

Oh wow and OpenAPI 3.1 seems to not have good support in the tools we are using spec-first/connexion#1396 -> swagger-api/swagger-ui#5891

[ashb]

I wonder (again) if it's worth moving to https://github.com/p1c2u/openapi-core which is quicker on supporting things, even if Connexion is not abandonware any longer.

[uranusjr]

We’ll probably need so do a survey on client generator options. IIRC at least the Java client is not doing very well with supporting new things either, and it doesn’t really help if we start providing a schema that clients can’t use…

[ashb]

I mean honestly my caring of Java is pretty low, sorry-notsorry 🙊

[ashb]

That said it does seem that Support for 3.1 in tools in the OpenAPI ecosystem is very poor, and that to some extent 3.1 should have been 4.0 (at least if they were following SemVer, but I don't know if they claim to) as it's a small breaking change to how some schemas are defined.

[ashb]

Is it worth mentioning (somewhere? In the example? Is that possible) that you can pass both tasks and task_ids at the same time to this endpoint?

[uranusjr]

Is this meant to be communicated to the users? If not, YAML can contain comments.

[ashb]

Yes, was thinking about communicating this to users via the rendered API docs, on https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html

[ashb]

Code and approach looks good to me, though probably worth leaving a comment somewhere in the code as to why they API is designed the way it is (limitations of OpenAPI 3.0)

(That or we could make the API less strict and validate it at the server side, but I think that's probably not worth doing)

[uranusjr]

That or we could make the API less strict and validate it at the server side

Right now the API is very loose and can accept either tasks, task_ids, or even both. And we can still continue to accept task_ids (albeit deprecated) when we move to OpenAPI 3.1. What is the validation we might need you have in mind?

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.