UI error when clicking on graph view when a task has pod overrides

[gfeldman]

Apache Airflow version

2.4.0

What happened

When I click on the graph view or the gaant view for a DAG that has a task with pod_overrides, I get

Something bad has happened.

Airflow is used by many users, and it is very likely that others had similar problems and you can easily find
a solution to your problem.

Consider following these steps:

  * gather the relevant information (detailed logs with errors, reproduction steps, details of your deployment)

  * find similar issues using:
     * [GitHub Discussions](https://github.com/apache/airflow/discussions)
     * [GitHub Issues](https://github.com/apache/airflow/issues)
     * [Stack Overflow](https://stackoverflow.com/questions/tagged/airflow)
     * the usual search engine you use on a daily basis

  * if you run Airflow on a Managed Service, consider opening an issue using the service support channels

  * if you tried and have difficulty with diagnosing and fixing the problem yourself, consider creating a [bug report](https://github.com/apache/airflow/issues/new/choose).
    Make sure however, to include all relevant details and results of your investigation so far.

Python version: 3.8.14
Airflow version: 2.4.0
Node: airflow-webserver-6d4d7d5ccd-qc2x5
-------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 2525, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 1822, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 1820, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 1796, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/auth.py", line 47, in decorated
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/decorators.py", line 118, in view_func
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/decorators.py", line 81, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/session.py", line 75, in wrapper
    return func(*args, session=session, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/views.py", line 2810, in graph
    return self.render_template(
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/views.py", line 541, in render_template
    return super().render_template(
  File "/home/airflow/.local/lib/python3.8/site-packages/flask_appbuilder/baseviews.py", line 322, in render_template
    return render_template(
  File "/home/airflow/.local/lib/python3.8/site-packages/flask/templating.py", line 147, in render_template
    return _render(app, template, context)
  File "/home/airflow/.local/lib/python3.8/site-packages/flask/templating.py", line 130, in _render
    rv = template.render(context)
  File "/home/airflow/.local/lib/python3.8/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/home/airflow/.local/lib/python3.8/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/templates/airflow/graph.html", line 21, in top-level template code
    {% from 'appbuilder/loading_dots.html' import loading_dots %}
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/templates/airflow/dag.html", line 37, in top-level template code
    {% set execution_date_arg = request.args.get('execution_date') %}
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/templates/airflow/main.html", line 21, in top-level template code
    {% from 'airflow/_messages.html' import show_message %}
  File "/home/airflow/.local/lib/python3.8/site-packages/flask_appbuilder/templates/appbuilder/baselayout.html", line 2, in top-level template code
    {% import 'appbuilder/baselib.html' as baselib %}
  File "/home/airflow/.local/lib/python3.8/site-packages/flask_appbuilder/templates/appbuilder/init.html", line 50, in top-level template code
    {% block tail %}
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/templates/airflow/graph.html", line 137, in block 'tail'
    let taskInstances = {{ task_instances|tojson }};
  File "/home/airflow/.local/lib/python3.8/site-packages/jinja2/filters.py", line 1688, in do_tojson
    return htmlsafe_json_dumps(value, dumps=dumps, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/jinja2/utils.py", line 658, in htmlsafe_json_dumps
    dumps(obj, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/flask/json/provider.py", line 230, in dumps
    return json.dumps(obj, **kwargs)
  File "/usr/local/lib/python3.8/json/__init__.py", line 234, in dumps
    return cls(
  File "/usr/local/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/home/airflow/.local/lib/python3.8/site-packages/flask/json/provider.py", line 122, in _default
    raise TypeError(f"Object of type {type(o).__name__} is not JSON serializable")
TypeError: Object of type V1Pod is not JSON serializable

What you think should happen instead

The UI should render the dag visualization.

How to reproduce

• Add a pod_override to a task
• Run the task
• click on the graph view

Operating System

Debian GNU/Linux 11 (bullseye) docker image

Versions of Apache Airflow Providers

apache-airflow-providers-airbyte==3.1.0
apache-airflow-providers-amazon==5.1.0
apache-airflow-providers-apache-spark==3.0.0
apache-airflow-providers-celery==3.0.0
apache-airflow-providers-cncf-kubernetes==4.3.0
apache-airflow-providers-common-sql==1.2.0
apache-airflow-providers-datadog==3.0.0
apache-airflow-providers-docker==3.1.0
apache-airflow-providers-elasticsearch==4.2.0
apache-airflow-providers-ftp==3.1.0
apache-airflow-providers-google==8.3.0
apache-airflow-providers-grpc==3.0.0
apache-airflow-providers-hashicorp==3.1.0
apache-airflow-providers-http==4.0.0
apache-airflow-providers-imap==3.0.0
apache-airflow-providers-jira==3.0.1
apache-airflow-providers-microsoft-azure==4.2.0
apache-airflow-providers-odbc==3.1.1
apache-airflow-providers-pagerduty==3.0.0
apache-airflow-providers-postgres==5.2.1
apache-airflow-providers-redis==3.0.0
apache-airflow-providers-salesforce==5.1.0
apache-airflow-providers-sendgrid==3.0.0
apache-airflow-providers-sftp==4.0.0
apache-airflow-providers-slack==5.1.0
apache-airflow-providers-sqlite==3.2.1
apache-airflow-providers-ssh==3.1.0
apache-airflow-providers-tableau==3.0.1

Deployment

Official Apache Airflow Helm Chart

Deployment details

official helm

Anything else

every time after the dag is run

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[gfeldman]

The pod override looks like

k8s.V1Pod(
        spec=k8s.V1PodSpec(
            containers=[
                k8s.V1Container(
                    name="base",
                    resources=k8s.V1ResourceRequirements(
                        limits=resources_dict,
                        requests=resources_dict,
                    ),
                ),
            ],
            tolerations=tolerations,
            node_selector=node_selector,
        )
    )

where I import from kubernetes.client import models as k8s

[hterik]

Upgraded from 2.3.3 to 2.4 and experience the same thing when trying to open dagruns produced from 2.3.3 in grid view. But if i wipe the DB and just start fresh runs it works.

[gfeldman]

I started from a clean database and still running into the issue

[ashb]

(A related question is why on earth is graph view even dumping that override field? It doesn't need it)

[ashb]

Fixed by #26554

[ltutar]

We also have this issue. Any idea when this fix will be released?