Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error rendering template: '_GeneratorContextManager' object has no attribute 'query' #39162

Open
1 of 2 tasks
raphaelauv opened this issue Apr 22, 2024 · 6 comments
Open
1 of 2 tasks

Error rendering template: '_GeneratorContextManager' object has no attribute 'query' #39162

raphaelauv opened this issue Apr 22, 2024 · 6 comments
Labels
area:core area:dynamic-task-mapping AIP-42 Can't Reproduce The problem cannot be reproduced kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet pending-response stale Stale PRs per the .github/workflows/stale.yml policy file

Comments

@raphaelauv
Copy link
Contributor

raphaelauv commented Apr 22, 2024 •
edited

Apache Airflow version

2.9.0

What happened?

on a KPO mapped task , the rendered template fail

Screenshot from 2024-04-22 09-46-17

[2024-04-22T07:38:22.812+0000] {app.py:1744} ERROR - Exception on /extra_links [GET]
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 2529, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/www/auth.py", line 252, in decorated
    return _has_access(
           ^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/www/auth.py", line 163, in _has_access
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/session.py", line 79, in wrapper
    return func(*args, session=session, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/www/views.py", line 3111, in extra_links
    url = task.get_extra_links(ti, link_name)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/abstractoperator.py", line 493, in get_extra_links
    return link.get_link(self.unmap(None), ti_key=ti.key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/operators/trigger_dagrun.py", line 69, in get_link
    return build_airflow_url_with_query(query)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/helpers.py", line 253, in build_airflow_url_with_query
    return flask.url_for(f"Airflow.{view}", **query)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/helpers.py", line 256, in url_for
    return current_app.url_for(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 2034, in url_for
    return self.handle_url_build_error(error, endpoint, values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 2023, in url_for
    rv = url_adapter.build(  # type: ignore[union-attr]
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/werkzeug/routing/map.py", line 917, in build
    raise BuildError(endpoint, values, method, self)
werkzeug.routing.exceptions.BuildError: Could not build url for endpoint 'Airflow.grid' with values ['base_date']. Did you forget to specify values ['dag_id']?
[2024-04-22T07:39:41.734+0000] {base.py:84} INFO - Using connection ID 'XXXXXXX' for task execution.
[2024-04-22T07:39:41.735+0000] {connection_wrapper.py:384} INFO - AWS Connection (conn_id='XXXXXXX', conn_type='aws') credentials retrieved from login and password.
/home/airflow/.local/lib/python3.12/site-packages/airflow/www/views.py:1746 RemovedInAirflow3Warning:
            This attribute is deprecated.
            Please use `airflow.models.taskinstance.TaskInstance.get_previous_start_date` method.
            
/home/airflow/.local/lib/python3.12/site-packages/airflow/www/views.py:1746 RemovedInAirflow3Warning:
            This attribute is deprecated.
            Please use `airflow.models.taskinstance.TaskInstance.get_previous_ti` method.
            
[2024-04-22T07:44:28.101+0000] {base.py:84} INFO - Using connection ID 'XXXXXXX' for task execution.
[2024-04-22T07:44:28.103+0000] {connection_wrapper.py:384} INFO - AWS Connection (conn_id='XXXXXXX', conn_type='aws') credentials retrieved from login and password.
/home/airflow/.local/lib/python3.12/site-packages/airflow/www/views.py:1746 RemovedInAirflow3Warning:
            This attribute is deprecated.
            Please use `airflow.models.taskinstance.TaskInstance.get_previous_start_date` method.
            
/home/airflow/.local/lib/python3.12/site-packages/airflow/www/views.py:1746 RemovedInAirflow3Warning:
            This attribute is deprecated.
            Please use `airflow.models.taskinstance.TaskInstance.get_previous_ti` method.
            
/home/airflow/.local/lib/python3.12/site-packages/airflow/www/views.py:1746 RemovedInAirflow3Warning:
            This attribute is deprecated.
            Please use `airflow.models.taskinstance.TaskInstance.get_previous_start_date` method.
            
/home/airflow/.local/lib/python3.12/site-packages/airflow/www/views.py:1746 RemovedInAirflow3Warning:
            This attribute is deprecated.
            Please use `airflow.models.taskinstance.TaskInstance.get_previous_ti` method.
            
[2024-04-22T07:48:04.801+0000] {views.py:1368} INFO - Retrieving rendered templates.
[2024-04-22T07:48:04.875+0000] {task_instance_session.py:38} WARNING - No task session set for this task. Continuing but this likely causes a resource leak.
[2024-04-22T07:48:04.875+0000] {task_instance_session.py:39} WARNING - Please report this and stacktrace below to https://github.com/apache/airflow/issues
[2024-04-22T07:48:04.880+0000] {task_instance_session.py:41} WARNING - File: "<frozen runpy>", 198 , in _run_module_as_main
[2024-04-22T07:48:04.881+0000] {task_instance_session.py:41} WARNING - File: "<frozen runpy>", 88 , in _run_code
[2024-04-22T07:48:04.881+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/gunicorn/__main__.py", 7 , in <module>
[2024-04-22T07:48:04.881+0000] {task_instance_session.py:43} WARNING -   run()
[2024-04-22T07:48:04.881+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/gunicorn/app/wsgiapp.py", 67 , in run
[2024-04-22T07:48:04.882+0000] {task_instance_session.py:43} WARNING -   WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
[2024-04-22T07:48:04.882+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/gunicorn/app/base.py", 236 , in run
[2024-04-22T07:48:04.882+0000] {task_instance_session.py:43} WARNING -   super().run()
[2024-04-22T07:48:04.882+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/gunicorn/app/base.py", 72 , in run
[2024-04-22T07:48:04.883+0000] {task_instance_session.py:43} WARNING -   Arbiter(self).run()
[2024-04-22T07:48:04.883+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/gunicorn/arbiter.py", 202 , in run
[2024-04-22T07:48:04.883+0000] {task_instance_session.py:43} WARNING -   self.manage_workers()
[2024-04-22T07:48:04.883+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/gunicorn/arbiter.py", 571 , in manage_workers
[2024-04-22T07:48:04.883+0000] {task_instance_session.py:43} WARNING -   self.spawn_workers()
[2024-04-22T07:48:04.884+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/gunicorn/arbiter.py", 642 , in spawn_workers
[2024-04-22T07:48:04.884+0000] {task_instance_session.py:43} WARNING -   self.spawn_worker()
[2024-04-22T07:48:04.884+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/gunicorn/arbiter.py", 609 , in spawn_worker
[2024-04-22T07:48:04.884+0000] {task_instance_session.py:43} WARNING -   worker.init_process()
[2024-04-22T07:48:04.885+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/gunicorn/workers/base.py", 142 , in init_process
[2024-04-22T07:48:04.885+0000] {task_instance_session.py:43} WARNING -   self.run()
[2024-04-22T07:48:04.885+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/gunicorn/workers/sync.py", 126 , in run
[2024-04-22T07:48:04.885+0000] {task_instance_session.py:43} WARNING -   self.run_for_one(timeout)
[2024-04-22T07:48:04.885+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/gunicorn/workers/sync.py", 70 , in run_for_one
[2024-04-22T07:48:04.886+0000] {task_instance_session.py:43} WARNING -   self.accept(listener)
[2024-04-22T07:48:04.886+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/gunicorn/workers/sync.py", 32 , in accept
[2024-04-22T07:48:04.886+0000] {task_instance_session.py:43} WARNING -   self.handle(listener, client, addr)
[2024-04-22T07:48:04.886+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/gunicorn/workers/sync.py", 135 , in handle
[2024-04-22T07:48:04.886+0000] {task_instance_session.py:43} WARNING -   self.handle_request(listener, req, client, addr)
[2024-04-22T07:48:04.886+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/gunicorn/workers/sync.py", 178 , in handle_request
[2024-04-22T07:48:04.887+0000] {task_instance_session.py:43} WARNING -   respiter = self.wsgi(environ, resp.start_response)
[2024-04-22T07:48:04.887+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", 2552 , in __call__
[2024-04-22T07:48:04.887+0000] {task_instance_session.py:43} WARNING -   return self.wsgi_app(environ, start_response)
[2024-04-22T07:48:04.887+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/werkzeug/middleware/proxy_fix.py", 187 , in __call__
[2024-04-22T07:48:04.887+0000] {task_instance_session.py:43} WARNING -   return self.app(environ, start_response)
[2024-04-22T07:48:04.888+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", 2529 , in wsgi_app
[2024-04-22T07:48:04.888+0000] {task_instance_session.py:43} WARNING -   response = self.full_dispatch_request()
[2024-04-22T07:48:04.888+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", 1823 , in full_dispatch_request
[2024-04-22T07:48:04.888+0000] {task_instance_session.py:43} WARNING -   rv = self.dispatch_request()
[2024-04-22T07:48:04.888+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", 1799 , in dispatch_request
[2024-04-22T07:48:04.889+0000] {task_instance_session.py:43} WARNING -   return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
[2024-04-22T07:48:04.889+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/airflow/www/auth.py", 252 , in decorated
[2024-04-22T07:48:04.889+0000] {task_instance_session.py:43} WARNING -   return _has_access(
[2024-04-22T07:48:04.889+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/airflow/www/auth.py", 163 , in _has_access
[2024-04-22T07:48:04.889+0000] {task_instance_session.py:43} WARNING -   return func(*args, **kwargs)
[2024-04-22T07:48:04.890+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/session.py", 79 , in wrapper
[2024-04-22T07:48:04.890+0000] {task_instance_session.py:43} WARNING -   return func(*args, session=session, **kwargs)
[2024-04-22T07:48:04.890+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/airflow/www/views.py", 1410 , in rendered_templates
[2024-04-22T07:48:04.890+0000] {task_instance_session.py:43} WARNING -   ti.get_rendered_template_fields(session=session)
[2024-04-22T07:48:04.890+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/session.py", 76 , in wrapper
[2024-04-22T07:48:04.890+0000] {task_instance_session.py:43} WARNING -   return func(*args, **kwargs)
[2024-04-22T07:48:04.891+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", 3041 , in get_rendered_template_fields
[2024-04-22T07:48:04.891+0000] {task_instance_session.py:43} WARNING -   self.render_templates()
[2024-04-22T07:48:04.891+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", 3079 , in render_templates
[2024-04-22T07:48:04.891+0000] {task_instance_session.py:43} WARNING -   original_task.render_template_fields(context, jinja_env)
[2024-04-22T07:48:04.891+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/mappedoperator.py", 827 , in render_template_fields
[2024-04-22T07:48:04.892+0000] {task_instance_session.py:43} WARNING -   session = get_current_task_instance_session()
[2024-04-22T07:48:04.892+0000] {task_instance_session.py:41} WARNING - File: "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/task_instance_session.py", 40 , in get_current_task_instance_session
[2024-04-22T07:48:04.892+0000] {task_instance_session.py:43} WARNING -   for filename, line_number, name, line in traceback.extract_stack():

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct
@raphaelauv raphaelauv added area:core kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet labels Apr 22, 2024
@RNHTTR RNHTTR added pending-response provider:cncf-kubernetes Kubernetes provider related issues labels Apr 22, 2024
@RNHTTR
Copy link
Collaborator

RNHTTR commented Apr 22, 2024

Can you provide reproduction steps? I'm running a mapped KPO on Airflow 2.9.0 and the rendered template view is fine

image

@raphaelauv
Copy link
Contributor Author

I do not know how to reproduce , only context I have the task was run in 2.8.0-pyt11 and stack is currently in 2.9.0-py12

@github-actions GitHub Actions
Copy link

github-actions bot commented May 8, 2024

This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.

@github-actions github-actions bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label May 8, 2024
@mro-dmoura
Copy link

I've been experiencing this error in mapped tasks. Many mapped tasks are failing with "[2024-05-08, 03:07:51 CDT] {scheduler_job_runner.py:769} ERROR - Executor reports task instance <TaskInstance: scheduled__2024-05-07T08:00:00+00:00 [queued]> finished (failed) although the task says it's queued. (Info: None) Was the task killed externally?". When investigating, the rendered template is showing the error: "Error rendering template: '_GeneratorContextManager' object has no attribute 'query'".
This started happening after upgrading from 2.8.1 to 2.9.1.
Any ideas?

@RNHTTR
Copy link
Collaborator

RNHTTR commented May 8, 2024 •
edited

@mro-dmoura A couple questions:

  1. Is this happening consistently or intermittently?
  2. Is this a mapped KubernetesPodOperator?

@RNHTTR RNHTTR added Can't Reproduce The problem cannot be reproduced area:dynamic-task-mapping AIP-42 and removed provider:cncf-kubernetes Kubernetes provider related issues labels May 8, 2024
@raphaelauv
Copy link
Contributor Author

It's only sometimes.

the same task ( mapped ) is not always problematic for all dag_run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area:core area:dynamic-task-mapping AIP-42 Can't Reproduce The problem cannot be reproduced kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet pending-response stale Stale PRs per the .github/workflows/stale.yml policy file
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@raphaelauv @RNHTTR @mro-dmoura