AirflowDagRunner fails with RuntimeParameter for scheduled runs

[peteboothroyd]

This is not a contribution.

TFX Version: 1.2
Airflow version 1.14

Describe the current behavior

When using a RuntimeParameter with the AirflowDagRunner the dag will fail for scheduled runs where the dag_run.conf is missing.

Specifically this error is thrown:

[2021-10-20 13:15:40,967] {taskinstance.py:1150} ERROR - 'None' has no attribute 'get'
Traceback (most recent call last):
  File "/usr/local/airflow/venv/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 965, in _run_raw_task
    self.render_templates(context=context)
  File "/usr/local/airflow/venv/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1424, in render_templates
    self.task.render_template_fields(context)
  File "/usr/local/airflow/venv/lib/python3.7/site-packages/airflow/models/baseoperator.py", line 719, in render_template_fields
    self._do_render_template_fields(self, self.template_fields, context, jinja_env, set())
  File "/usr/local/airflow/venv/lib/python3.7/site-packages/airflow/models/baseoperator.py", line 726, in _do_render_template_fields
    rendered_content = self.render_template(content, context, jinja_env, seen_oids)
  File "/usr/local/airflow/venv/lib/python3.7/site-packages/airflow/models/baseoperator.py", line 770, in render_template
    return {key: self.render_template(value, context, jinja_env) for key, value in content.items()}
  File "/usr/local/airflow/venv/lib/python3.7/site-packages/airflow/models/baseoperator.py", line 770, in <dictcomp>
    return {key: self.render_template(value, context, jinja_env) for key, value in content.items()}
  File "/usr/local/airflow/venv/lib/python3.7/site-packages/airflow/models/baseoperator.py", line 770, in render_template
    return {key: self.render_template(value, context, jinja_env) for key, value in content.items()}
  File "/usr/local/airflow/venv/lib/python3.7/site-packages/airflow/models/baseoperator.py", line 770, in <dictcomp>
    return {key: self.render_template(value, context, jinja_env) for key, value in content.items()}
  File "/usr/local/airflow/venv/lib/python3.7/site-packages/airflow/models/baseoperator.py", line 755, in render_template
    return jinja_env.from_string(content).render(**context)
  File "/usr/local/airflow/venv/lib/python3.7/site-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "/usr/local/airflow/venv/lib/python3.7/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/usr/local/airflow/venv/lib/python3.7/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 1, in top-level template code
jinja2.exceptions.UndefinedError: 'None' has no attribute 'get'

As far as I can tell this occurs due to this line which assumes that dag_run.conf is defined, whereas it appears to be None in some situations (see apache/airflow#5388). The template probably needs to be adapted to dag_run.conf.get("{prop_name}", {default}) if dag_run.conf else {default}

Note that this has been running successfully where we manually trigger the dags through the airflow UI.

@casassg how do you normally get around this?

[casassg]

I didn't realize about this. I also expected this to return an empty dictionary. I have to admit we have not been using this a lot.

That said, I'm sure a PR can be made and tested :D Would be happy to review and tag folks from Google to help review!

[peteboothroyd]

Thanks for the info, I would love to contribute a patch but unfortunately it's very difficult on my end to get approval given the CLA requirement.

[singhniraj08]

@peteboothroyd,

Are you still looking for a resolution? We are planning on prioritising the issues based on the community interests. Please let us know if this issue still persists with the latest TFX 1.13 release so that we can work on fixing it. Thank you for your contributions.

[peteboothroyd]

This is not a priority for us.