Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sync v2 1 2 stable v2 #16862

Merged
merged 7 commits into from Jul 9, 2021
Merged

Sync v2 1 2 stable v2 #16862

merged 7 commits into from Jul 9, 2021

Commits on Jul 9, 2021

  1. Fix CeleryKubernetesExecutor (apache#16700) 

    closes apache#16326

Currently when running celery tasks when running with ``CeleryKubernetesExecutor``,
we see the following error. This error occurs as the ``BaseJob`` (via ``LocalTaskJob``) tries to needlessly
instantiate a `KubernetesExecutor` which in turn tries to create a multiprocessing process/Manager
which fails.

```
[2021-06-29 00:23:45,301: ERROR/ForkPoolWorker-16] Failed to execute task daemonic processes are not allowed to have children.
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/executors/celery_executor.py", line 116, in _execute_in_fork
    args.func(args)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/cli/cli_parser.py", line 48, in command
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/cli.py", line 91, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/cli/commands/task_command.py", line 237, in task_run
    _run_task_by_selected_method(args, dag, ti)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/cli/commands/task_command.py", line 64, in _run_task_by_selected_method
    _run_task_by_local_task_job(args, ti)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/cli/commands/task_command.py", line 117, in _run_task_by_local_task_job
    pool=args.pool,
  File "<string>", line 4, in __init__
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/state.py", line 433, in _initialize_instance
    manager.dispatch.init_failure(self, args, kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    with_traceback=exc_tb,
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/state.py", line 430, in _initialize_instance
    return manager.original_init(*mixed[1:], **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/jobs/local_task_job.py", line 76, in __init__
    super().__init__(*args, **kwargs)
  File "<string>", line 6, in __init__
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/jobs/base_job.py", line 97, in __init__
    self.executor = executor or ExecutorLoader.get_default_executor()
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/executors/executor_loader.py", line 62, in get_default_executor
    cls._default_executor = cls.load_executor(executor_name)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/executors/executor_loader.py", line 79, in load_executor
    return cls.__load_celery_kubernetes_executor()
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/executors/executor_loader.py", line 116, in __load_celery_kubernetes_executor
    kubernetes_executor = import_string(cls.executors[KUBERNETES_EXECUTOR])()
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/executors/kubernetes_executor.py", line 421, in __init__
    self._manager = multiprocessing.Manager()
  File "/usr/local/lib/python3.6/multiprocessing/context.py", line 56, in Manager
    m.start()
  File "/usr/local/lib/python3.6/multiprocessing/managers.py", line 513, in start
    self._process.start()
  File "/usr/local/lib/python3.6/multiprocessing/process.py", line 103, in start
    'daemonic processes are not allowed to have children'
AssertionError: daemonic processes are not allowed to have children
```

We don't need to instantiate an executor when running ``LocalTaskJob`` as executor isn't used in it.

(cherry picked from commit 7857a9b)
    @kaxil @jhtimmins
    kaxil authored and jhtimmins committed Jul 9, 2021
    Configuration menu
    Copy the full SHA
    b350481 View commit details
    Browse the repository at this point in the history

  2. yarn audit (apache#16440) 

    Bump a lot of npm modules in packages.json to resolve all severe and moderate vulnerabilities found when using `yarn audit`

(cherry picked from commit f18e4ba)
    @bbovenzi @jhtimmins
    bbovenzi authored and jhtimmins committed Jul 9, 2021
    Configuration menu
    Copy the full SHA
    79399c0 View commit details
    Browse the repository at this point in the history

  3. Mask value if the key is token (apache#16474) 

    Some connections (including the databricks connection) use the key 'token' in the 'extra' field (this has always been the case). Including it here so that these sensitive tokens are also masked by default.

The prior implementation just masked all of the 'extra' json: "XXXXXXXX" if conn.extra_dejson else None https://github.com/apache/airflow/blob/88199eefccb4c805f8d6527bab5bf600b397c35e/airflow/hooks/base.py#L78

(cherry picked from commit d1d04fe)
    @robertsaxby @jhtimmins
    robertsaxby authored and jhtimmins committed Jul 9, 2021
    Configuration menu
    Copy the full SHA
    0a52997 View commit details
    Browse the repository at this point in the history

  4. Fix impersonation issue with LocalTaskJob (apache#16852) 

    Running a task with run_as_user fails because PIDs are not matched
correctly.

This change fixes it by matching the parent process ID (the `sudo`
process) of the task instance to the current process ID of the task_runner
process when we use impersonation

Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
(cherry picked from commit feea380)
    @ephraimbuddy @jhtimmins
    ephraimbuddy authored and jhtimmins committed Jul 9, 2021
    Configuration menu
    Copy the full SHA
    3d67a75 View commit details
    Browse the repository at this point in the history

  5. Update release date and changelog for 2.1.2.

    @jhtimmins
    jhtimmins committed Jul 9, 2021
    Configuration menu
    Copy the full SHA
    93aab40 View commit details
    Browse the repository at this point in the history

  6. Introduce compat shim airflow.compat.functools (apache#15969) 

    This module shims 'cached_property' and 'cache' so modules don't need to
all do their own ad-hoc try-except ImportError.

(cherry picked from commit 3db347e)
    @uranusjr @jhtimmins
    uranusjr authored and jhtimmins committed Jul 9, 2021
    Configuration menu
    Copy the full SHA
    bcbe65e View commit details
    Browse the repository at this point in the history

  7. Removes pylint from our toolchain (apache#16682) 

    We've agreed during the voting process that Pylint support
should be disabled: https://lists.apache.org/thread.html/r9e2cc385db8737ec0874ad09872081bd083593ee29e8303e58d21efb%40%3Cdev.airflow.apache.org%3E

This PR:

* removes all # pylint comments
* removes pylint pre-commits and related scripts/files
* removes CI jobs running pylint checks
* removes documentation about pylint
* removes unnecessary #noga (adds pre-commit for that)
* fixes some remaining pydocstyle errors after removing #noqa's

(cherry picked from commit 866a601)
    @potiuk @jhtimmins
    potiuk authored and jhtimmins committed Jul 9, 2021
    Configuration menu
    Copy the full SHA
    fe060b5 View commit details
    Browse the repository at this point in the history