Skip to content

Commit

Permalink
Add Docs for default_pool slots (#15997)
Browse files Browse the repository at this point in the history
Until now this was undocumented, however it was used in:
https://github.com/apache/airflow/blob/aa4713e43f92d3e4c68c3ad00e2d44caaf29aafe/airflow/utils/db.py#L75

This was done in 1.10.4 in 2c99ec6
commit. This commit also updates the name from `non_pooled_task_slot_count` to `default_pool_task_slot_count` as
this is what it actually does.
  • Loading branch information
kaxil committed May 22, 2021
1 parent 476d0f6 commit 8f9538e
Show file tree
Hide file tree
Showing 5 changed files with 26 additions and 1 deletion.
10 changes: 10 additions & 0 deletions UPDATING.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,16 @@ https://developers.google.com/style/inclusive-documentation
-->

### Default Task Pools Slots can be set using ``[core] default_pool_task_slot_count``

By default tasks are running in `default_pool`. `default_pool` is initialized with `128` slots and user can change the
number of slots through UI/CLI/API for an existing deployment.

For new deployments, you can use `default_pool_task_slot_count` setting in `[core]` section. This setting would
not have any effect in an existing deployment where the ``default_pool`` already exists.

Previously this was controlled by `non_pooled_task_slot_count` in `[core]` section, which was not documented.

## Airflow 2.1.0

### New "deprecated_api" extra
Expand Down
9 changes: 9 additions & 0 deletions airflow/config_templates/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -435,6 +435,15 @@
type: string
example: ~
default: ""
- name: default_pool_task_slot_count
description: |
Task Slot counts for ``default_pool``. This setting would not have any effect in an existing
deployment where the ``default_pool`` is already created. For existing deployments, users can
change the number of slots using Webserver, API or the CLI
version_added: 2.2.0
type: string
example: ~
default: "128"

- name: logging
description: ~
Expand Down
5 changes: 5 additions & 0 deletions airflow/config_templates/default_airflow.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -250,6 +250,11 @@ hide_sensitive_var_conn_fields = True
# extra JSON.
sensitive_var_conn_names =

# Task Slot counts for ``default_pool``. This setting would not have any effect in an existing
# deployment where the ``default_pool`` is already created. For existing deployments, users can
# change the number of slots using Webserver, API or the CLI
default_pool_task_slot_count = 128

[logging]
# The folder where airflow should store its log files
# This path must be absolute
Expand Down
1 change: 1 addition & 0 deletions airflow/configuration.py
Original file line number Diff line number Diff line change
Expand Up @@ -162,6 +162,7 @@ class AirflowConfigParser(ConfigParser): # pylint: disable=too-many-ancestors
('operators', 'default_queue'): ('celery', 'default_queue', '2.1.0'),
('core', 'hide_sensitive_var_conn_fields'): ('admin', 'hide_sensitive_variable_fields', '2.1.0'),
('core', 'sensitive_var_conn_names'): ('admin', 'sensitive_variable_fields', '2.1.0'),
('core', 'default_pool_task_slot_count'): ('core', 'non_pooled_task_slot_count', '1.10.4'),
}

# A mapping of old default values that we want to change and warn the user
Expand Down
2 changes: 1 addition & 1 deletion airflow/utils/db.py
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ def add_default_pool_if_not_exists(session=None):
if not Pool.get_pool(Pool.DEFAULT_POOL_NAME, session=session):
default_pool = Pool(
pool=Pool.DEFAULT_POOL_NAME,
slots=conf.getint(section='core', key='non_pooled_task_slot_count', fallback=128),
slots=conf.getint(section='core', key='default_pool_task_slot_count'),
description="Default pool",
)
session.add(default_pool)
Expand Down

0 comments on commit 8f9538e

Please sign in to comment.