Scheduler Goes In crashloop

[akshaysyaduvanshi]

Docker Image -> apache/airflow:2.8.3-python3.10

Scheduler goes in crashloop with below error

[2024-05-04T12:29:41.271+0000] {scheduler_job_runner.py:258} INFO - Exiting gracefully upon receiving signal 15
[2024-05-04T12:29:42.272+0000] {process_utils.py:131} INFO - Sending Signals.SIGTERM to group 40. PIDs of all processes in the group: [40]
[2024-05-04T12:29:42.272+0000] {process_utils.py:86} INFO - Sending the signal Signals.SIGTERM to group 40
[2024-05-04T12:29:42.273+0000] {manager.py:451} INFO - Exiting gracefully upon receiving signal 15
[2024-05-04T12:29:42.404+0000] {process_utils.py:79} INFO - Process psutil.Process(pid=40, status='terminated', exitcode=0, started='12:24:52') (40) terminated with exit code 0
[2024-05-04T12:29:42.405+0000] {kubernetes_executor.py:762} INFO - Shutting down Kubernetes executor
[2024-05-04T12:29:42.405+0000] {scheduler_job_runner.py:878} ERROR - Exception when executing Executor.end
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/jobs/scheduler_job_runner.py", line 855, in _execute
self._run_scheduler_loop()
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/jobs/scheduler_job_runner.py", line 1010, in _run_scheduler_loop
time.sleep(min(self._scheduler_idle_sleep_time, next_event or 0))
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/jobs/scheduler_job_runner.py", line 261, in _exit_gracefully
sys.exit(os.EX_OK)
SystemExit: 0

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/jobs/scheduler_job_runner.py", line 876, in _execute
self.job.executor.end()
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py", line 765, in end
self._flush_task_queue()
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py", line 721, in _flush_task_queue
self.log.debug("Executor shutting down, task_queue approximate size=%d", self.task_queue.qsize())
File "", line 2, in qsize
File "/usr/local/lib/python3.10/multiprocessing/managers.py", line 817, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 206, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 411, in _send_bytes
self._send(header + buf)
File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe

And Scheduler Liveness probe fails with below error

Warning Unhealthy 3s (x14 over 6m33s) kubelet Liveness probe failed: Traceback (most recent call last):
File "", line 5, in
ModuleNotFoundError: No module named 'airflow.jobs.scheduler_job'

Any idea what is causing this?

[Taragolis]

[2024-05-04T12:29:41.271+0000] {scheduler_job_runner.py:258} INFO - Exiting gracefully upon receiving signal 15

SIGTERM signal is sent to the pod

Which more possible caused by ModuleNotFoundError: No module named 'airflow.jobs.scheduler_job' this module was replaced some time ago.

This module used only in Airflow < 2.1.0 in Helm Chart for Apache Airflow

airflow/chart/templates/_helpers.yaml

Lines 647 to 677 in e95f83e

	{{- define "scheduler_liveness_check_command" }}
	{{- if semverCompare ">=2.5.0" .Values.airflowVersion }}
	- sh
	- -c
	- |
	CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint \
	airflow jobs check --job-type SchedulerJob --local
	{{- else if semverCompare ">=2.1.0" .Values.airflowVersion }}
	- sh
	- -c
	- |
	CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint \
	airflow jobs check --job-type SchedulerJob --hostname $(hostname)
	{{- else }}
	- sh
	- -c
	- |
	CONNECTION_CHECK_MAX_COUNT=0 exec /entrypoint python -Wignore -c "
	import os
	os.environ['AIRFLOW__CORE__LOGGING_LEVEL'] = 'ERROR'
	os.environ['AIRFLOW__LOGGING__LOGGING_LEVEL'] = 'ERROR'
	from airflow.jobs.scheduler_job import SchedulerJob
	from airflow.utils.db import create_session
	from airflow.utils.net import get_hostname
	import sys
	with create_session() as session:
	job = session.query(SchedulerJob).filter_by(hostname=get_hostname()).order_by(
	SchedulerJob.latest_heartbeat.desc()).limit(1).first()
	sys.exit(0 if job.is_alive() else 1)"
	{{- end }}
	{{- end }}

[akshaysyaduvanshi]

@Taragolis Yes i have updated my liveness probe check to below

livenessProbe:
initialDelaySeconds: {{ .Values.scheduler.livenessProbe.initialDelaySeconds }}
timeoutSeconds: {{ .Values.scheduler.livenessProbe.timeoutSeconds }}
failureThreshold: {{ .Values.scheduler.livenessProbe.failureThreshold }}
periodSeconds: {{ .Values.scheduler.livenessProbe.periodSeconds }}
exec:
command:
- sh
- -c
- |
CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint
airflow jobs check --job-type SchedulerJob --local

[akshaysyaduvanshi]

@Taragolis I had updated the health check code, but did not upgrade the helm chart version, after upgrading helm chart version , it is not crashing anymore.