Fix IntegrityError during webserver startup

[ephraimbuddy]

When starting up the webserver, we create all the gunicorn workers needed and each of these workers creates the webserver app instead of getting a copy of the app. This results in all the workers trying to write to the database at the same time causing a lot of IntegrityError in the database.

The fix was to preload the app before forking workers

closes: #23512

[potiuk]

No issues with plugins any more ?

[ephraimbuddy]

No issues with plugins any more ?

Yes. What is loaded at that point is web UI plugin. The ensure_plugins_loaded which calls the main airflow plugin is not loaded with the app. The problem before was whether it could be a problem with lazy_load_plugin config but it's not as only the web UI plugins are loaded during loading of the app:

airflow/airflow/www/extensions/init_views.py

Lines 123 to 147 in 126b7b8

	def init_plugins(app):
	"""Integrate Flask and FAB with plugins"""
	from airflow import plugins_manager
	plugins_manager.initialize_web_ui_plugins()
	appbuilder = app.appbuilder
	for view in plugins_manager.flask_appbuilder_views:
	name = view.get('name')
	if name:
	log.debug("Adding view %s with menu", name)
	appbuilder.add_view(view["view"], name, category=view["category"])
	else:
	# if 'name' key is missing, intent is to add view without menu
	log.debug("Adding view %s without menu", str(type(view["view"])))
	appbuilder.add_view_no_menu(view["view"])
	for menu_link in sorted(plugins_manager.flask_appbuilder_menu_links, key=lambda x: x["name"]):
	log.debug("Adding menu link %s to %s", menu_link["name"], menu_link["href"])
	appbuilder.add_link(**menu_link)
	for blue_print in plugins_manager.flask_blueprints:
	log.debug("Adding blueprint %s:%s", blue_print["name"], blue_print["blueprint"].import_name)
	app.register_blueprint(blue_print["blueprint"])

The lazy_load_plugin config is called during startup

airflow/airflow/__init__.py

Lines 86 to 89 in 126b7b8

	if not settings.LAZY_LOAD_PLUGINS:
	from airflow import plugins_manager
	plugins_manager.ensure_plugins_loaded()

which loads the main airflow plugin(not just UI)

[potiuk]

Cool!

[zachliu]

just tested, this works!
👍 👍 👍
👍 👍
👍

[atrbgithub]

@ephraimbuddy Thanks for this!

Will this also help where there are multiple webservers being started on different instances?

We originally mitigated this issue by slowing scaling up the workers on each instance as described here. That worked fine for when there was only one webserver instance, but not when there were multiple.

With this fix, when two webservers both start up and attempt the preload, is there a chance they will hit the same issue?