Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

upgrade form 2023.10.4 to 2024.4. Containers server and worker restart #9701

Open
Julien-Quidam opened this issue May 13, 2024 · 5 comments
Open

upgrade form 2023.10.4 to 2024.4. Containers server and worker restart #9701

Julien-Quidam opened this issue May 13, 2024 · 5 comments
Labels
bug Something isn't working

Comments

@Julien-Quidam
Copy link

Describe the bug
I upgrade images the server and the worker
ghcr.io/goauthentik/server:2023.10.4 to ghcr.io/goauthentik/server:latest

Containers server and worker restart again and again

To Reproduce
1 - stop all containers
authentik-worker-1
authentik-server-1
authentik-redis-1
authentik-postgresql-1

2 - change image on authentik-worker-1 and authentik-server-1

3 - restart all containers

Ok :
authentik-redis-1
authentik-postgresql-1

Nok:
authentik-worker-1
authentik-server-1

Containers server and the worker restart again and again. No error in log.

Expected behavior
Containers server and worker start

Logs

{"event": "Loaded config", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1715607634.2827733, "file": "/authentik/lib/default.yml"}
{"event": "Loaded environment variables", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1715607634.283426, "count": 9}
{"event": "Starting authentik bootstrap", "level": "info", "logger": "authentik.lib.config", "timestamp": 1715607634.7895787}
{"event": "PostgreSQL connection successful", "level": "info", "logger": "authentik.lib.config", "timestamp": 1715607634.813482}
{"event": "Redis Connection successful", "level": "info", "logger": "authentik.lib.config", "timestamp": 1715607634.8301268}
{"event": "Finished authentik bootstrap", "level": "info", "logger": "authentik.lib.config", "timestamp": 1715607634.8303654}
{"event": "Booting authentik", "level": "info", "logger": "authentik.lib.config", "timestamp": 1715607638.2763298, "version": "2024.4.1"}
{"event": "Enabled authentik enterprise", "level": "info", "logger": "authentik.lib.config", "timestamp": 1715607638.3674917}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1715607638.3692343, "path": "authentik.enterprise.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1715607638.3716676, "path": "authentik.admin.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1715607638.3725333, "path": "authentik.crypto.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1715607638.37624, "path": "authentik.providers.scim.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1715607638.378682, "path": "authentik.stages.authenticator_totp.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1715607638.3818927, "path": "authentik.sources.ldap.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1715607638.382671, "path": "authentik.events.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1715607638.3830607, "path": "authentik.enterprise.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1715607638.3843555, "path": "authentik.blueprints.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1715607638.3850732, "path": "authentik.sources.oauth.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1715607638.3878953, "path": "authentik.outposts.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1715607638.390013, "path": "authentik.sources.plex.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1715607638.3927524, "path": "authentik.policies.reputation.settings"}
{"domain_url": null, "event": "Loaded MMDB database", "file": "/geoip/GeoLite2-ASN.mmdb", "last_write": 1714147348.0, "level": "info", "logger": "authentik.events.context_processors.mmdb", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:39.885142"}
{"domain_url": null, "event": "Loaded MMDB database", "file": "/geoip/GeoLite2-City.mmdb", "last_write": 1714147347.0, "level": "info", "logger": "authentik.events.context_processors.mmdb", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:39.888174"}
{"app_name": "authentik.tenants", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.tenants.checks", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:42.346837"}
{"app_name": "authentik.tenants", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.tenants.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:42.347727"}
{"app_name": "authentik.admin", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.admin.tasks", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:42.449257"}
{"app_name": "authentik.admin", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.admin.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:42.450194"}
{"app_name": "authentik.crypto", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.crypto.tasks", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:42.455979"}
{"app_name": "authentik.flows", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.flows.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:42.525525"}
{"app_name": "authentik.outposts", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.outposts.tasks", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:42.579616"}
{"app_name": "authentik.outposts", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.outposts.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:42.581382"}
{"app_name": "authentik.policies.reputation", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.policies.reputation.tasks", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:42.582738"}
{"app_name": "authentik.policies.reputation", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.policies.reputation.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:42.584056"}
{"app_name": "authentik.policies", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.policies.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:42.602606"}
{"app_name": "authentik.providers.proxy", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.providers.proxy.tasks", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:42.603854"}
{"app_name": "authentik.providers.proxy", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.providers.proxy.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:42.604647"}
{"app_name": "authentik.providers.scim", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.providers.scim.tasks", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.044267"}
{"app_name": "authentik.providers.scim", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.providers.scim.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.045852"}
{"app_name": "authentik.rbac", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.rbac.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.047206"}
{"app_name": "authentik.sources.ldap", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.sources.ldap.tasks", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.060057"}
{"app_name": "authentik.sources.ldap", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.sources.ldap.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.063793"}
{"app_name": "authentik.sources.oauth", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.sources.oauth.tasks", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.078878"}
{"app_name": "authentik.sources.saml", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.sources.saml.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.079887"}
{"app_name": "authentik.sources.scim", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.sources.scim.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.080847"}
{"app_name": "authentik.stages.authenticator_duo", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.stages.authenticator_duo.tasks", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.081868"}
{"app_name": "authentik.stages.authenticator_static", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.stages.authenticator_static.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.082638"}
{"app_name": "authentik.stages.authenticator_webauthn", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.stages.authenticator_webauthn.tasks", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.150745"}
{"app_name": "authentik.stages.email", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.stages.email.tasks", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.154305"}
{"app_name": "authentik.core", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.core.tasks", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.157033"}
{"app_name": "authentik.core", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.core.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.157678"}
{"app_name": "authentik.enterprise", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.enterprise.tasks", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.159569"}
{"app_name": "authentik.enterprise", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.enterprise.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.160965"}
{"app_name": "authentik.enterprise.providers.rac", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.enterprise.providers.rac.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.183413"}
{"app_name": "authentik.events", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.events.tasks", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.184416"}
{"app_name": "authentik.events", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.events.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.184704"}

Type 'manage.py help <subcommand>' for help on a specific subcommand.

Available subcommands:

[auth]
    changepassword
    createsuperuser

[authenticator_webauthn]
    update_webauthn_mds

[blueprints]
    apply_blueprint
    export_blueprint
    make_blueprint_schema

[channels]
    runworker

[contenttypes]
    remove_stale_contenttypes

[core]
    bootstrap_tasks
    build_source_docs
    dev_server
    repair_permissions
    shell
    worker

[crypto]
    import_certificate

[daphne]
    runserver

[django]
    check
    compilemessages
    createcachetable
    dbshell
    diffsettings
    dumpdata
    flush
    inspectdb
    loaddata
    makemessages
    makemigrations
    optimizemigration
    sendtestemail
    showmigrations
    sqlflush
    sqlmigrate
    sqlsequencereset
    squashmigrations
    startapp
    startproject
    test
    testserver

[django_tenants]
    all_tenants_command
    clone_tenant
    collectstatic_schemas
    create_missing_schemas
    create_tenant
    create_tenant_superuser
    delete_tenant
    migrate
    migrate_schemas
    rename_schema
    tenant_command

[drf_spectacular]
    spectacular

[email]
    test_email

[flows]
    benchmark

[guardian]
    clean_orphan_obj_perms

[ldap]
    ldap_check_connection
    ldap_sync

[recovery]
    create_admin_group
    create_recovery_key

[rest_framework]
    generateschema

[scim]
    scim_sync

[sessions]
    clearsessions

[staticfiles]
    collectstatic
    findstatic
 *  Terminal will be reused by tasks, press any key to close it.

Version and Deployment (please complete the following information):

  • authentik version: 2024.4
  • Deployment: docker-compose
@Julien-Quidam Julien-Quidam added the bug Something isn't working label May 13, 2024
@depuits
Copy link
Contributor

depuits commented May 21, 2024

I'm having a similar issue when trying to upgrade from 2024.4.1 to 2024.4.2.

I do see following error in the worker log

DBG event=Loaded app settings logger=authentik.lib.config timestamp=1715538755.9956036 path=authentik.sources.plex.settings
DBG event=Loaded app settings logger=authentik.lib.config timestamp=1715538755.9970844 path=authentik.providers.scim.settings
DBG event=Loaded app settings logger=authentik.lib.config timestamp=1715538755.999732 path=authentik.crypto.settings
/ak-root/venv/lib/python3.12/site-packages/opencontainers/distribution/reggie/defaults.py:17: SyntaxWarning: invalid escape sequence '\('"http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+"
_runit-log:x:998:authentik

Reverting back to version 2024.4.1 does fix the problem.

@FaykoB
Copy link

FaykoB commented May 21, 2024

I'm having a similar issue when trying to upgrade from 2024.4.1 to 2024.4.2.

I do see following error in the worker log

DBG event=Loaded app settings logger=authentik.lib.config timestamp=1715538755.9956036 path=authentik.sources.plex.settings
DBG event=Loaded app settings logger=authentik.lib.config timestamp=1715538755.9970844 path=authentik.providers.scim.settings
DBG event=Loaded app settings logger=authentik.lib.config timestamp=1715538755.999732 path=authentik.crypto.settings
/ak-root/venv/lib/python3.12/site-packages/opencontainers/distribution/reggie/defaults.py:17: SyntaxWarning: invalid escape sequence '\('"http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+"
_runit-log:x:998:authentik

Reverting back to version 2024.4.1 does fix the problem.

I removed redis,postgres, and both authentik containers then reinstalled the first 2 with the latest tags and the authentik worker with 2024.4.1 and I'm still getting an escape sequence failure. Should i remove everything and go even earlier?

@bnounours
Copy link

Hello,
I have the same with the migration from 2024.4.1 to 2024.4.2. In fact the escape error is not the root cause there is the same warning in 2024.4.1.
But I connected inside the container with a docker exec -it <name of server container> bash
I tried to use `manage.py script and got a core dump

root@656567b6e290:/# export AUTHENTIK_LOG_LEVEL=trace
root@656567b6e290:/# ./manage.py 
{"event": "Loaded config", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1717231859.026418, "file": "/authentik/lib/default.yml"}
{"event": "Loaded environment variables", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1717231859.027352, "count": 15}
{"event": "Starting authentik bootstrap", "level": "info", "logger": "authentik.lib.config", "timestamp": 1717231860.2170274}
{"event": "PostgreSQL connection successful", "level": "info", "logger": "authentik.lib.config", "timestamp": 1717231860.234503}
{"event": "Redis Connection successful", "level": "info", "logger": "authentik.lib.config", "timestamp": 1717231860.237499}
{"event": "Finished authentik bootstrap", "level": "info", "logger": "authentik.lib.config", "timestamp": 1717231860.2379699}
{"event": "Booting authentik", "level": "info", "logger": "authentik.lib.config", "timestamp": 1717231866.3635058, "version": "2024.4.2"}
{"event": "Enabled authentik enterprise", "level": "info", "logger": "authentik.lib.config", "timestamp": 1717231866.5071452}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1717231866.5095181, "path": "authentik.enterprise.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1717231866.511984, "path": "authentik.outposts.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1717231866.5135534, "path": "authentik.sources.plex.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1717231866.5165665, "path": "authentik.admin.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1717231866.5177824, "path": "authentik.policies.reputation.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1717231866.5207522, "path": "authentik.sources.ldap.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1717231866.5235906, "path": "authentik.sources.oauth.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1717231866.5267992, "path": "authentik.events.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1717231866.5306711, "path": "authentik.crypto.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1717231866.5318296, "path": "authentik.blueprints.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1717231866.5359561, "path": "authentik.stages.authenticator_totp.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1717231866.5364292, "path": "authentik.enterprise.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1717231866.5375683, "path": "authentik.providers.scim.settings"}
/ak-root/venv/lib/python3.12/site-packages/opencontainers/distribution/reggie/defaults.py:17: SyntaxWarning: invalid escape sequence '\('
  "http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+"
Illegal instruction (core dumped)

I think this is the reason why the server restart over an over again

@bnounours
Copy link

bnounours commented Jun 1, 2024 •
edited

Continuing the investigation, it is when loading lib avatars

Python 3.12.3 (main, Apr 24 2024, 11:28:46) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import authentik.lib.avatars
Illegal instruction (core dumped)

It is when loading the lxml lib

> /authentik/lib/avatars.py(12)<module>()
-> from lxml import etree  # nosec
(Pdb) 
Illegal instruction (core dumped)

I tried directly on the container

root@af1eb6b4d850:/# source ak-root/venv/bin/activate
(venv) root@af1eb6b4d850:/# python
Python 3.12.3 (main, Apr 24 2024, 11:28:46) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from lxml import etree
Illegal instruction (core dumped)

@bnounours
Copy link

bnounours commented Jun 1, 2024 •
edited

Upgrade lxml to 5.2.2 in the container unblock the server start. In the pyproject.toml there is no version limit for lxml in release 2024.4.2 in release 2024.4.1 there is a version fixed. I tried the same version as the one in the release 2024.4.1 but there is a compatibility error raising after.
The version lxml==5.2.1 seems to be the one problematic

To upgrade in the container:

sudo docker exec -it <authentik-server-container> bash
source /ak-root/venv/bin/activate
pip install lxml==5.2.2

You can test with

(venv) root@4437d18bb20e:/# python
Python 3.12.3 (main, Apr 24 2024, 11:28:46) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from lxml import etree
>>>

If you don't have core dump it is resolved. Then restart (not recreate !!) container and server works. The same needs to be done on worker (same image)

If the image restart too quickly here is one line command to do it

docker exec -it --user root <worker or server container> bash -c 'source /ak-root/venv/bin/activate; pip install lxml==5.2.2'
``

bnounours added a commit to bnounours/authentik that referenced this issue Jun 1, 2024
@bnounours
Fix Bug goauthentik#9701
b25e1b6
@bnounours bnounours mentioned this issue Jun 1, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@depuits @bnounours @Julien-Quidam @FaykoB