Log saturation prevents Synapse from working

[dvaceov]

Description

I'm using the Element service attached to Synapse and I've noticed that when the synapse logs are saturated, the Element service stops working.

Steps to reproduce

• Connection to Element not working
• Synapse service log dump
• Functional Element connection

Homeserver

another homeserver

Synapse Version

1.89

Installation Method

pip (from PyPI)

Database

single PostgreSQL server

Workers

Single process

Platform

Synapse runs in a Python virtual environment, under python 3.9, within a Linux Centos7 virtual machine.

Configuration

No response

Relevant log output

no relevant log output

Anything else that would be useful to know?

No response

[DMRobertson]

I've noticed that when the synapse logs are saturated

Can you tell us more?

• What does saturated mean here---no more disk space to write logs?
• What is your Synapse logging config?
• How are logs being handled outside of Synapse? E.g. are they being written to the systemd journal?

[dvaceov]

Hello,
The saturation I mentioned came from the fact that the log partition was full.

Here is the log configuration file for Synapse (the synapse log file is located in the /var/log partition, which has overflowed.)

`# Log configuration for Synapse.

This is a YAML file containing a standard Python logging configuration

dictionary. See [1] for details on the valid settings.

Synapse also supports structured logging for machine readable logs which can

be ingested by ELK stacks. See [2] for details.

[1]: https://docs.python.org/3/library/logging.config.html#configuration-dictionary-schema

[2]: https://matrix-org.github.io/synapse/latest/structured_logging.html

version: 1

formatters:
precise:
format: '%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(request)s - %(message)s'

handlers:
file:
class: logging.handlers.TimedRotatingFileHandler
formatter: precise
filename: /var/log/synapse/homeserver.log
when: midnight
backupCount: 3 # Does not include the current log file.
encoding: utf8

# Default to buffering writes to log file for efficiency.
# WARNING/ERROR logs will still be flushed immediately, but there will be a
# delay (of up to `period` seconds, or until the buffer is full with
# `capacity` messages) before INFO/DEBUG logs get written.
buffer:
    class: synapse.logging.handlers.PeriodicallyFlushingMemoryHandler
    target: file

    # The capacity is the maximum number of log lines that are buffered
    # before being written to disk. Increasing this will lead to better
    # performance, at the expensive of it taking longer for log lines to
    # be written to disk.
    # This parameter is required.
    capacity: 10

    # Logs with a level at or above the flush level will cause the buffer to
    # be flushed immediately.
    # Default value: 40 (ERROR)
    # Other values: 50 (CRITICAL), 30 (WARNING), 20 (INFO), 10 (DEBUG)
    flushLevel: 30  # Flush immediately for WARNING logs and higher

    # The period of time, in seconds, between forced flushes.
    # Messages will not be delayed for longer than this time.
    # Default value: 5 seconds
    period: 5

# A handler that writes logs to stderr. Unused by default, but can be used
# instead of "buffer" and "file" in the logger handlers.
console:
    class: logging.StreamHandler
    formatter: precise

loggers:
synapse.storage.SQL:
# beware: increasing this to DEBUG will make synapse log sensitive
# information such as access tokens.
level: INFO

root:
level: INFO

# Write logs to the `buffer` handler, which will buffer them together in memory,
# then write them to a file.
#
# Replace "buffer" with "console" to log to stderr instead.
#
handlers: [buffer]

disable_existing_loggers: false`

Logs are written only to the homeserver.log file and are not replicated in systemd.