Systemd notify and watchdog support for sidekiqswarm

[krasnoukhov]

Changes that were made for #4488 don't work for sidekiqswarm. I'm sure lots of enterprise customers will benefit from having the same neat integration when running multiple processes.

[mperham]

Unfortunately there are some serious impl issues. The notify functionality doesn't map well to a parent process with N child processes.

• systemd allows a watchdog timer which will restart a process which does not ping the watchdog within N seconds. This watchdog makes little sense in a multi-process scenario; you want to monitor the individual children and restart any that lock up.
• what does it mean for the parent swarm process to be ready? when all children have been forked? N seconds after boot where no children have died?

I'll do some googling and see if anyone has thought about or worked through this issue.

[mperham]

I suspect because the process model is completely different I will need to rework the systemd integration to be completely different for the sidekiq and sidekiqswarm binaries. No easy solution that I can see.

[misdoro]

Sidekiq swarm may be replaced by systemd template services, letting systemd run many sidekiq instances on a single host, with
sidekiq@.service template
including something like

ExecStartPost=/bin/sh -c "/bin/echo $MAINPID > /var/run/sidekiq%I.pid"
ExecStopPost=/bin/sh -c "rm /var/run/sidekiq%I.pid"

if you need pid files.

Then you may start as many sidekiq instances as you want e.g. with a simple bash script:

/bin/bash
for i in $(seq 1 ${SIDEKIQ_WORKERS}); do sudo systemctl enable --now sidekiq@$i.service; done

Also a custom systemd target may be created:
add a section in
sidekiq@.service:

[Install]
WantedBy=sidekiq.target

and add
sidekiq.target:

[Install]
WantedBy=multi-user.target

That will enable stopping or starting all sidekiq processes with a single command
sudo systemctl stop/start sidekiq.target

[gingerlime]

@misdoro +1 That's pretty much what we're using, but instead of just using sequential ids for each worker, you can even assign a worker per queue (although this might not be the most flexible solution, depending on your needs)

[Unit]
Description=sidekiq %I
# start us only once the network and logging subsystems are available,
# consider adding redis-server.service if Redis is local and systemd-managed.
After=syslog.target network.target
PartOf=sidekiq.target
ReloadPropagatedFrom=sidekiq.target

# See these pages for lots of options:
#
#   https://www.freedesktop.org/software/systemd/man/systemd.service.html
#   https://www.freedesktop.org/software/systemd/man/systemd.exec.html
#
# THOSE PAGES ARE CRITICAL FOR ANY LINUX DEVOPS WORK; read them multiple
# times! systemd is a critical tool for all developers to know and understand.
#
[Service]
Type=simple
WorkingDirectory=/var/local/app
RuntimeDirectory=sidekiq

# NOTE: %I will be replaced by the queue name used when initializing, e.g.
#       systemctl start sidekiq@batch.service
ExecStart=/usr/bin/bundle exec sidekiq -e production -q %I

# writes the main process id to a pid file
ExecStartPost=/bin/sh -c "echo $MAINPID > /run/sidekiq/sidekiq.%I.pid"

# use `systemctl reload sidekiq` to send the quiet signal to Sidekiq
# at the start of your deploy process.
ExecReload=/bin/kill -TSTP $MAINPID

User=www-data
Group=www-data
UMask=0002

# Greatly reduce Ruby memory fragmentation and heap usage
# https://www.mikeperham.com/2018/04/25/taming-rails-memory-bloat/
Environment=MALLOC_ARENA_MAX=2

# if we crash, restart
RestartSec=1
Restart=on-failure

# output goes to /var/log/syslog (does not seem to affect logging, controlled by Semantic Logger)
StandardOutput=syslog
StandardError=syslog

# This will default to "bundler" if we don't specify it
SyslogIdentifier=sidekiq

[Install]
WantedBy=sidekiq.target

[krasnoukhov]

Add systemd Type=notify support for swarm [#4511]

@mperham Could you please clarify what this means in Ent-Changes?

[mperham]

My comment was poor, partially because I forgot the work I had done. 😁 Boot notification and parent watchdog were implemented, child watchdog was not implemented because systemd can't map one watchdog onto N child processes.

[krasnoukhov]

Ok thanks, I guess that's good enough!