Production Guidance

[btoconnor]

Is there any sort of guidance available for deploying a shipit installation to production? We got our shipit installation up and running, and for all intents and purposes it seems to work.

However, there seems to be something going on when we launch this in production behind nginx and a load balancer. After a handful of requests, the shipit application seems to hang. I suspect this has to do with my configuration of Puma and nginx, but I'm not entirely sure how to resolve it.

I see a bunch of logs like this:

I, [2018-12-07T19:59:46.960891 #8060]  INFO -- : [Pubsubstub::StreamAction] Starting heartbeat
I, [2018-12-07T19:59:46.961431 #8060]  INFO -- : [Pubsubstub::Subscription] Connecting client #7750495252574393681 (stack.4)
D, [2018-12-07T19:59:46.961553 #8060] DEBUG -- : [Pubsubstub::Subscription] Sending event #1544212786961 to client #7750495252574393681
I, [2018-12-07T19:59:46.961000 #8060]  INFO -- : [Pubsubstub::StreamAction] Starting subscriber
I, [2018-12-07T19:59:46.965115 #8060]  INFO -- : [Pubsubstub::Subscriber] Subscribed to *.pubsub
I, [2018-12-07T19:59:51.095810 #8060]  INFO -- : [Pubsubstub::Subscription] Connecting client #12530072831261279246 (stack.4)
D, [2018-12-07T19:59:51.095899 #8060] DEBUG -- : [Pubsubstub::Subscription] Sending event #1544212791095 to client #12530072831261279246
I, [2018-12-07T19:59:54.324114 #8060]  INFO -- : [Pubsubstub::Subscription] Connecting client #4240903384686625976 (stack.4)
D, [2018-12-07T19:59:54.324189 #8060] DEBUG -- : [Pubsubstub::Subscription] Sending event #1544212794324 to client #4240903384686625976
I, [2018-12-07T19:59:55.794921 #8060]  INFO -- : [Pubsubstub::Subscription] Connecting client #1068417308053608841 (stack.4)
D, [2018-12-07T19:59:55.795003 #8060] DEBUG -- : [Pubsubstub::Subscription] Sending event #1544212795794 to client #1068417308053608841
I, [2018-12-07T19:59:58.054863 #8060]  INFO -- : [Pubsubstub::Subscription] Connecting client #15653348681481192242 (stack.4)
D, [2018-12-07T19:59:58.054943 #8060] DEBUG -- : [Pubsubstub::Subscription] Sending event #1544212798054 to client #15653348681481192242
D, [2018-12-07T20:00:01.961373 #8060] DEBUG -- : [Pubsubstub::Subscription] Sending event #1544212801961 to client #1068417308053608841
D, [2018-12-07T20:00:01.961484 #8060] DEBUG -- : [Pubsubstub::Subscription] Sending event #1544212801961 to client #4240903384686625976
D, [2018-12-07T20:00:01.961572 #8060] DEBUG -- : [Pubsubstub::Subscription] Sending event #1544212801961 to client #7750495252574393681
D, [2018-12-07T20:00:01.961624 #8060] DEBUG -- : [Pubsubstub::Subscription] Sending event #1544212801961 to client #15653348681481192242
D, [2018-12-07T20:00:01.961756 #8060] DEBUG -- : [Pubsubstub::Subscription] Sending event #1544212801961 to client #12530072831261279246
D, [2018-12-07T20:00:16.961546 #8060] DEBUG -- : [Pubsubstub::Subscription] Sending event #1544212816961 to client #7750495252574393681
D, [2018-12-07T20:00:16.961862 #8060] DEBUG -- : [Pubsubstub::Subscription] Sending event #1544212816961 to client #4240903384686625976
I, [2018-12-07T20:00:16.961960 #8060]  INFO -- : [Pubsubstub::Subscription] Disconnecting client #7750495252574393681
I, [2018-12-07T20:00:16.962196 #8060]  INFO -- : [Pubsubstub::Subscription] Disconnecting client #4240903384686625976
D, [2018-12-07T20:00:16.961659 #8060] DEBUG -- : [Pubsubstub::Subscription] Sending event #1544212816961 to client #15653348681481192242
D, [2018-12-07T20:00:16.961737 #8060] DEBUG -- : [Pubsubstub::Subscription] Sending event #1544212816961 to client #12530072831261279246
D, [2018-12-07T20:00:16.961778 #8060] DEBUG -- : [Pubsubstub::Subscription] Sending event #1544212816961 to client #1068417308053608841
I, [2018-12-07T20:00:16.971113 #8060]  INFO -- : [Pubsubstub::Subscription] Disconnecting client #12530072831261279246
I, [2018-12-07T20:00:16.971196 #8060]  INFO -- : [Pubsubstub::Subscription] Disconnecting client #1068417308053608841

Any advice?

[byroot]

Hi. Seems like all the puma threads are used for the SSE (server push) connections.

You can try to increase the number of puma threads, or alternatively you can move the long lived connections in a separate process #496

The later is a bit more setup but is recommended for serious use.

[btoconnor]

Thanks for getting back to me. I had indeed figured it was the threads being exhausted. For now we've increased the maximum threads for a puma worker, and will investigate setting up a dedicated server just for SSE.

On a similar note - I noticed that the rails template used to create the application provides a sidekiq configuration to utilize 1 thread per sidekiq worker. Is that essential for running Shipit? Is it safe to increase thread count, or do we need to run multiple sidekiq workers to be able to do more than 1 deploy per shipit instance?

[byroot]

Is it safe to increase thread count

Yes absolutely. Historically we started with unicorn & resque, so weren't sure we were thread safe. But we're now running puma & sidekiq with several threads since a while and we're fairly confident thread safety issues have been ironed out.

I'll update that when I fix the template.rb.

[ohsabry]

Is it safe to crank :concurrency to close or equal to the AR connection pool size?

[casperisfine]

Yes. But ideally you want to monitor your application, because if you crank up the number of threads too high, you might lower performance (because GIL).