From 8c1fccdf8859593861577e610eda669f2e11f800 Mon Sep 17 00:00:00 2001 From: Stephen Gutekanst Date: Wed, 2 Oct 2019 17:01:51 -0700 Subject: [PATCH] use http-server-stabilizer + 4 worker subprocesses This change makes syntect_server resilient to the two classes of problems we've seen in production usage of it: 1. Some specific language grammar/file pairs can cause syntect to panic internally. This is usually because syntect doesn't implement a specific sublime-syntax feature in some way and it [panics instead of returning result types](https://github.com/trishume/syntect/issues/98). 2. Much rarer, some specific language/grammar file pairs can cause syntect to get stuck in an infinite loop internally -- never to return and consuming an entire CPU core until it is restarted manually. Previously we tried to solve #1 through stack unwinding (c5773da), but since the 2nd issue above also appeared it proved to not be sufficient on its own. It is still useful, though, because it can do per-request recovery of the first failure scenario above and as such it will be added back in. Even without stack unwinding, http-server-stabilizer helps both cases above by running and monitoring replicas of syntect_server. See the README in https://github.com/slimsag/http-server-stabilizer for details. It is important to note that all this does is stop these individual file failures from harming other requests to syntect_server. They are still issues on their own, and logging and Prometheus monitoring is now in place for us to identify when this is occurring and in which file it occurred so we can track down the issue and make small reproduction cases to file and fix upstream. Since only one instance of syntect_server was previously running and we now run multiple, more memory is needed. Each instance requires about 1.1 GB at peak (depending on which languages are used). The default is now to run 4 workers, so 4.4 GB is the minimum required and 6 GB is suggested. In the event only one worker is ran (via setting the env var `WORKERS=1`), stability is still greatly improved since the 2nd failure case above can only last a short period of time instead of until the container is restarted manually. Part of sourcegraph/sourcegraph#5406 --- Dockerfile | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/Dockerfile b/Dockerfile index faca4ae..ce9ba45 100644 --- a/Dockerfile +++ b/Dockerfile @@ -18,11 +18,22 @@ WORKDIR /repo RUN env 'CC_x86_64-unknown-linux-musl=musl-gcc' cargo rustc --release --target x86_64-unknown-linux-musl -- -C 'linker=musl-gcc' RUN cp ./target/x86_64-unknown-linux-musl/release/syntect_server /syntect_server +################################ +# Build http-server-stabilizer # +################################ +FROM golang:1.13.1-alpine@sha256:2293e952c79b8b3a987e1e09d48b6aa403d703cef9a8fa316d30ba2918d37367 as hss + +RUN apk add --no-cache git=2.22.0-r0 +RUN git clone https://github.com/slimsag/http-server-stabilizer /repo +WORKDIR /repo +RUN git checkout v1.0.0 && go build -o /http-server-stabilizer . + ####################### # Compile final image # ####################### FROM sourcegraph/alpine:3.9@sha256:e9264d4748e16de961a2b973cc12259dee1d33473633beccb1dfb8a0e62c6459 COPY --from=ss syntect_server / +COPY --from=hss http-server-stabilizer / # Use tini (https://github.com/krallin/tini) for proper signal handling. RUN apk add --no-cache tini=0.18.0-r0 @@ -30,7 +41,6 @@ ENTRYPOINT ["/sbin/tini", "--"] EXPOSE 9238 ENV ROCKET_ENV "production" -ENV ROCKET_PORT 9238 ENV ROCKET_LIMITS "{json=10485760}" # syntect_server does not need a secret key since it uses no cookies, but @@ -46,4 +56,11 @@ ENV ROCKET_SECRET_KEY "+SecretKeyIsIrrelevantAndUnusedPleaseIgnore=" # what we observed when this was enabled with the default 5s. ENV ROCKET_KEEP_ALIVE=0 -CMD ["/syntect_server"] +# The more workers, the more resilient syntect_server is to getting stuck on +# bad grammar/file combinations. If it happens with four workers, only 1/4th of +# requests will be affected for a short period of time. Each worker can require +# at peak around 1.1 GiB of memory. +ENV WORKERS=4 + +ENV QUIET=true +CMD ["sh", "-c", "/http-server-stabilizer -listen=:9238 -prometheus-app-name=syntect_server -workers=$WORKERS -- env ROCKET_PORT={{.Port}} /syntect_server"]