Thumbor 7 (#103)

* updated Dockerfile and docker-entrypoint based on #67 (comment) * moving from Semaphore to Github actions * using thumbor's built-in multiprocess * taking changes from #101 * moving docker images to Github (ghcr.io) * updated requirements with some tc (thumbor community) dependencies that support python 3 / thumbor 7 * added multiarch support (thanks @danquack and @mpdude) * added option to specify THUMBOR_HOST (thanks @roman-stelmakh-emesa) TODO * [x] [circus / multiprocess is broken](#67 (comment)) * [x] move dependencies to `requirements.txt` ? * [x] add more libs to `requirements.txt` (which?) * [x] is the conf file backwards compatible? * [x] ModuleNotFoundError: No module named 'urllib2' * [x] remotecv image isn't created/pushed? @danquack @mpdude ? * [ ] caching/performance @danquack @mpdude * [ ] testing --- * refs #67 Thumbor 7.0.0a2 * updated Dockerfile and docker-entrypoint based on #67 (comment) * updated push, so we will push alpha tags on this branch, but not overwriting `latest` (and also semaphore CI) * fixing procs to 1, to bypass circus issue... tests might pass? * updated semaphore vars * Revert "updated semaphore vars" This reverts commit 354a14f. * kk-multiprocess build * updating benchmarks * merging multiproc + 7a2 => 7a5 * fallback values * allowing custom commands in entrypoint * fixed command * empty commit * Restore OS packages * Currently working without using tc-aws * preparing for thumbor 7.0 * removed proc-requirements copy * I think we still need envtpl for the config * moving lib-requirements to requirements * using python 3.9 * cleanup; switching to nginx-proxy/nginx-proxy (github project moved) * removed build for deprecated nginx-proxy * apparently, the docker hub image still uses jwilder... * update cairosvg * envtpl version to 0.7.2 to fix AttributeError: module 'jinja2' has no attribute 'contextfunction' * Create GHA for building * bump to thumbor 7.4.7 (latest) * moved requirements to txt ; updated community versions to latest with python3 support * fixed typos * added bats tests to github actions * split bats install and run * nginx-proxy-cache image might be required for testing * fixes bats tests (#120) * refactor github action to run sequentially (#121) * using build script * based on #116 from @danquack (Thank you!) * one job runs sequentially * builds and loads * runs tests * pushes if branch is master or thumbor-7 (for now) using the build script * testing thumbor-7 push * removed semaphore ; update push for thumbor-7 * explicit branch names * adding gha caching * switching to ghcr.io * login to ghcr.io * testing push * trying explicit username/repo name * Revert "trying explicit username/repo name" This reverts commit 4f63c79. * Revert "testing push" This reverts commit 263740c. * Revert "Revert "testing push"" This reverts commit b6b95b5. * fix tags * removing local tag for push * change order * explicit registry name * another attempt to push to ghcr (using personal access token) * another attempt with GITHUB_TOKEN * switching to personal access token * docker login * use action * One more pass at trying to get this to work * Revert "One more pass at trying to get this to work" This reverts commit f40ef14. * Revert "use action" This reverts commit 39fe365. * Revert "docker login" This reverts commit 4a31dc4. * Remove Multi Arch support from SIMD (#126) * Disable multi arch support for SIMD images * updated docs + recipes for GHCR * adding retry to tests (setup using docker-compose can be unpredictable) * local tags for testing * base tag for testing * explicitly using :test tags * improve workflow to login to ghcr only when pushing * showing available images * better caching? * adding more info before running tests about images * switched from imgur (returns 429) to freeimage.host * fixes tests with new cached data * updated docs to remove imgur and use freeimage.host * Overwrite host address 0.0.0.0 with THUMBOR_HOST (#129) * Overwrite host address 0.0.0.0 with THUMBOR_HOST --------- Co-authored-by: Roman Stelmakh <47328012+roman-stelmakh-emesa@users.noreply.github.com> * refs ghcr.io * typo * cleanup circus * syncing remotecv with explicit thumbor version * tweak build script to add pull and adjust platform for remotecv * should fix remotecv problem --------- Co-authored-by: Yoav <yoav@gingerlime.com> Co-authored-by: Bernardo Heynemann <heynemann@gmail.com> Co-authored-by: maximka777 <maksim.borovskij@yandex.ru> Co-authored-by: Daniel Quackenbush <25692880+danquack@users.noreply.github.com> Co-authored-by: Roman Stelmakh <47328012+roman-stelmakh-emesa@users.noreply.github.com>
MinimalCompact · May 6, 2023 · d58fc82 · d58fc82
1 parent 03c2f2d
commit d58fc82
Show file tree

Hide file tree

Showing 33 changed files with 219 additions and 806 deletions.
diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml
@@ -0,0 +1,48 @@
+name: Build test and push
+on:
+  pull_request:
+  workflow_dispatch:
+permissions:
+  contents: read
+  packages: write
+jobs:
+  build-test-push:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v2
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v2
+      - name: get thumbor version
+        id: thumbor
+        run: |
+          echo "VERSION=$(cat thumbor/requirements.txt| grep thumbor== | cut -d= -f 3)" >> $GITHUB_OUTPUT
+      - name: build images and load them for testing
+        run: |
+          ./build --extra "--load --cache-from=type=gha --cache-to=type=gha" --branch "${GITHUB_REF_NAME}"
+      - name: install bats
+        run: |
+          git clone https://github.com/bats-core/bats-core.git
+          cd bats-core
+          sudo ./install.sh /usr/local
+      - name: run bats tests
+        uses: nick-fields/retry@v2
+        with:
+          timeout_minutes: 10
+          max_attempts: 3
+          command: |
+            docker image ls -a
+            for filename in tests/*.bats; do sudo bats --tap "$filename" || exit 1; done
+      - name: login to ghcr.io
+        if: github.head_ref == 'thumbor-7' || github.ref == 'refs/heads/master'
+        uses: docker/login-action@v2
+        with:
+          registry: ghcr.io
+          username: minimalcompact
+          password: ${{ secrets.GH_TOKEN }}
+      - name: push when master
+        if: github.ref == 'refs/heads/master'
+        run: |
+          ./build --multiarch --push --branch master
diff --git a/.semaphore/semaphore.yml b/.semaphore/semaphore.yml
diff --git a/README.md b/README.md
@@ -8,23 +8,25 @@ Key Features and Goals:
 
 * The latest version of thumbor and dependencies, in a docker image
 * Supports both solo thumbor and multiprocess in one image
-* [Nginx frontend docker image with built-in caching](nginx-proxy-cache/README.md), using [nginx-proxy](https://github.com/jwilder/nginx-proxy)
+* [Nginx frontend docker image with built-in caching](nginx-proxy-cache/README.md), using [nginx-proxy](https://github.com/nginx-proxy/nginx-proxy)
 * SIMD support via docker tags
 * remotecv docker image (for async smart cropping and feature detection)
 * Clear version tagging to match Thumbor versions
+* NOTE: from Thumbor 7.x images are no longer updated on docker hub and were moved to **ghcr.io**
+        `latest` will be deleted from docker hub eventually
 
 ## Quick Start
 
 ```
-$ docker run -p 80:80 minimalcompact/thumbor
-$ wget http://localhost/unsafe/500x150/i.imgur.com/Nfn80ck.png
+$ docker run -p 80:80 ghcr.io/minimalcompact/thumbor
+$ wget http://localhost/unsafe/500x150/iili.io/H8m6pHv.png
 ```
 
 multi-process
 
 ```
-$ docker run -p 80:80 -e THUMBOR_NUM_PROCESSES=8 minimalcompact/thumbor
-$ wget http://localhost/unsafe/500x150/i.imgur.com/Nfn80ck.png
+$ docker run -p 80:80 -e THUMBOR_NUM_PROCESSES=8 ghcr.io/minimalcompact/thumbor
+$ wget http://localhost/unsafe/500x150/iili.io/H8m6pHv.png
 
 ```
 

diff --git a/benchmarks/locust/Dockerfile b/benchmarks/locust/Dockerfile
@@ -1,6 +1,6 @@
-FROM python:3.6
+FROM python:3.9
 
-RUN python3.6 -m pip install locustio==0.11.0
+RUN python3.9 -m pip install locustio==0.14.5
 
 EXPOSE 8089
 EXPOSE 5557

diff --git a/benchmarks/locust/docker-compose.yml b/benchmarks/locust/docker-compose.yml
@@ -1,11 +1,10 @@
 version: "3"
 services:
   thumbor:
-    image: minimalcompact/thumbor
+    image: ghcr.io/minimalcompact/thumbor
     environment:
       # we can control the number of processes and threads via ENV
       - THUMBOR_NUM_PROCESSES=${THUMBOR_NUM_PROCESSES:-1}
-      - ENGINE_THREADPOOL_SIZE=${ENGINE_THREADPOOL_SIZE:-0}
       # this would allow CORS from any origin (you can restrict to specific origins if you want)
       - CORS_ALLOW_ORIGIN=*
       # returns a webp image if browser Accept headers match
@@ -27,7 +26,6 @@ services:
     environment:
       - THUMBOR_DOCKER_PROCS=${THUMBOR_DOCKER_PROCS:-1}
       - THUMBOR_NUM_PROCESSES=${THUMBOR_NUM_PROCESSES:-1}
-      - ENGINE_THREADPOOL_SIZE=${ENGINE_THREADPOOL_SIZE:-0}
     volumes:
       - ./scripts:/opt/scripts
       - ./reports:/opt/reports
@@ -41,7 +39,7 @@ services:
               --master
               --expect-slaves 3
               --only-summary
-              --csv=/opt/reports/${THUMBOR_DOCKER_PROCS}-${THUMBOR_NUM_PROCESSES}-${ENGINE_THREADPOOL_SIZE}"
+              --csv=/opt/reports/${THUMBOR_DOCKER_PROCS}-${THUMBOR_NUM_PROCESSES}"
   # this is the locust slave. We can control the number of slaves using docker-compose scale
   locust-slave:
     build:

diff --git a/benchmarks/locust/run_benchmark b/benchmarks/locust/run_benchmark
@@ -4,37 +4,18 @@ docker-compose build
 docker-compose down
 
 for procs in 1 3 6 9; do
-  for threads in 0 3 6; do
-    export THUMBOR_DOCKER_PROCS=1
+  for docker_procs in 1 3 6 9; do
+    export THUMBOR_DOCKER_PROCS=$docker_procs
     export THUMBOR_NUM_PROCESSES=$procs
-    export ENGINE_THREADPOOL_SIZE=$threads
 
     echo
-    echo "***************** BEGIN BENCHMARK WITH ${THUMBOR_DOCKER_PROCS} Docker processes, ${THUMBOR_NUM_PROCESSES} thumbor processes and ${ENGINE_THREADPOOL_SIZE} threads ********************"
+    echo "***************** BEGIN BENCHMARK WITH ${THUMBOR_DOCKER_PROCS} Docker processes, ${THUMBOR_NUM_PROCESSES} thumbor processes ********************"
     echo
     docker-compose up -d --scale locust-slave=3 --scale thumbor=${THUMBOR_DOCKER_PROCS} --force-recreate --always-recreate-deps --renew-anon-volumes
     docker-compose logs -f locust
     docker-compose down
     echo
-    echo "***************** END BENCHMARK WITH ${THUMBOR_DOCKER_PROCS} Docker processes, ${THUMBOR_NUM_PROCESSES} thumbor processes and ${ENGINE_THREADPOOL_SIZE} threads ********************"
-    echo
-  done
-done
-
-for procs in 3 6 9; do
-  for threads in 0 3 6; do
-    export THUMBOR_DOCKER_PROCS=$procs
-    export THUMBOR_NUM_PROCESSES=1
-    export ENGINE_THREADPOOL_SIZE=$threads
-
-    echo
-    echo "***************** BEGIN BENCHMARK WITH ${THUMBOR_DOCKER_PROCS} Docker processes, ${THUMBOR_NUM_PROCESSES} thumbor processes and ${ENGINE_THREADPOOL_SIZE} threads ********************"
-    echo
-    docker-compose up -d --scale locust-slave=3 --scale thumbor=${THUMBOR_DOCKER_PROCS} --force-recreate --always-recreate-deps --renew-anon-volumes
-    docker-compose logs -f locust
-    docker-compose down
-    echo
-    echo "***************** END BENCHMARK WITH ${THUMBOR_DOCKER_PROCS} Docker processes, ${THUMBOR_NUM_PROCESSES} thumbor processes and ${ENGINE_THREADPOOL_SIZE} threads ********************"
+    echo "***************** END BENCHMARK WITH ${THUMBOR_DOCKER_PROCS} Docker processes, ${THUMBOR_NUM_PROCESSES} thumbor processes ********************"
     echo
   done
 done

diff --git a/benchmarks/locust/scripts/benchmark.py b/benchmarks/locust/scripts/benchmark.py
@@ -1,4 +1,4 @@
-from locust import HttpLocust, TaskSet, task
+from locust import HttpLocust, TaskSet, task, between
 
 
 class HealthTaskSet(TaskSet):
@@ -17,3 +17,4 @@ def fit_in(self):
 
 class Benchmark(HttpLocust):
     task_set = SimpleTaskSet
+    wait_time = between(0, 0.1)
diff --git a/build b/build
@@ -1,48 +1,81 @@
 #!/bin/bash
 set -e
 
-echo "--> BUILDING minimalcompact/thumbor"
-docker build --pull -f thumbor/Dockerfile -t minimalcompact/thumbor thumbor/
-# introspecting version by running the built image
-THUMBOR_VERSION=`docker run --rm minimalcompact/thumbor bash -c "pip freeze |grep ^thumbor== | cut -d= -f 3"`
-echo "THUMBOR VERSION: $THUMBOR_VERSION"
+PLATFORM="linux/amd64"
+EXTRA_ARGUMENTS=""
+while true; do
+  case "$1" in
+    ### Push requires authentication to minimalcompact docker hub
+    -p | --push)
+        EXTRA_ARGUMENTS="${EXTRA_ARGUMENTS} --push"
+        shift
+        ;;
+    -b | --branch)
+        BRANCH="$2";
+        shift 2
+        ;;
+    ### Images have issues building multi arch, so hold behind a feature flag
+    -m | --multiarch)
+        PLATFORM="${PLATFORM},linux/arm64"
+        shift
+        ;;
+    ## Create should be run once, or in ci, if docker buildx ls doesnt contain supported platforms
+    -c | --create)
+         docker buildx create --use
+         shift
+         ;;
+    -e | --extra )
+        EXTRA_ARGUMENTS="${EXTRA_ARGUMENTS} $2"
+        shift 2
+        ;;
+     * ) break ;;
+  esac
+done
+
+THUMBOR_VERSION=$(cat thumbor/requirements.txt| grep thumbor== | cut -d= -f 3)
 
-echo "--> TAGGING minimalcompact/thumbor:$THUMBOR_VERSION"
-docker tag minimalcompact/thumbor minimalcompact/thumbor:$THUMBOR_VERSION
-echo "--> TAGGING minimalcompact/thumbor:latest"
-docker tag minimalcompact/thumbor minimalcompact/thumbor:latest
+echo "Branch: ${BRANCH} | Thumbor version: ${THUMBOR_VERSION}"
+echo "*******************************************************"
+
+echo "--> BUILDING minimalcompact/thumbor-nginx-proxy-cache"
+[ "$BRANCH" == "master" ] && EXTRA_TAG="--tag ghcr.io/minimalcompact/thumbor-nginx-proxy-cache:latest"
+docker buildx build $EXTRA_ARGUMENTS --platform $PLATFORM  --pull -f nginx-proxy-cache/Dockerfile \
+ --tag ghcr.io/minimalcompact/thumbor-nginx-proxy-cache:test \
+ --tag ghcr.io/minimalcompact/thumbor-nginx-proxy-cache:$THUMBOR_VERSION \
+ $EXTRA_TAG \
+ nginx-proxy-cache/
+
+echo "--> BUILDING minimalcompact/thumbor"
+[ "$BRANCH" == "master" ] && EXTRA_TAG="--tag ghcr.io/minimalcompact/thumbor:latest"
+docker buildx build $EXTRA_ARGUMENTS --platform $PLATFORM --pull -f thumbor/Dockerfile \
+ --tag ghcr.io/minimalcompact/thumbor:test \
+ --tag ghcr.io/minimalcompact/thumbor:$THUMBOR_VERSION \
+  $EXTRA_TAG \
+ thumbor/
 
 echo "--> BUILDING minimalcompact/thumbor:simd-sse4"
-docker build --build-arg SIMD_LEVEL=sse4 -f thumbor/Dockerfile -t minimalcompact/thumbor-simd-sse4 thumbor/
-echo "--> TAGGING minimalcompact/thumbor:$THUMBOR_VERSION-simd-sse4"
-docker tag minimalcompact/thumbor-simd-sse4 minimalcompact/thumbor:$THUMBOR_VERSION-simd-sse4
-echo "--> TAGGING minimalcompact/thumbor:latest-simd-sse4"
-docker tag minimalcompact/thumbor-simd-sse4 minimalcompact/thumbor:latest-simd-sse4
+[ "$BRANCH" == "master" ] && EXTRA_TAG="--tag ghcr.io/minimalcompact/thumbor:latest-simd-sse4t"
+## SSE doesnt support multi arch
+docker buildx build $EXTRA_ARGUMENTS --platform linux/amd64 --build-arg SIMD_LEVEL=sse4 --pull -f thumbor/Dockerfile \
+ --tag ghcr.io/minimalcompact/thumbor:$THUMBOR_VERSION-simd-sse4 \
+ $EXTRA_TAG \
+ thumbor/
 
 echo "--> BUILDING minimalcompact/thumbor:simd-avx2"
-docker build --build-arg SIMD_LEVEL=avx2  -f thumbor/Dockerfile -t minimalcompact/thumbor-simd-avx2 thumbor/
-echo "--> TAGGING minimalcompact/thumbor:$THUMBOR_VERSION-simd-avx2"
-docker tag minimalcompact/thumbor-simd-avx2 minimalcompact/thumbor:$THUMBOR_VERSION-simd-avx2
-echo "--> TAGGING minimalcompact/thumbor:latest-simd-avx2"
-docker tag minimalcompact/thumbor-simd-avx2 minimalcompact/thumbor:latest-simd-avx2
-
-echo "--> BUILDING minimalcompact/thumbor-nginx-proxy (DEPRECATED)"
-docker build --pull -f nginx-proxy/Dockerfile -t minimalcompact/thumbor-nginx-proxy nginx-proxy/
-echo "--> TAGGING minimalcompact/thumbor-nginx-proxy:$THUMBOR_VERSION"
-docker tag minimalcompact/thumbor-nginx-proxy minimalcompact/thumbor-nginx-proxy:$THUMBOR_VERSION
-echo "--> TAGGING minimalcompact/thumbor-nginx-proxy:latest"
-docker tag minimalcompact/thumbor-nginx-proxy minimalcompact/thumbor-nginx-proxy:latest
-
-echo "--> BUILDING minimalcompact/thumbor-nginx-proxy-cache"
-docker build --pull -f nginx-proxy-cache/Dockerfile -t minimalcompact/thumbor-nginx-proxy-cache nginx-proxy-cache/
-echo "--> TAGGING minimalcompact/thumbor-nginx-proxy-cache:$THUMBOR_VERSION"
-docker tag minimalcompact/thumbor-nginx-proxy-cache minimalcompact/thumbor-nginx-proxy-cache:$THUMBOR_VERSION
-echo "--> TAGGING minimalcompact/thumbor-nginx-proxy-cache:latest"
-docker tag minimalcompact/thumbor-nginx-proxy-cache minimalcompact/thumbor-nginx-proxy-cache:latest
+[ "$BRANCH" == "master" ] && EXTRA_TAG="--tag ghcr.io/minimalcompact/thumbor:latest-simd-avx2"
+## AVX2 Doesnt support multi arch
+docker buildx build $EXTRA_ARGUMENTS --platform linux/amd64 --build-arg SIMD_LEVEL=avx2 --pull -f thumbor/Dockerfile \
+ --tag ghcr.io/minimalcompact/thumbor:$THUMBOR_VERSION-simd-avx2 \
+ $EXTRA_TAG \
+ thumbor/
 
 echo "--> BUILDING minimalcompact/remotecv"
-docker build --build-arg THUMBOR_TAG=latest -f remotecv/Dockerfile -t minimalcompact/remotecv remotecv/
-echo "--> TAGGING minimalcompact/remotecv:$THUMBOR_VERSION"
-docker tag minimalcompact/remotecv minimalcompact/remotecv:$THUMBOR_VERSION
-echo "--> TAGGING minimalcompact/remotecv:latest"
-docker tag minimalcompact/remotecv minimalcompact/remotecv:latest
+[ "$BRANCH" == "master" ] && EXTRA_TAG="--tag ghcr.io/minimalcompact/remotecv:latest"
+docker buildx build $EXTRA_ARGUMENTS --platform $PLATFORM --build-arg THUMBOR_TAG=$THUMBOR_VERSION --pull -f remotecv/Dockerfile \
+ --tag ghcr.io/minimalcompact/remotecv:$THUMBOR_VERSION \
+ $EXTRA_TAG \
+ remotecv/
+
+echo "*******************************************************"
+echo Docker images:
+docker image ls -a
diff --git a/nginx-proxy-cache/README.md b/nginx-proxy-cache/README.md
@@ -1,10 +1,10 @@
 # Minimal Compact thumbor nginx proxy cache
 
-A caching proxy for thumbor based on [nginx-proxy](https://github.com/jwilder/nginx-proxy)
+A caching proxy for thumbor based on [nginx-proxy](https://github.com/nginx-proxy/nginx-proxy)
 
 ## Basics
 
-[nginx-proxy](https://github.com/jwilder/nginx-proxy) provides a way to dynamically attach docker images and get them proxied by Nginx.
+[nginx-proxy](https://github.com/nginx-proxy/nginx-proxy) provides a way to dynamically attach docker images and get them proxied by Nginx.
 
 This image adds a proxy caching layer into `nginx-proxy`, using the built-in `proxy_cache` directive.
 

diff --git a/nginx-proxy/Dockerfile b/nginx-proxy/Dockerfile
diff --git a/nginx-proxy/deprecated.sh b/nginx-proxy/deprecated.sh
diff --git a/nginx-proxy/imgpath.js b/nginx-proxy/imgpath.js