Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to docker buildx with multi arch flag #116

Closed
wants to merge 3 commits into from

Conversation

danquack
Copy link
Contributor

@danquack danquack commented Apr 11, 2023

Docker buildx is dockers recommended way to build with multi arch, which relates to #102
Since there are still issues with multi arch, I put it behind a feature flag.

The suggested execution now is:
OLD bash build master
NEW bash build -p -b master

To test if multi arch builds, you can pass the -m flag.

@danquack
Copy link
Contributor Author

I expect the pipeline to fail, since the script is changing. @gingerlime or @heynemann would you mind checking this out?

@gingerlime
Copy link
Contributor

gingerlime commented Apr 25, 2023

Hey @danquack. Thanks for your contribution and sorry for the delay. It's a bit of a difficult time for me right now. Regardless, I'm not entirely up-to-speed with the latest thumbor nor docker etc, but hope that with your help we can get things moving forward.

Looking at the build on Semaphore (I guess you cannot see it?), it fails with these errors:

#0 26.11 ERROR: Cannot install -r /app/requirements.txt (line 10), -r /app/requirements.txt (line 12) and -r /app/requirements.txt (line 2) because these package versions have conflicting dependencies.
#0 26.12 
#0 26.12 The conflict is caused by:
#0 26.12     remotecv 3.0.0 depends on Pillow<8.0.0 and >=7.0.0
#0 26.12     cairosvg 2.5.2 depends on pillow
#0 26.12     thumbor 7.0.0 depends on Pillow<9 and >=8.4.0
#0 26.12 
#0 26.12 To fix this you could try to:
#0 26.12 1. loosen the range of package versions you've specified
#0 26.12 2. remove package versions to allow pip attempt to solve the dependency conflict
#0 26.12 
#0 26.12 ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
#0 26.22 WARNING: You are using pip version 22.0.4; however, version 23.0.1 is available.
#0 26.22 You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
------
Dockerfile:34
--------------------
  32 |     
  33 |     COPY requirements.txt /app/requirements.txt
  34 | >>> RUN pip install --trusted-host None --no-cache-dir -r /app/requirements.txt
  35 |     RUN pip install --no-dependencies tc-aws==6.2.15
  36 |     RUN pip install --no-dependencies tc-core==0.4.1
--------------------
ERROR: failed to solve: process "/bin/sh -c pip install --trusted-host None --no-cache-dir -r /app/requirements.txt" did not complete successfully: exit code: 1

It probably makes sense to switch things to github actions, so there's better visibility. I hope to get round to it, but hopefully the semaphore yml files are fairly straight-forward to understand so if someone else can step in, it can help speed things up?

@gingerlime gingerlime mentioned this pull request Apr 25, 2023
8 tasks
This was referenced Apr 25, 2023
@danquack
Copy link
Contributor Author

Looking at the build on Semaphore (I guess you cannot see it?), it fails with these errors:

#0 26.11 ERROR: Cannot install -r /app/requirements.txt (line 10), -r /app/requirements.txt (line 12) and -r /app/requirements.txt (line 2) because these package versions have conflicting dependencies.
#0 26.12 
#0 26.12 The conflict is caused by:
#0 26.12     remotecv 3.0.0 depends on Pillow<8.0.0 and >=7.0.0
#0 26.12     cairosvg 2.5.2 depends on pillow
#0 26.12     thumbor 7.0.0 depends on Pillow<9 and >=8.4.0
#0 26.12 
#0 26.12 To fix this you could try to:
#0 26.12 1. loosen the range of package versions you've specified
#0 26.12 2. remove package versions to allow pip attempt to solve the dependency conflict
#0 26.12 
#0 26.12 ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
#0 26.22 WARNING: You are using pip version 22.0.4; however, version 23.0.1 is available.
#0 26.22 You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
------
Dockerfile:34
--------------------
  32 |     
  33 |     COPY requirements.txt /app/requirements.txt
  34 | >>> RUN pip install --trusted-host None --no-cache-dir -r /app/requirements.txt
  35 |     RUN pip install --no-dependencies tc-aws==6.2.15
  36 |     RUN pip install --no-dependencies tc-core==0.4.1
--------------------
ERROR: failed to solve: process "/bin/sh -c pip install --trusted-host None --no-cache-dir -r /app/requirements.txt" did not complete successfully: exit code: 1

I created #119, as I am not sure there's a python match around it.

@gingerlime
Copy link
Contributor

I guess we can close this and move things over to github actions? let me know if you think it's worth re-opening however.

@gingerlime gingerlime closed this Apr 26, 2023
@danquack danquack deleted the thumbor-7 branch April 26, 2023 16:50
gingerlime pushed a commit that referenced this pull request Apr 28, 2023
* based on #116
  from @danquack (Thank you!)
* one job runs sequentially
* builds and loads
* runs tests
* pushes if branch is master or thumbor-7 (for now) using the build
  script
gingerlime added a commit that referenced this pull request Apr 28, 2023
* using build script
* based on #116
  from @danquack (Thank you!)
* one job runs sequentially
* builds and loads
* runs tests
* pushes if branch is master or thumbor-7 (for now) using the build
  script
gingerlime added a commit that referenced this pull request May 6, 2023
* updated Dockerfile and docker-entrypoint based on
  #67 (comment)
* moving from Semaphore to Github actions
* using thumbor's built-in multiprocess
* taking changes from #101
* moving docker images to Github (ghcr.io)
* updated requirements with some tc (thumbor community) dependencies that support python 3 / thumbor 7
* added multiarch support (thanks @danquack and @mpdude)
* added option to specify THUMBOR_HOST (thanks @roman-stelmakh-emesa)

TODO

* [x] [circus / multiprocess is broken](#67 (comment))
* [x] move dependencies to `requirements.txt` ?
* [x] add more libs to `requirements.txt` (which?)
* [x] is the conf file backwards compatible?
* [x] ModuleNotFoundError: No module named 'urllib2'
* [x] remotecv image isn't created/pushed? @danquack @mpdude ?
* [ ] caching/performance @danquack @mpdude 
* [ ] testing

---

* refs #67 Thumbor 7.0.0a2

* updated Dockerfile and docker-entrypoint based on
  #67 (comment)
* updated push, so we will push alpha tags on this branch, but not
  overwriting `latest` (and also semaphore CI)

* fixing procs to 1, to bypass circus issue... tests might pass?

* updated semaphore vars

* Revert "updated semaphore vars"

This reverts commit 354a14f.

* kk-multiprocess build

* updating benchmarks

* merging multiproc + 7a2 => 7a5

* fallback values

* allowing custom commands in entrypoint

* fixed command

* empty commit

* Restore OS packages

* Currently working without using tc-aws

* preparing for thumbor 7.0

* removed proc-requirements copy

* I think we still need envtpl for the config

* moving lib-requirements to requirements

* using python 3.9

* cleanup; switching to nginx-proxy/nginx-proxy (github project moved)

* removed build for deprecated nginx-proxy

* apparently, the docker hub image still uses jwilder...

* update cairosvg

* envtpl version to 0.7.2 to fix AttributeError: module 'jinja2' has no attribute 'contextfunction'

* Create GHA for building

* bump to thumbor 7.4.7 (latest)

* moved requirements to txt ; updated community versions to latest with python3 support

* fixed typos

* added bats tests to github actions

* split bats install and run

* nginx-proxy-cache image might be required for testing

* fixes bats tests (#120)

* refactor github action to run sequentially (#121)

* using build script
* based on #116
  from @danquack (Thank you!)
* one job runs sequentially
* builds and loads
* runs tests
* pushes if branch is master or thumbor-7 (for now) using the build
  script

* testing thumbor-7 push

* removed semaphore ; update push for thumbor-7

* explicit branch names

* adding gha caching

* switching to ghcr.io

* login to ghcr.io

* testing push

* trying explicit username/repo name

* Revert "trying explicit username/repo name"

This reverts commit 4f63c79.

* Revert "testing push"

This reverts commit 263740c.

* Revert "Revert "testing push""

This reverts commit b6b95b5.

* fix tags

* removing local tag for push

* change order

* explicit registry name

* another attempt to push to ghcr (using personal access token)

* another attempt with GITHUB_TOKEN

* switching to personal access token

* docker login

* use action

* One more pass at trying to get this to work

* Revert "One more pass at trying to get this to work"

This reverts commit f40ef14.

* Revert "use action"

This reverts commit 39fe365.

* Revert "docker login"

This reverts commit 4a31dc4.

* Remove Multi Arch support from SIMD (#126)

* Disable multi arch support for SIMD images

* updated docs + recipes for GHCR

* adding retry to tests (setup using docker-compose can be unpredictable)

* local tags for testing

* base tag for testing

* explicitly using :test tags

* improve workflow to login to ghcr only when pushing

* showing available images

* better caching?

* adding more info before running tests about images

* switched from imgur (returns 429) to freeimage.host

* fixes tests with new cached data

* updated docs to remove imgur and use freeimage.host

* Overwrite host address 0.0.0.0 with THUMBOR_HOST (#129)

* Overwrite host address 0.0.0.0 with THUMBOR_HOST

---------

Co-authored-by: Roman Stelmakh <47328012+roman-stelmakh-emesa@users.noreply.github.com>

* refs ghcr.io

* typo

* cleanup circus

* syncing remotecv with explicit thumbor version

* tweak build script to add pull and adjust platform for remotecv

* should fix remotecv problem

---------

Co-authored-by: Yoav <yoav@gingerlime.com>
Co-authored-by: Bernardo Heynemann <heynemann@gmail.com>
Co-authored-by: maximka777 <maksim.borovskij@yandex.ru>
Co-authored-by: Daniel Quackenbush <25692880+danquack@users.noreply.github.com>
Co-authored-by: Roman Stelmakh <47328012+roman-stelmakh-emesa@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants