Windows build from source cannot find sourceforge dependencies

[raygard]

What did you do?

Attempted to build Pillow from source on Windows

What did you expect to happen?

Build should run OK

What actually happened?

winbuild\build_prepare.py could not download dependencies because SF_MIRROR https://iweb.dl.sourceforge.net is not online.

What are your OS, Python and Pillow versions?

• OS: Windows 10 Pro
• Python: 3.10.4
• Pillow: latest main source

Fetching https://iweb.dl.sourceforge.net/project/libjpeg-turbo/2.1.3/libjpeg-turbo-2.1.3.tar.gz (attempt 3)...
Traceback (most recent call last):
  File "E:\r\pil\PILdev\0530\Pillow-main\winbuild\build_prepare.py", line 588, in <module>
    build_dep_all()
  File "E:\r\pil\PILdev\0530\Pillow-main\winbuild\build_prepare.py", line 466, in build_dep_all
    script = build_dep(dep_name)
  File "E:\r\pil\PILdev\0530\Pillow-main\winbuild\build_prepare.py", line 431, in build_dep
    extract_dep(dep["url"], dep["filename"])
  File "E:\r\pil\PILdev\0530\Pillow-main\winbuild\build_prepare.py", line 391, in extract_dep
    raise RuntimeError(ex)
RuntimeError: <urlopen error [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond>

I find that the line SF_MIRROR = "https://iweb.dl.sourceforge.net" is the culprit.

Also, ping https://iweb.dl.sourceforge.net gets me:
Ping request could not find host https://iweb.dl.sourceforge.net. Please check the name and try again.

I changed to SF_MIRROR = "https://cytranet.dl.sourceforge.net" and it works.

I don't know if iweb is down for good or not. Maybe build_prepare.py can be modified to have fallback mirrors, or somehow detect which of several SF mirrors are up?

[hugovk]

Can we use the link from https://sourceforge.net/projects/libjpeg-turbo/files/2.1.3/ ?

That is:

https://sourceforge.net/projects/libjpeg-turbo/files/2.1.3/libjpeg-turbo-2.1.3.tar.gz/download

That should do auto-mirror selection. For example, I did a wget of that from here in Finland and it selected a mirror from neighbouring Sweden.

https://altushost-swe.dl.sourceforge.net/project/libjpeg-turbo/2.1.3/libjpeg-turbo-2.1.3.tar.gz

[raygard]

I tried wget https://sourceforge.net/projects/libjpeg-turbo/files/2.1.3/libjpeg-turbo-2.1.3.tar.gz/download just now (--version shows GNU Wget 1.21.2 built on mingw32) and just got the HTML download page. No actual package downloaded.
Also retried ping https://iweb.dl.sourceforge.net and now get:

Pinging sourceforge.mirror.iweb.ca [192.175.120.182] with 32 bytes of data:
Reply from 184.107.1.171: Destination host unreachable.

not quite the same as before but still no joy.

[nulano]

I do not see iweb on the mirrors list.

The SourceForge documentation says wget downloads with automatic mirror should be supported. The website being returned appears to be a HTTP 302 redirect, which does seem to be supported by urllib.request used in build_prepare.py.

>>> import urllib.request
>>> len(urllib.request.urlopen("https://sourceforge.net/projects/libjpeg-turbo/files/2.1.3/libjpeg-turbo-2.1.3.tar.gz/download").read())
2260756

[raygard]

@nulano, thank you! that works. I have created PR #6345 to fix this. I don't know how to test it other than to say it works on my machine. Would appreciate any help testing from others who build Pillow on Windows.

[radarhere]

It will be tested in the PR by AppVeyor and GitHub Actions.

[nulano]

It will be tested in the PR by AppVeyor and GitHub Actions.

Normally it would be, but in this case the files are already downloaded from https://github.com/python-pillow/pillow-depends which is why the issue did not affect the CI builds in the first place.

To test this, I would remove the --depends=C:\pillow-depends\ argument from

Pillow/.appveyor.yml

Line 32 in 561d7ca

c:\python37\python.exe c:\pillow\winbuild\build_prepare.py -v --depends=C:\pillow-depends\

and add it to

Pillow/.github/workflows/test-windows.yml

Line 75 in 561d7ca

& python.exe winbuild\build_prepare.py -v --python=$env:pythonLocation --srcdir

in a separate branch just for testing. If you have GitHub Actions and/or AppVeyor enabled in your fork, you can just link to the results run automatically after a push.

[raygard]

@nulano this looks to be above my comfort level with GitHub etc. I hope someone else can help me with this. I think the PR is fine but would love some help before it goes out to the world.

[nulano]

@nulano this looks to be above my comfort level with GitHub etc. I hope someone else can help me with this. I think the PR is fine but would love some help before it goes out to the world.

I've made the changes, see my response in the PR: #6345 (comment)

[raygard]

@hugovk, thank you for the original auto-mirror-download suggestion; I don't know why it didn't work with my (maybe outdated?) Windows port of wget. @nulano, thank you for all the help. You guys are awesome!

Now, do I need to do anything more on this? Are these mods to the PR something that I need to, um, push or merge or whatever? (I'm so behind on learning git / GitHub). I'm guessing if I click "Create pull request" on @nulano's changes, then it will create a new PR or update my existing one? If it creates a new PR, what do I do with the old one? And do I want a PR that changes the Windows build or test workflow? Do you want it to download all the dependencies over and over? Obviously I need a clue...

[radarhere]

Oh, no, we don't want to download all the dependencies over and over. Precisely to avoid this problem, where one of the dependencies might become unavailable. Instead, we host dependencies for our CI scripts over at https://github.com/python-pillow/pillow-depends

This discussion was just about temporarily ignoring pillow-depends, to demonstrate that your new URL does work. The PR doesn't need to contain that change.

[raygard]

Thanks, that’s what I thought. So I think I do not need to do anything more on this pull request.

…

On Wed, Jun 1, 2022 at 9:18 PM Andrew Murray ***@***.***> wrote: Oh, no, we don't want to download all the dependencies over and over. Precisely to avoid this problem, where one of the dependencies might become unavailable. Instead, we host dependencies for our CI scripts over at https://github.com/python-pillow/pillow-depends This discussion was just about temporarily ignoring pillow-depends, to demonstrate that your new URL does work. The PR doesn't need to contain that change. — Reply to this email directly, view it on GitHub <#6343 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF5CW6BGA7WIQUP47HRQTOTVNARXXANCNFSM5XPEUA6A> . You are receiving this because you authored the thread.Message ID: ***@***.***>