Add support for verifying downloaded file's SHA-512 hash

[EricRahm]

Each build has an associated <product>...<locale><platform>.checksums file. The SHA-512 hash contained in the file could be used to verify the hash of the download.

Proposal:

• Add a |--verify| flag
• Download the checksums file and verify the the SHA-512 hash for the download matches
• If the hash fails follow the standard retry logic already in place

[whimboo]

@EricRahm I assume this hash only applies to the installer binaries to download, but not all the other files including the .zip file for Windows.

[EricRahm]

@whimboo There's a hash for every build file. For example: https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015/01/2015-01-29-03-02-02-mozilla-central/firefox-38.0a1.en-US.win64-x86_64.checksums includes both the installer and the zip.

[whimboo]

Ah, great. Thanks. Sounds like a good improvement.

[parthbakshi]

I would like to work on this bug, whats the best way to work on it.

[EricRahm]

@whimboo Do you have any notes for @parthbakshi?

[whimboo]

Hi @parthbakshi. Great to hear that you have interests to work on this issue! Sorry for the late response but I was away for a full month.

So the best way is to fork the repository and create a local clone of it. Then you should make yourself familiar with the different kinds of Firefox builds (Nightly, Aurora, Beta, Release, candidate builds, tinderbox builds) which are supported by mozdownload. Once done it would be good to know how this checksum file looks like for those different kinds of builds. After that the download() method of the scraper would have to be extended to being able to also get the checksum file, parse the entries, and do the checks if specified via the command line (--verify).

Let me know if you have further questions. You can also reach me on the moznet IRC network in the #automation channel.

[parthbakshi]

Ok i have forked the repo, Is there a doc where i can read about the various builds and how the checksums/SHA-512 actually look?

[whimboo]

The different builds are also called channels. You can get short introductions on MDN. Maybe by following the links you will get more comprehensive information. For the checksum you might want to have a look at the hashlib module documentation of Python.

[parthbakshi]

Whenever i run the mozdownload on my machine im getting an error executing it, is there anything that i am missing?

(venv)pbakshi@parthbakshi:~/personal-projects/mozilla/mozdownload$ ../venv/bin/mozdownload --version=latest
Traceback (most recent call last):
  File "../venv/bin/mozdownload", line 9, in <module>
    load_entry_point('mozdownload==1.14', 'console_scripts', 'mozdownload')()
  File "/home/pbakshi/personal-projects/mozilla/venv/lib/python2.6/site-packages/pkg_resources/__init__.py", line 546, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/home/pbakshi/personal-projects/mozilla/venv/lib/python2.6/site-packages/pkg_resources/__init__.py", line 2666, in load_entry_point
    return ep.load()
  File "/home/pbakshi/personal-projects/mozilla/venv/lib/python2.6/site-packages/pkg_resources/__init__.py", line 2339, in load
    return self.resolve()
  File "/home/pbakshi/personal-projects/mozilla/venv/lib/python2.6/site-packages/pkg_resources/__init__.py", line 2345, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/pbakshi/personal-projects/mozilla/venv/lib/python2.6/site-packages/mozdownload-1.14-py2.6.egg/mozdownload/__init__.py", line 5, in <module>
    from scraper import *
  File "/home/pbakshi/personal-projects/mozilla/venv/lib/python2.6/site-packages/mozdownload-1.14-py2.6.egg/mozdownload/scraper.py", line 19, in <module>
    import mozlog
  File "/home/pbakshi/personal-projects/mozilla/venv/lib/python2.6/site-packages/mozlog-2.11-py2.6.egg/mozlog/__init__.py", line 19, in <module>
    import structured
  File "/home/pbakshi/personal-projects/mozilla/venv/lib/python2.6/site-packages/mozlog-2.11-py2.6.egg/mozlog/structured/__init__.py", line 5, in <module>
    import commandline
  File "/home/pbakshi/personal-projects/mozilla/venv/lib/python2.6/site-packages/mozlog-2.11-py2.6.egg/mozlog/structured/commandline.py", line 10, in <module>
    from structuredlog import StructuredLogger, set_default_logger
  File "/home/pbakshi/personal-projects/mozilla/venv/lib/python2.6/site-packages/mozlog-2.11-py2.6.egg/mozlog/structured/structuredlog.py", line 14, in <module>
    from logtypes import Unicode, TestId, Status, SubStatus, Dict, List, Int, Any
  File "/home/pbakshi/personal-projects/mozilla/venv/lib/python2.6/site-packages/mozlog-2.11-py2.6.egg/mozlog/structured/logtypes.py", line 97
    known_kwargs = {name: value for name, value in kwargs.iteritems()
                                  ^
SyntaxError: invalid syntax

[whimboo]

As mentioned on IRC this is most likely Python version 2.6 which is causing those problems. Please upgrade to latest 2.7 and it should work.

[whimboo]

@parthbakshi, given the amount of time which has been passed by here, I assume you will not work on this issue anymore? It's somewhat important for us now given that in the last days we had a couple of corrupted downloads without seeing a failure from mozdownload.

[parthbakshi]

Sorry work got in the way you can reassign it to someone else.

On Thu, Oct 29, 2015 at 6:03 PM Henrik Skupin notifications@github.com
wrote:

@parthbakshi https://github.com/parthbakshi, given the amount of time
which has been passed by here, I assume you will not work on this issue
anymore? It's somewhat important for us now given that in the last days we
had a couple of corrupted downloads without seeing a failure from
mozdownload.

—
Reply to this email directly or view it on GitHub
#241 (comment)
.

[whimboo]

Thanks for letting me know!

[haouech]

Hello,
I'm wondering, we need to pass a checksum as a parameter for the download() method and after the _download() has finished we need to generate and compare hashes?

[whimboo]

So for each platform we have a checksums file located right next to the builds. This file contains checksums for all files related to the platform. So we should check first if such a file is present, download it, and then we can generate the hash locally and compare it with the one from the checksums file.

[haouech]

I would like to work on this issue

[whimboo]

Sure, go ahead and let us know how you proceed and where you need help. Don't hesitate to ask your questions on IRC. You can find me in the #automation channel. Thanks.

[Hyperion101010]

is it still open i would like to work on it

[whimboo]

The active PR for this issue is #454. But we haven't gotten a reply from @haouech for a long time. Maybe you could reply so we can open up this issue again for someone else to work on? Thanks.

[Hyperion101010]

hi there @whimboo for the #454 issue as you said :you were going to distribute the code in pieces
like write it systematically
i would be happy to help
just two things
1)can you give me the names of files responsible for check_sum in the mozilla build ,
also a short intro on what are you planning like a short set-up will be quite helpful
2)and yeah open the issue let's work together (crytography has always been a fantasy)

[sayanmondal2098]

I would like to work on this issue

[shanudjn]

I am new here . I want to work on this bug . Looking for mentorship to get started. Thank you.

[m-agboola]

@whimboo Is this issue still open? If Yes, I'd like to work on it

[whimboo]

It turned out that this particular issue was too complicated to get fixed in the past as mentored issue. As such I will remove the mentored flag.

[kaushiksk]

Hey @whimboo I'd like to take a dig at this. Seemed like @rei2hu got most of the things right from what I can see in his PR. I have one question, are all checksum file names similar to what was mentioned earlier in this thread? i.e. extension of the release file replaced with .checksum or is there a specific format followed for each release? Any resource to help me with this?