Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix PE checksums in Windows release #6194

Closed
The-Compiler opened this issue Feb 22, 2021 · 7 comments
Closed

Fix PE checksums in Windows release #6194

The-Compiler opened this issue Feb 22, 2021 · 7 comments

Comments

@The-Compiler
Copy link
Member

I found out that adding a PE checksum to qutebrowser.exe, the number of false positives for v2.0.2 goes down from 31 to 7. Similarly v2.0.0 goes down to 6 from 29 and v2.0.1 goes down to 8 from 15.

An open source tool to update the checksum: https://gist.github.com/jay/d662cc9615f3e1ffc75e4ae9485da685
The binary: FixPEChecksum.exe.gz

Maybe this should be forwarded to Pyinstaller, as I think it would benefit more programs with the same issue. AFAIK Pyintaller
does not write a checksum to the executable (has value 0).

Originally posted by @bitraid in #6081 (comment)

@The-Compiler
Copy link
Member Author

The-Compiler commented Feb 22, 2021

I wrote a small script which uses the Python pefile module to verify the checksum of everything in the 2.0.2 x64 standlone zip:

import sys
import itertools
import pathlib
import pefile

files = sys.argv[1:]
if not files:
    p = pathlib.Path('.')
    files = itertools.chain(p.rglob('**/*.pyd'), p.rglob('**/*.dll'), p.rglob('**/*.exe'))

for fn in files:
    pe = pefile.PE(fn)
    print(f'{pe.verify_checksum()} {fn}')

the result:

False .venv/lib/python3.9/site-packages/setuptools/cli-32.exe
False .venv/lib/python3.9/site-packages/setuptools/cli-64.exe
False .venv/lib/python3.9/site-packages/setuptools/cli.exe
False .venv/lib/python3.9/site-packages/setuptools/gui-32.exe
False .venv/lib/python3.9/site-packages/setuptools/gui-64.exe
False .venv/lib/python3.9/site-packages/setuptools/gui.exe
False PyQt5/QtCore.pyd
False PyQt5/QtGui.pyd
False PyQt5/QtNetwork.pyd
False PyQt5/QtOpenGL.pyd
False PyQt5/QtPositioning.pyd
False PyQt5/QtPrintSupport.pyd
False PyQt5/QtQml.pyd
False PyQt5/QtQuick.pyd
False PyQt5/QtQuickWidgets.pyd
False PyQt5/QtSql.pyd
False PyQt5/QtWebChannel.pyd
False PyQt5/QtWebEngine.pyd
False PyQt5/QtWebEngineCore.pyd
False PyQt5/QtWebEngineWidgets.pyd
False PyQt5/QtWidgets.pyd
False PyQt5/_QOpenGLFunctions_2_0.pyd
False PyQt5/sip.cp39-win_amd64.pyd
False adblock/adblock.cp39-win_amd64.pyd
False libcrypto-1_1-x64.dll
False libeay32.dll
False libssl-1_1-x64.dll
False markupsafe/_speedups.cp39-win_amd64.pyd
False opengl32sw.dll
False qutebrowser.exe
False ssleay32.dll
False yaml/_yaml.cp39-win_amd64.pyd

True .venv/lib/python3.9/site-packages/pip/_vendor/distlib/t32.exe
True .venv/lib/python3.9/site-packages/pip/_vendor/distlib/t64.exe
True .venv/lib/python3.9/site-packages/pip/_vendor/distlib/w32.exe
True .venv/lib/python3.9/site-packages/pip/_vendor/distlib/w64.exe
True MSVCP140.dll
True MSVCP140_1.dll
True PyQt5/Qt/bin/QtWebEngineProcess.exe
True PyQt5/Qt/plugins/bearer/qgenericbearer.dll
True PyQt5/Qt/plugins/iconengines/qsvgicon.dll
True PyQt5/Qt/plugins/imageformats/qgif.dll
True PyQt5/Qt/plugins/imageformats/qicns.dll
True PyQt5/Qt/plugins/imageformats/qico.dll
True PyQt5/Qt/plugins/imageformats/qjpeg.dll
True PyQt5/Qt/plugins/imageformats/qsvg.dll
True PyQt5/Qt/plugins/imageformats/qtga.dll
True PyQt5/Qt/plugins/imageformats/qtiff.dll
True PyQt5/Qt/plugins/imageformats/qwbmp.dll
True PyQt5/Qt/plugins/imageformats/qwebp.dll
True PyQt5/Qt/plugins/platforms/qminimal.dll
True PyQt5/Qt/plugins/platforms/qoffscreen.dll
True PyQt5/Qt/plugins/platforms/qwebgl.dll
True PyQt5/Qt/plugins/platforms/qwindows.dll
True PyQt5/Qt/plugins/platformthemes/qxdgdesktopportal.dll
True PyQt5/Qt/plugins/position/qtposition_positionpoll.dll
True PyQt5/Qt/plugins/position/qtposition_serialnmea.dll
True PyQt5/Qt/plugins/position/qtposition_winrt.dll
True PyQt5/Qt/plugins/printsupport/windowsprintersupport.dll
True PyQt5/Qt/plugins/sqldrivers/qsqlite.dll
True PyQt5/Qt/plugins/sqldrivers/qsqlodbc.dll
True PyQt5/Qt/plugins/sqldrivers/qsqlpsql.dll
True PyQt5/Qt/plugins/styles/qwindowsvistastyle.dll
True PyQt5/Qt/qml/Qt/labs/animation/labsanimationplugin.dll
True PyQt5/Qt/qml/Qt/labs/calendar/qtlabscalendarplugin.dll
True PyQt5/Qt/qml/Qt/labs/folderlistmodel/qmlfolderlistmodelplugin.dll
True PyQt5/Qt/qml/Qt/labs/location/locationlabsplugin.dll
True PyQt5/Qt/qml/Qt/labs/platform/qtlabsplatformplugin.dll
True PyQt5/Qt/qml/Qt/labs/qmlmodels/labsmodelsplugin.dll
True PyQt5/Qt/qml/Qt/labs/settings/qmlsettingsplugin.dll
True PyQt5/Qt/qml/Qt/labs/sharedimage/sharedimageplugin.dll
True PyQt5/Qt/qml/Qt/labs/wavefrontmesh/qmlwavefrontmeshplugin.dll
True PyQt5/Qt/qml/QtBluetooth/declarative_bluetooth.dll
True PyQt5/Qt/qml/QtGraphicalEffects/private/qtgraphicaleffectsprivate.dll
True PyQt5/Qt/qml/QtGraphicalEffects/qtgraphicaleffectsplugin.dll
True PyQt5/Qt/qml/QtLocation/declarative_location.dll
True PyQt5/Qt/qml/QtMultimedia/declarative_multimedia.dll
True PyQt5/Qt/qml/QtNfc/declarative_nfc.dll
True PyQt5/Qt/qml/QtPositioning/declarative_positioning.dll
True PyQt5/Qt/qml/QtQml/Models.2/modelsplugin.dll
True PyQt5/Qt/qml/QtQml/RemoteObjects/qtqmlremoteobjects.dll
True PyQt5/Qt/qml/QtQml/StateMachine/qtqmlstatemachine.dll
True PyQt5/Qt/qml/QtQml/WorkerScript.2/workerscriptplugin.dll
True PyQt5/Qt/qml/QtQml/qmlplugin.dll
True PyQt5/Qt/qml/QtQuick.2/qtquick2plugin.dll
True PyQt5/Qt/qml/QtQuick/Controls.2/Fusion/qtquickcontrols2fusionstyleplugin.dll
True PyQt5/Qt/qml/QtQuick/Controls.2/Imagine/qtquickcontrols2imaginestyleplugin.dll
True PyQt5/Qt/qml/QtQuick/Controls.2/Material/qtquickcontrols2materialstyleplugin.dll
True PyQt5/Qt/qml/QtQuick/Controls.2/Universal/qtquickcontrols2universalstyleplugin.dll
True PyQt5/Qt/qml/QtQuick/Controls.2/qtquickcontrols2plugin.dll
True PyQt5/Qt/qml/QtQuick/Controls/Styles/Flat/qtquickextrasflatplugin.dll
True PyQt5/Qt/qml/QtQuick/Controls/qtquickcontrolsplugin.dll
True PyQt5/Qt/qml/QtQuick/Dialogs/Private/dialogsprivateplugin.dll
True PyQt5/Qt/qml/QtQuick/Dialogs/dialogplugin.dll
True PyQt5/Qt/qml/QtQuick/Extras/qtquickextrasplugin.dll
True PyQt5/Qt/qml/QtQuick/Layouts/qquicklayoutsplugin.dll
True PyQt5/Qt/qml/QtQuick/LocalStorage/qmllocalstorageplugin.dll
True PyQt5/Qt/qml/QtQuick/Particles.2/particlesplugin.dll
True PyQt5/Qt/qml/QtQuick/PrivateWidgets/widgetsplugin.dll
True PyQt5/Qt/qml/QtQuick/Scene2D/qtquickscene2dplugin.dll
True PyQt5/Qt/qml/QtQuick/Scene3D/qtquickscene3dplugin.dll
True PyQt5/Qt/qml/QtQuick/Shapes/qmlshapesplugin.dll
True PyQt5/Qt/qml/QtQuick/Templates.2/qtquicktemplates2plugin.dll
True PyQt5/Qt/qml/QtQuick/Timeline/qtquicktimelineplugin.dll
True PyQt5/Qt/qml/QtQuick/Window.2/windowplugin.dll
True PyQt5/Qt/qml/QtQuick/XmlListModel/qmlxmllistmodelplugin.dll
True PyQt5/Qt/qml/QtQuick3D/Effects/qtquick3deffectplugin.dll
True PyQt5/Qt/qml/QtQuick3D/Helpers/qtquick3dhelpersplugin.dll
True PyQt5/Qt/qml/QtQuick3D/Materials/qtquick3dmaterialplugin.dll
True PyQt5/Qt/qml/QtQuick3D/qquick3dplugin.dll
True PyQt5/Qt/qml/QtRemoteObjects/qtremoteobjects.dll
True PyQt5/Qt/qml/QtSensors/declarative_sensors.dll
True PyQt5/Qt/qml/QtTest/qmltestplugin.dll
True PyQt5/Qt/qml/QtWebChannel/declarative_webchannel.dll
True PyQt5/Qt/qml/QtWebEngine/qtwebengineplugin.dll
True PyQt5/Qt/qml/QtWebSockets/declarative_qmlwebsockets.dll
True Qt5Bluetooth.dll
True Qt5Core.dll
True Qt5DBus.dll
True Qt5Gui.dll
True Qt5Location.dll
True Qt5Multimedia.dll
True Qt5Network.dll
True Qt5Nfc.dll
True Qt5OpenGL.dll
True Qt5Positioning.dll
True Qt5PositioningQuick.dll
True Qt5PrintSupport.dll
True Qt5Qml.dll
True Qt5QmlModels.dll
True Qt5QmlWorkerScript.dll
True Qt5Quick.dll
True Qt5Quick3D.dll
True Qt5Quick3DAssetImport.dll
True Qt5Quick3DRender.dll
True Qt5Quick3DRuntimeRender.dll
True Qt5Quick3DUtils.dll
True Qt5QuickControls2.dll
True Qt5QuickParticles.dll
True Qt5QuickShapes.dll
True Qt5QuickTemplates2.dll
True Qt5QuickTest.dll
True Qt5QuickWidgets.dll
True Qt5RemoteObjects.dll
True Qt5Sensors.dll
True Qt5SerialPort.dll
True Qt5Sql.dll
True Qt5Svg.dll
True Qt5Test.dll
True Qt5WebChannel.dll
True Qt5WebEngine.dll
True Qt5WebEngineCore.dll
True Qt5WebEngineWidgets.dll
True Qt5WebSockets.dll
True Qt5Widgets.dll
True Qt5XmlPatterns.dll
True VCRUNTIME140.dll
True VCRUNTIME140_1.dll
True _asyncio.pyd
True _bz2.pyd
True _ctypes.pyd
True _decimal.pyd
True _elementtree.pyd
True _hashlib.pyd
True _lzma.pyd
True _multiprocessing.pyd
True _overlapped.pyd
True _queue.pyd
True _socket.pyd
True _ssl.pyd
True _uuid.pyd
True d3dcompiler_47.dll
True libEGL.dll
True libGLESv2.dll
True libcrypto-1_1.dll
True libffi-7.dll
True libssl-1_1.dll
True pyexpat.pyd
True python3.dll
True python39.dll
True select.pyd
True unicodedata.pyd

So it's certainly not only qutebrowser.exe being affected, but maybe the others are already well-known as they are widespread enough... I wonder if it's version= and our gen_version.py triggering this somehow? Although PyInstaller seems to use the Windows API to adjust that information...

I hope we can use pefile to fix the checksum. That'd certainly be easiest, and perhaps also possible to upstream easily into PyInstaller.

@bitraid
Copy link
Contributor

bitraid commented Feb 22, 2021

The API used to update the resources doesn't save/update the checksum (and a quick search at the code, doesn't seem to do so at a later time). According to Microsoft, the checksum validation is performed for "all drivers, any DLL loaded at boot time, and any DLL that is loaded into a critical Windows process", so maybe it wasn't deemed necessary by Pyinstaller developers.
Fixing the checksum is fairly easy: It is a dword value stored at offset 0x58 of the PE header. The offset of the PE Header is a 32-bit pointer at offset 0x3c of MS-DOS header (beginning of file).
For reference, this is how I do it in powershell: https://gist.github.com/bitraid/fe8fa15fc1c30b1ca9cc380e5bcebde6
(This is actually a cmd/powershell hybrid script. In line 52 it gets the PE header offset, and in lines 66,67 writes the new checksum).

@The-Compiler
Copy link
Member Author

The-Compiler commented Feb 22, 2021

I think all we need with pefile is actually this:

import pefile

pe = pefile.PE('qutebrowser.exe')
pe.OPTIONAL_HEADER.CheckSum = pe.generate_checksum()
pe.close()
pe.write('qutebrowser.exe')

Running that over the 2.0.2 release (64bit standalone) changes the checksum at file offset 0x160 from "00 00 00 00" to "5B AD 44 00".

It results in this file, which pefile deems correct:
qutebrowser.zip

I haven't gotten around to cross-checking it with another tool or checking it at VirusTotal. If you want to give it a look, that'd be much appreciated!

@bitraid
Copy link
Contributor

bitraid commented Feb 22, 2021

Yes, the checksum is correct!

@bitraid
Copy link
Contributor

bitraid commented Feb 22, 2021

And VirusTotal gives 6 detections on updated check (same hash as my previous test).

@The-Compiler
Copy link
Member Author

I've tried faking a "3.0.0 release" with a new version number and everything, based on the current master.

Without the checksum patching, we get 8 false-positives. Much better than expected... 😅 But it includes Microsoft Defender.

After the patching, that's down to 6, most notably not including Microsoft Defender anymore.

I suppose this fixes #6081 then, at least to the extent that we can do so. I've also reported this to PyInstaller: pyinstaller/pyinstaller#5579

Thanks for getting the ball rolling, I didn't even know that this existed!

The-Compiler added a commit that referenced this issue Feb 22, 2021
This should help with virus scanner false positives.

See pyinstaller/pyinstaller#5579
Fixes #6081
Fixes #6194

(cherry picked from commit 2b91081)
@The-Compiler
Copy link
Member Author

I've now regenerated and pushed v2.0.2.post1 releases for Windows with the fixed checksum. Now Microsoft does detect it as malware again...

Argh. This is all so random. 😢

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants