Numerical software on Windows
This is a survey of the build dependencies of numerical software on Windows.
Fast numerics need optimized libraries for BLAS and LAPACK. They also need a Fortran compiler to make use of standard Fortran numerical libraries.
Most numerical programming development happens in Unix, and most open-source
software is cross-platform. Building many numerical libraries needs a
Unix-like build system such as make and autoconf.
A selected list:
- numpy and scipy numerical libraries for Python
- R language for statistical computing
- GNU Octave numerical computing language
- Sage mathematics software system
- Scilab software for numerical computation
- Julia dynamic programming language for technical computing
All of these need to compile Fortran code and use BLAS / LAPACK libraries - see below.
See Visual Studio page on Wikipedia. This is the platform compiler for Windows. Express edition is free to download. Compiles C and C++, but Microsoft does not make a Fortran compiler.
See MSVC and Python for a list of MS Visual C versions and download links.
Free x86 and AMD64 (x86-64) VC90 c-compilers for Python-2.7 are now available from Microsoft.
Cygwin is a Unix-like environment on Windows. It includes the full GNU compiler collection and the Unix build tools. gcc compilers link against a C runtime in a Cygwin-specific DLL providing a POSIX compatibility layer. This can cause problems when different libraries compile against different versions of the DLL. It is easy to run into problems with multiple copies of the DLL
MinGW is a port of of the GNU compiler collection to Windows that links against the Microsoft C runtime libraries. It is 32-bit only.
See MingGW-w64 sourceforge page and section in MinGW wikipedia article
Another port of of the GNU compiler collection to Windows implementing multilib (32- and 64- bit compilation from the same compiler).
Much younger than the MinGW project, and under rapid development.
See Intel C compilers. Seem to be somewhere upward of $1000 for Windows C compiler.
See the Intel developer license page.
The license that appears to apply to the Intel run-times, linked statically or dynamically, is the Intel EULA for software development products. See this discussion for more detail.
Relevant clauses for us are:
- Distribution of the Redistributables is also subject to the following limitations: [...] (d) will provide the Redistributables subject to a license agreement that prohibits disassembly and reverse engineering of the Redistributables except in cases when you provide Your Product subject to an open source license that is not an Excluded License, for example, the BSD license, or the MIT license, (e) will indemnify, hold harmless, and defend Intel and its suppliers from and against any claims or lawsuits, including attorney's fees, that arise or result from Your modifications, derivative works or Your distribution of Your Product.
"Redistributables" does cover run-time DLLs, and probably also applies to statically linked object code - see discussion.
With this license, distributing our binaries under a BSD license, we don't have to add the disassembly clause to the license agreement, but we do have to pay Intel's legal fees if someone sues Intel because of a problem in Numpy / Scipy.
Also see below for discussion of Intel Fortran compiler.
See: mingwpy discussion of BLAS / LAPACK.
The options for a 64-bit Fortran compiler on Windows are:
- Cygwin gfortran
- MinGW-w64 gfortran
- Intel Fortran. "From $849". From this forum discussion, even libraries that are statically linked to the Intel Fortran runtime are covered by the same end-user license, and specifically, the developer distributing the binaries is liable for Intel's legal fees if Intel gets sued as a result of using the binary (see above).
- Portland Group Visual Fortran. Free license offer for US and German academics until June 30th 2014. The license for run-time files forces us to add a clause to our license forbidding disassembly of their run-times, and makes us liable for their legal fees if they are sued, making it very unattractive for open source development. Standard cost appears to be $140 for a single seat Academic license.
- Absoft (from $289 academic, $697 commercial).
- gfortran from Fortran tools for Windows
- NAG Fortran builder
- Lahey Fortran - from $175 for an educational license for Lahey / GNU Fortran.
- Simply Fortran - GNU Fortran with an IDE and package manager. It is free to download, but will disable some of the features of the IDE after 30 days unless you register. Registration costs $99. I (MB) emailed the support email on April 7 2016 to ask if SimplyFortran was able to link with the MSVC 2015 runtime (the MSVC version used to build Python 3.5), but have not yet had a reply.
- (At some point) PGI LLVM Fortran compiler. Announced in November 2015, this "multi-year" project is expected to appear as a source code release by "late 2016".
If all Fortran code is compatible with Fortran 77, it is also possible to compile the Fortran code with a C compiler by converting the Fortran code to C with f2c.
Various problems can arise - see OpenBLAS with VS and MSVC and MinGW DLLs for examples.
Applications and DLLs have to make sure that they do not use different MSVC runtime libraries in the same process.
MinGW-w64 lacks intrinsic support for choosing MSVC run-times as a gcc option. For example, Python 3.3 was compiled with MSVC 10 (VS 2010). A Python extension will be in the same process as Python, and so will also need to link against the MSVC 10 run-times.
The workaround is to create and patch the gcc specs file for the particular MSVC runtime you want to link against.
- Cygwin contains a full range of Unix build tools.
- msys is "a collection of GNU utilities such as bash, make, gawk and grep to allow building of applications and programs which depend on traditionally UNIX tools to be present. It is intended to supplement MinGW and the deficiencies of the cmd shell." MSYS can also be used with MinGW-w64. It contains make and autoconf and a minimal bash shell.
- msys2 is "an updated, modern version of MSYS, both of which are Cygwin (POSIX compatibility layer) forks with the aim of better interoperability with native Windows software." It can install MinGW-w64 from its own package manager.
Another option is using cross-compilers like MXE. MXE does MinGW builds from Linux.
If a program tries to load a DLL called mylibrary.dll, Windows has to
decide which directories it should search for mylibrary.dll.
Applications and libraries often need to ship DLLs, and they need to make sure the application / libraries finds the correct version.
One place that Windows looks for DLLs is on the system $PATH (the %PATH%
variable in cmd.exe, or $env:PATH in Powershell). It is not safe in
general to put our own mylibrary.dll into some system directory like
WINDOWS/system because another application may depend on a different
version of the DLL that is also called mylibrary.dll.
For reference here is the standard DLL search algorithm. See this exploration of DLL search settings for more explanation.
Alternative strategies for finding the right DLL are:
-
Putting the required DLL in the same directory as the calling executable, assuming the standard search order is in place.
-
Using the
LOAD_WITH_ALTERED_SEARCH_PATHflag to the DLL loading callLoadLibraryEx. In this case you get the alternate DLL search algorithm, which looks in the directory containing the loading DLL instead of the directory containing the calling executable. -
Setting an extra DLL search directory with SetDllDirectory. The Microsoft documentation on the LoadLibraryEx call says this about
SetDllDirectory:However, be aware that using SetDllDirectory effectively disables safe DLL search mode while the specified directory is in the search path and it is not thread safe. If possible, it is best to use AddDllDirectory to modify a default process search path.
See the standard DLL search algorithm for more detail on safe DLL search mode.
-
Adding a specific directory to the search path with AddDllDirectory (dead link). This only works for updated Windows 7 and later.
-
Using private assemblies to specify a set of DLLs specific to the application.
See Building numpy / scipy (dead link).
Standard Windows compile uses:
- MinGW or Cygwin for build tools
- ATLAS for BLAS / LAPACK
- MinGW or Cygwin for Fortran
Standard numpy and scipy binary releases on Windows use pre-compiled ATLAS libraries and are 32-bit only because of the difficulty of compiling ATLAS on 64-bit Windows. Because of the problem of locating DLLs, ATLAS is linked statically.
See building R from source and R Windows toolset.
In the Windows toolset page, a typical quote:
We have found that the build process for R is quite sensitive to the choice of tools: please follow our instructions exactly, even to the choice of particular versions of the tools
The R build depends on a customized set of Windows tools for R. These include copies of Cygwin command line tools and a copy of the MinGW-w64 toolchain
So, R uses:
- MinGW / Cygwin copies for build tools
- Copies of reference BLAS
- MinGW for Fortran
See Octave for Windows and Octave-forge Windows directory.
Octave Windows binary releases on the main sourceforge site are behind compared with Mac (3.6.4 vs 3.8.0). The top of the Octave Windows page says:
GNU Octave is primarily developed on GNU/Linux and other POSIX conformal systems. The ports of GNU Octave to Windows use different approaches to get most of the original Octave and adapt it to Microsoft Windows idiosyncrasies (eg: dynamic libraries, file paths, permissions, environment variables, GUI system, etc). Bear this in mind and don't panic if you get unexpected results.
There is a current Octave build for windows (dead link) using the MXE cross compiler.
Tools, BLAS and Fortran use are a little confusing across the range of Octave builds:
- There appear to be Octave builds with MinGW, Cygwin and MSVC. The current MinGW builds look like they are 32-bit. Approach seems to be to specify a custom MSYS / MinGW environment via install.cmd scripts to get dependencies. MSVC builds apparently need MSYS - see this email thread.
- BLAS is both of (choice of) OpenBLAS and ATLAS
- MinGW builds use gfortran. It's not clear to me (MB) how the MSVC builds deal with Fortran. They may use MinGW gfortran - see links in this thread
See installing sage from source
Sage can only be built on Cygwin on Windows. Uses:
- Cygwin for build chain
- Blas from ATLAS
- Fortran from gfortran
See compiling scilab for Windows
The instructions include this command to checkout requirements for Windows:
svn export --force --username anonymous --password Scilab svn://svn.scilab.org/scilab/trunk/Dev-Tools/SE/Prerequirements/Windows scilab
This fetches a large number of Windows binary packages that appear to include compiled reference BLAS, ATLAS, OpenBLAS, MKL, Python, java.
So:
- MSVC for build chain
- One of reference BLAS, ATLAS, OpenBLAS, MKL
- Intel Fortran XE 2011 or f2c for Fortran sources.
Julia uses MinGW-builds - binary builds of dual 32- / 64-bit MinGW compilers.
- MinGW-builds / MSYS2 or MinGW-builds / MSYS for Unix build tools
- OpenBLAS compiled as part of the build process
- Fortran by gfortran