Skip to content
This repository has been archived by the owner on Jul 12, 2022. It is now read-only.
/ jrcstu-forge Public archive

Vagrant & docker images for developing JRC-STU's (mostly) python projects

License

Notifications You must be signed in to change notification settings

JRCSTU/jrcstu-forge

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Reproducible Cross-platform python programs

Requirements & Deliverables

The fundamental requirements for reproducible cross-platform python programs

Ensure a reproducible cross-platform development and deployment infrastructure-as-code, with Vagrant images & conda packages, as...

Vagrant VM for a cross-platform dev-env

It's like a replacement for co2mpas-ALLINONE... but for Devs only!

  • Practically it is a proof-of-concept that you can develop with Vagrant.
  • Confusing when switching in-and-out of VM with [Win-key] - better work completely in the VM.

Conda

Conda points for cautious

Conda is an extensive & flexible ecosystem, that definitely fits our corss-platform purpose, as can be seen in [this iconic presentation)[http://chdoig.github.io/pydata2015-dallas-conda/]

The ecosystem's built packages (the "recipes") are maintained by a cohord of open-source developers around the globe, and they are not always coherent. In addition, there are still some open edges in the conda specification itself. Documentation is not always present, but there is much exmaple code in the cloud.

Building the cross-platform executable, happens in 3 stages:

  • select the conda packages and build any missing ones (that means, we must stabiizeour in-house packages before starting).
  • Build an evironment andcontaining all needed packages and test that it works.
  • Run the constructor tool and build the installation-archive.

Here are some things things to watch out for, during these 3 stages:

Building python packages:

  • There is a mismatch between setup.py managed packages in PyPi & conda packages produced with meta.yaml recipes:

Specifically the extras of setuptools correspond to of 2-3 different conda features:

Not all recipe authors know and use these features. Consequently, some packages have too many dependencies or missing some required ones.

  • Another missmatch is that some conda packages have different names from PyPi ones (eg. Qt packages). This is certainly so for metapackages, since they do not exist in the pip world.

Installing conda packages:

Issues here relate to the shortcomings of the conda cli and the way it handles environment.yml files (a file maintainted by each conda-env).

  • A rather important issue regarding reproducibility is that packages uploaded in the conda-cloud do not(!) have a stable hash. Their hashes may change if the package is overwwritten (relevant only for user-owned channels, not conda-forge), or when their metadata change in order to fix issues detected at a later time (eg. to set upper boundary on an incompatible dependency).

    Although we do not expect to bump to such issues, it's good if they are addressed in conda-5.

  • Porting the specs of an existing conda-environment into different platform.

Assembling the executable with constructor

  • The major issue is that it does not support noarch packages. These packages were introduced trelatively recently, and constructor is only now started to catch up.

  • While a constructor on linux & osx can be build executable for both these 2 platforms, a real Windows is required to build the *.exe file.


Details

Ragarding stuff for the developers, read the REAMEs in the respective folders.

File contents

+--Vagrant/Win10/      Contains the `Vagrantfile`+scrips for the CO2MPAS development,
|                       configured with:
|                           + miniconda3, git, notepad++, totalcommander, cmder, nodejs, docker
|                           + chromium, firefox, edge (+selenium-drivers)
|                           + IDEs: vscode, pycharm
+--conda/
        +--recipes/     folders containing `meta.yaml` files etc that build co2mpas
        |               (or its dependent libs) as pure-conda packages,
        |               needed for *constructor* installer, currently uploaded to:
                        https://anaconda.org/ankostis 
        +--constructs/
            +--CO2MPAS  The specs for building the executable.