Turris OS building scripts

Turris OS is based on OpenWrt. This repository contains script and additional patches to convert OpenWrt tree to Turris OS one.

OpenWrt consists of core repository and additional packages feeds defined in feeds.conf.

Repository layout

compile_pkgs

script to prepare OpenWrt based source tree and to compile Turris OS in it. The final result are Turris OS packages.

generate_lists

this script generates lists for Turris updater (updater-ng). Those are exposed alongside of Turris OS packages and control updater’s behavior.

generate_medkit

script creating Turris medkit (factory image) from Turris OS packages and Turris updater’s lists.

configs/

this directory contains configurations inserted to generated OpenWrt configuration. There are subdirectories for every Turris router and directory common that applies on all of them.

feeds.conf

this file defines which feeds and in what version are about to be used for Turris OS packages compilation (OpenWrt based build). There is also non-standard commented line for OpenWrt used by compile_pkgs script to get OpenWrt source tree.

patches

this directory contains all patches applied on both base OpenWrt tree as well as on any feed. It contains additional directories named after feed. Given patch is applied on a feed of the same name as is name of directory it is in relative to patches/. There can optionally be additional directories named after the feeds to sort patches to groups. Please read readme file in patches directory for more in depth explanation.

NEWS

this is Turris OS changelog. It is parsed and used in git tags or in package informing users about changes.

helpers/

directory containing libraries and utility scripts for top level scripts and some less important scripts as well.

Requirements

You need to have a GNU/Linux distribution to be able to compile Turris OS and packages.

To install all dependencies for Debian distribution, you can use:

apt install ca-certificates git build-essential zlib1g-dev gawk libssl-dev \
 subversion unzip libncurses-dev wget python3 file rsync gcc-multilib \
 g++-multilib libc6-dev-i386 protobuf-c-compiler which libyaml-dev

For other distributions, we suggest to look at OpenWrt documentation.

Getting started

Create an empty directory you want to build Turris OS in, enter that directory and from it call compile_pkgs script. In most basic use-case, this should be enough.

Warning

It will delete all previous content of the current directory!

You can repeat individual steps of the compilation separately, check with the following command compile_pkgs --help for a list of available commands, generic options, and their brief descriptions.

Before you started you need to choose what version of Turris OS you want to build. There are development versions in branches named hbk, hbl and hbd (see workflow). Then there are tagged final releases. Those are in format of for example v4.0.0 for Turris OS 4.0.0 or v4.1.2 for Turris OS 4.1.2.

To get to appropriate version you have to run command like this: git checkout hbk where in this case this takes you to development branch for Turris OS fixup releases.

It is advised to build the latest release version as with that you are most likely to get to the end. To get the latest version you can do: git checkout "$(git tag | sort -V | tail -1)".

Advanced usage

OpenWrt build system preparation

Sometimes you want to just get OpenWrt build system. That is possible with the following command:

./compile_pkgs prepare -t omnia

Where -t means -target, so the only valid values are turris1x, omnia, mox.

Warning

Be aware it removes previous content of current directory!

This gives you fully patched OpenWrt build system. The "patched" here is important. You should not use scripts/feeds script for fetching feeds as that can remove local patches. Instead to refetch feeds you can use ./compile_pkgs repatch_feeds.

Building packages

To build a package you have to first prepare build directory. That can be achieved by running in target directory.

./compile_pkgs prepare_tools -t omnia

Where -t means -target, so the only valid values are turris1x, omnia, mox.

Warning

Be aware it removes previous content of current directory!

Once, you have compiled required tools, you can build packages using this command:

make package/name/compile

The name is package name, which you want to compile.

You can also clean single package just by replacing compile with clean.

The resulting package is placed in directory bin/packages/ARCH/REPO where the ARCH is target architecture specific string and the REPO is the name of package’s source repository.

Occasionally, you can meet cases, where you need to run make package/name/compile with parameters -j1 V=s or -j1 V=sc to be able to see, what went wrong.

Generate lists

The lists are rules for Turris packages manager called Updater-ng. They are processed using M4 macro language. To generate the lists you have to run:

./generate_lists generated_lists

The output is placed to directory generated_lists (feel free to use some other path).

Warning

Be aware it removes previous content of current directory!

Generate medkits

Generating medkits is using Turris package manager Updater-ng and using for signing.

Medkits are always generated against official Turris repositories. There is no easy way to generate medkits against locally compiled packages and generated lists. This also means that you don’t have to generate lists nor compile packages to generate medkits.

Requirements to be able to compile Updater-ng and usign.

apt install cmake liblua5.1-0-dev libevent-dev libarchive-dev \
 libcurl4-openssl-dev liburiparser-dev uthash-dev libyaml-dev

If you installed all the requirements for Updater-ng, you can now use following command:

 generate_medkit -t omnia

Where -t means --target, so the only valid values are turris1x, omnia, mox.

Development and maintenance

Configuration

Default config for OpenWrt is put in configs directory either in directory common or in a directory named after the target board. Files in those directories get merged (common ones first) and will form the final .config file. Some parts of compile_pkgs might modify it further.

Release notes

There is a NEWS file in this directory that contains the version number and after it lines with new features in the release. This file is used to generate turris-version package. compile_pkgs script reads it, finds the newest version and from notes underneath it creates the package with release notes.

Releasing new version

When new version is released to Snails (see workflow for explanation) then new commit with hashes has to be created and pushed. This commit have to be tagged with appropriate version tag in format vVERSION where VERSION is released version. To make this all simple and to correctly set always all configurations we have script helpers/new_release.sh. Run this script after release from turris-build project root directory and it is going to automatically detect, commit and tag new version. You should review commit and tag it created for you and then push it using git push --tags.

Script helpers/new_release.sh can generate various errors and warnings. It is advised to run it even before release in verify mode to review possible problems with release.

Forking new release

According to workflow new releases are forked from parent branch. During this process care should be taken to tweak defaults to appropriate values. Following list should be taken as a checklist for new release branch.

1. Set PUBLISH_BRANCH in defaults.sh. Master branch should always be set to hbd and release branches should be set to hbs.

2. Set branches in feeds.conf. You should append string like this: ;openwrt-18.06. This has to be done for all OpenWRT feeds as well for OpenWRT it self. URL used by compile_pkgs is specified as first line in feeds.conf and with exception of first column it has same format as feeds. Note that that line is intentionally commented out as that is not feed.

Patching

If you need to patch OpenWRT or any of the feeds, put a git formatted patch into the respective subdirectory of patches directory. In patches directory, there is openwrt directory and directory for each feed we are patching. Inside those directories, patches are also divided into subdirectories like to-upstream, wip or hack to somehow distinguish between the quality of patches.

Rebasing patches

Sometimes patches no longer apply cleanly and need to be rebased. The most simple way to do that, given that we have everything in git, is the following:

find . -name '*.rej' -delete                # Clean all existing reject files
git am --reject .../patches/.../xyz.patch   # Try hard to apply patch
find . -name '*.rej'                        # List all failed chunks

After those commands, we’ll get our git tree in partially merged state and with few .rej files containing failed chunks. Actually, compile_pkgs script by default cleans up checkout directory and applies patches with git am --reject so if the build fails, checkout ends up in the state similar to what can be achieved by those commands. Files with .rej extension needs to be merged manually and afterward, we need to call git add on newly patched files. Once all conflicts are resolved, git am --continue will create real commit that we were trying to add using git am. Now all that is left is to export it using git format-patch -1 and overwrite patch stored in it of build repository.