TLS and HAProxy for development

Introduction

HAProxy is an excellent reverse proxy server as well as load balancer. The use of HAProxy when deploying TG applications is highly recommended as it decouples several important concerns away from the application itself:

• TLS termination proxy -- handles TLS infrastructure, including integration with Let's Encrypt.
• HTTP/2 front-end -- provides a simple way to take advantage of HTTP/2 multiplexing.
• Load balancing -- can be used for application load balancing.

Running TG applications behind HAProxy in production makes it also extremely desirable, if not required, to do the same in development. This article discusses how HAProxy and self-signed certificates can be established for local development purposes.

Some assumptions/prerequisites:

• macOS or Ubuntu is the OS of choice.
• Docker is installed.
• Chrome is the browser of choice.

Please note that much of the covered material should be transferrable to Windows.

The following steps are required:

1. Generate public/private keys and a self-signed certificate to be used by HAProxy to establish TLS for HTTP(S) and HTTP/2.
2. Install and configure HAProxy.
3. Register the certificate as trusted with the operating system.

Generate public/private keys and a self-signed certificate

This step involves the use of openssl, which needs to be installed if it isn't to proceed with this step. For Windows OS openssl can be downloaded from here, there is also available version for Win64. It is better to choose full version (not lightweight) for software developers.

In the past Chrome was looking at "commonName" of the certificate to match the domain name and the certificate in use. Since some version (58?) this has changed to look at the certificate's extension "subjectAltName", but the fact of that change is not properly disseminated. Therefore, certificate generation needs to take this into account.

Here is a command that can be used as a template to generate a certificate that will be usable with Chrome:

openssl req \
    -x509 -sha256 \
    -newkey rsa:4096 \
    -days 1024 \
    -nodes \
    -subj "/C=AU/ST=VIC/O=Fielden/CN=localhost" \
    -extensions SAN \
    -reqexts SAN \
    -config <(cat /etc/ssl/openssl.cnf \
            <(printf "\n[SAN]\nsubjectAltName=DNS:localhost,DNS:tgdev.com")) \
    -keyout "localhost.key" \
    -out "localhost.pem"

The three critical options to generate subjectAltName are:

• -extensions SAN -- declares extension SAN (arbitrary name, which stands for Subject Alt Name).
• -reqexts SAN -- declares that extension SAN is used for req extensions.
• -config <(cat /etc/ssl/openssl.cnf \ <(printf "\n[SAN]\nsubjectAltName=DNS:localhost,DNS:tgdev.com")) \ -- appends section [SAN] with entry subjectAltName=DNS:localhost,DNS:tgdev.com to the content of the default openssl.cnf configuration file.

Having both values DNS:localhostand DNS:tgdev.com for subjectAltName ensures that accessing either https://localhost or https://tgdev.com should correctly identify domain names by Chrome. Naturally, value DNS:tgdev.com can be changed to any other appropriate domain name that you may use for local development. However, remember to use that consistently throughout in all places where tgdev.com is referenced in this article.

For Windows PC certificate can be generated with following instruction:

openssl req ^
    -x509 -sha256 ^
    -newkey rsa:4096 ^
    -days 1024 ^
    -nodes ^
    -subj "/C=AU/ST=VIC/O=Fielden/CN=localhost" ^
    -addext "subjectAltName = DNS:localhost,DNS:tgdev.com" ^
    -keyout "localhost.key" ^
    -out "localhost.pem" 

This instruction also generates certificate with additional subjectAltName extension set to localhost and tgdev.com

NOTE: Need to test Android / iOs devices on local network? The easiest way to do this is to use your local static IP address for certificate generation by adding IP:192.168.1.40 to the script (this results to printf "\n[SAN]\nsubjectAltName=DNS:localhost,DNS:tgdev.com,IP:192.168.1.40"). After that use that IP address consistently throughout in all places where tgdev.com is referenced in this article. Also, the following command can be very usefull when you go to other network infrastructure, perhaps even with other domain: sudo ip address add 192.168.1.40/24 dev wlp1s0 -- this creates IP address alias on wifi interface wlp1s0 (linux and macOs). Change wlp1s0 to whatever interface you are using there.

There should be two files generated as the result of running the above command -- localhost.key and localhost.pem. It is a good idea to verify that certificate contains the subjectAltName. This can be done by running the following command:

openssl x509 -in ./localhost.pem -text -noout

The output should look like the screen capture below:

The two generated files need to be concatenated into file haproxy.pem, which is going to be used by HAProxy:

cat ./localhost.pem localhost.key > haproxy.pem

And for Windows PC:

type localhost.pem localhost.key >> haproxy.pem

All of these steps have been organised as a single script for UNIX-based operating systems and this for Windows machines.

Installing docker for windows

There are two is one available option to install docker for windows:

~ Install Docker Toolbox. This is the only option for older windows versions. Installing docker toolbox will require slightly different etc/host configuration that is discussed in Adjusting haproxy.cfg. Please follow this instruction to install it.~ Docker Toolbox has been deprecated and is no longer in active development. Please use Docker Desktop instead.

• Install Docker Desktop. This is an option for Windows 10 64-bit: Pro, Enterprise, or Education (Build 16299 or later). This option might have some issues running applications on 80 port, this will be discussed in Adjusting start_haproxy.sh or start_haproxy.bat section. Please follow this instructions to install Docker Desktop for windows

Installing and configuring HAProxy

HAProxy version 1.9.8 is assumed. Installing HAProxy with Docker is a breeze by running:

docker pull haproxy:1.9.8

For convenience, it is best to create a separate directory to contain HAProxy configuration files and a startup script to start/restart it. Let's say this directory is /Users/username/haproxy (macOS) or /home/username/haproxy (Ubuntu) or C:\Users\username\haproxy (Windows). This folder should contain three files:

1. haproxy.pem, which was generated in the previous step.
2. haproxy.cfg, which a configuration file for HAProxy and can be downloaded from here; requires some changes for local use.
3. start_haproxy.sh, which is a script to start/restart HAProxy and can be downloaded from here; requires some changes for local use.
4. start_haproxy.bat, which is a script to start/restart HAProxy on Windows PC and can be downloaded from here; requires some changes for local use.

Adjusting haproxy.cfg

As mentioned earlier, it is assumed that domain tgdev.com is used for running TG applications locally. This means that file application.propeties for TG applications has entry web.domain=tgdev.com and file /etc/hosts contains a mapping between this domain name and the local IP address (e.g. 192.168.1.43 tgdev.com). In case of Windows PC and docker toolbox /etc/hosts entry for tgdev.com should look like this: 192.168.99.100 tgdev.com, where 192.168.99.100 is a default ip for Linux VM on which docker toolbox runs docker. Let's also assume that file application.propeties has ports specified as 8091:

port.listen=8091
port=8091

The domain name is used in the referenced file haproxy.cfg on line 75 -- acl is_tgdev hdr_beg(host) tgdev.com, which can remain as is if tgdev.com is used.

The port as well as the local IP address need to be specified on line 104 of haproxy.cfg -- server eclipse1 local-ip-address:port check. More specifically, part local-ip-address:port needs to be changed to reflect the local IP address and the port designated for a TG app. For example, server eclipse1 192.168.1.43:8091 check.

This is pretty much all that needs to be adjusted in haproxy.cfg.

Adjusting start_haproxy.sh or start_haproxy.bat

The startup script contains the command to run HAProxy, which needs to include a mapping between the directory where HAProxy configuration lives locally and inside the running Docker container. Here is an excerpt from the referenced shell script:

docker run -d \
           -p 80:80 -p 443:443 -p 9000:9000 \
           --restart=always \
           --name haproxy \
           -v <local haproxy config directory>:/usr/local/etc/haproxy:ro \
           haproxy:1.9.8

And for windows:

docker run -d ^
           -p 80:80 -p 443:443 -p 9000:9000^
           --restart=always^
           --name haproxy^
           -v <local haproxy config directory>:/usr/local/etc/haproxy:ro^
           haproxy:1.9.8

Line -v <local haproxy config directory>:/usr/local/etc/haproxy:ro \ is of interest. Its part <local haproxy config directory> needs to be changed to the path where all three of the files mentioned above are located. Let's say this is directory /home/username/haproxy, and so start_haproxy.sh should be changed to reflect this:

docker run -d \
           -p 80:80 -p 443:443 -p 9000:9000 \
           --restart=always \
           --name haproxy \
           -v /home/username/haproxy:/usr/local/etc/haproxy:ro \
           haproxy:1.9.8

Docker toolbox or Docker desktop will run haproxy on separate VM in that case the question might rise: "What value should be for<local haproxy config directory>. Let's assume that the directory for haproxy config file was created in C:\Users\username\haproxy\ in that case <local haproxy config directory> will be /c/Users/username/haproxy, so the previous script for windows should look like this:

docker run -d ^
           -p 80:80 -p 443:443 -p 9000:9000^
           --restart=always^
           --name haproxy^
           -v /c/Users/username/haproxy:/usr/local/etc/haproxy:ro^
           haproxy:1.9.8

Also separate start_haproxy.bat file is provided here

Please note also the use of option --restart=always. It means that HAProxy will be started automatically upon crashes and Docker or computer restarts. Remove this option if it is preferred to start/stop HAProxy manually. For more details refer Docker documentation.

And as the last step, make the script executable by running chmod +x start_haproxy.sh.

Note on running haproxy with Docker Desktop

When running haproxy via Docker Desktop the error might happen. The error is in the screenshot in the red rectangle.

There are two possible solutions:

• As it can be seen from screenshot the problem is port 80 which for some reasons can not be accessed. In that case this port can be changed and haproxy run script should look like this:

docker run -d ^
           -p 8080:80 -p 443:443 -p 9000:9000^
           --restart=always^
           --name haproxy^
           -v /c/Users/username/haproxy:/usr/local/etc/haproxy:ro^
           haproxy:1.9.8

• The another solution requires to find the application that listens port 80 which makes it inaccessible for Docker. It might be another web server or IIS etc. netstat -aon |find ":80" will help to find PID of application that does it. It could be a case when PID of application is 4 which is the SYSTEM, then you may turn it off this post describes how to do that. But it might be a bit dangerous as it requires to edit register.

When first time running haproxy you should share it with Docker Desktop. After that you can stop, start or restart it using GUI like on the picture below

.

--restart=always option will start haproxy when docker starts. Docker Desktop will be added to autostart by default. For Docker Toolbox it is required to provide additional configurations to make it start on PC startup. The next batch commands should be added to autostart folder:

docker-machine start default
docker run -d ^
            -p 80:80 -p 443:443 -p 9000:9000^
            --restart=always^
            --name haproxy^
            -v /c/Users/username/haproxy:/usr/local/etc/haproxy:ro^
            haproxy:1.9.8

The batch file is here

Register the certificate as trusted with the operating system

Now everything should be ready for us to start a TG app behind HAProxy. And this is required so that we could obtain the certificate from Chrome to register it as trusted with OS.

Ordinarily the order in which TG app and HAProxy are started hardly matters. However, for the first time it highly recommended to first start a TG app and then, only after it is fully loaded, start HAProxy by running ./start_haproxy.sh.

Please note that TG app must be started in HTTP mode, not HTTPS. Make sure that the following lines appear in the console (e.g. Eclipse console) before starting HAProxy:

Starting the Jetty [HTTP/1.1] server on port 8091
Starting fielden.webapp.WebUiResources application

If HAProxy starts with an alert about tgdev having no server available, as depicted in the screen capture below, then either the TG app has not started or HAProxy binding on line 104 was not correctly updated. In that case please re-read section "Adjusting haproxy.cfg" above and make sure it is followed properly.

Assuming that HAProxy started without the above alert, open Chrome and load https://tgdev.com/login.

Regardless of the OS you're using, the result should look like the screen capture below. Open the Developer Tools and switch to the Security tab.

The steps to make our certificate trusted are different for macOS and Ubuntu. Let's start with macOS.

Making certificate trusted in macOS

Click "View certificate" button as indicated with label 1 in the screen capture below -- a certificate dialog is opened.

Drag the certificate icon from the dialog to some directory in Finder. This should create file localhost.cer on that folder.

Double click that file to open it in the Keychain Access application (this will prompt for a system password). The following screen capture shows the result of this after selecting category "Certificates" in this application to see only certificates. As you can see entry "localhost" is present.

Double click entry "localhost" in the Keychain Access window, and mark it as "Always Trusted" under "Trust", option "When using this certificate".

Closing this dialog will prompt for a system password to apply changes. And once applied, entry "localhost" should have a little "+" sign at the start as depicted in the screen capture below.

Now close the Keychain Access app, delete file localhost.cer as no longer needed and refresh the page in Chrome. The page should load successfully without any privacy exceptions as per the screen capture below.

Please note that you might need to restart Chrome for it to load updated certificate policies, but it was not necessary in my case.

Making certificate trusted in Ubuntu

The situation with Ubuntu is slightly more complicated, but does not requires as many screen captures (:. First, export the certificate from the certificate dialog, which appears after clicking button "View certificate" (the same as under macOS). The "Export" button is located in tab "Details".

Make sure your select option "single certificate" during the export as indicated in the screen capture below. Take a note of where the file is exported and the file name -- localhost.crt. This is needed for the steps that follow.

Start a terminal and change the directory to the one where localhost.crt has been exported. Then execute the following commands:

1. sudo apt-get install libnss3-tools — install utility certutil, which is needed to manage keys and certificates (you only need to install this once, the first time you want to import a certificate).
2. certutil -d sql:$HOME/.pki/nssdb -A -t "C,," -n localhost.crt -i localhost.crt — import the certificate into the local database.
3. certutil -d sql:$HOME/.pki/nssdb -L — this is just to list what the resultant DB contains to make sure our certificate is present.
4. go to chrome://settings/certificates, find org-Fielden in Authorities tab, open, edit UNTRUSTED localhost to make it trusted

This is it -- refresh the page in Chrome (may need to restart it) and the privacy exception should be no more.

For Firefox users running Linux: Firefox does not have a 'central' location where it looks for certificates. It just looks into the current profile (reference).

2. certutil -d $HOME/.mozilla/firefox/<YOUR_PROFILE_FOLDER>/ -A -t "C,," -n localhost.crt -i localhost.crt - import the certificate into the profile DB.
3. certutil -d $HOME/.mozilla/firefox/<YOUR_PROFILE_FOLDER>/ -L — this is just to list the profile DB.
4. go to about:preferences, find Certificates section (in Security) and open View Certificates.... In the Authorities tab the certificate should be present.

Making certificate trusted in Windows

First, export the .crt certificate file from the certificate dialog, which appears after clicking button View site information in the Chrome URL field. The Copy to file button is located in the tab Details. Now you can make it trusted.

1. Start the Microsoft Management Console by running mmc command in Powershell.
2. Enter the File menu and select Add/Remove Snap In.
3. Choose Certificates Snap-In and add it to the selected. Choose Computer account in the following wizard.

4. Now you can view your certificates in the MMC snap-in. Select Console Root in the left pane, then expand Certificates (Local Computer). Under Trusted Root Certification Authorities you can import new certificate file (.crt).

5. Now refresh the Chrome page using Ctrl+F5. Things should be fine.

Making certificate trusted in Android and iOS

1. Generate certificate request from localhost.pem and localhost.key

openssl x509 -x509toreq -in localhost.pem -out localhost.csr -signkey localhost.key

2. Generate CA.crt from certificate request with special options

openssl x509 -req -days 1024 -in localhost.csr -signkey localhost.key -extfile ./android_options.txt -out CA.crt

where android_options.txt has only one line: basicConstraints=CA:true

3. Convert it to DER form

openssl x509 -inform PEM -outform DER -in CA.crt -out CA.der.crt

4. Place CA.der.crt to Android /sdcard location or download file in iOS

5. iOS: Settings -> General -> Profile -> localhost -> install it

6. iOS: Settings -> General -> Certificate Trust Settings -> localhost -> enable full trust

7. Android: Settings -> Adittional Settings -> Privacy -> Credential Storage -> Install from storage

8. Android (check): Settings -> Adittional Settings -> Privacy -> Credential Storage -> Trusted Credentials -> User -> localhost