Triangulum

A Clojure library that provides:

• A web framework based around Jetty and Ring
• Utility scripts for managing databases, email, logging, SSL certs, systemd services, and more
• Utility namespaces with functions for common CLI and web server programming tasks

Adding Triangulum to your Project

1. Add the library’s coordinates to the :deps map in your project’s deps.edn file:

   sig-gis/triangulum {:git/url "https://github.com/sig-gis/triangulum"
                       :git/sha "REPLACE-WITH-CURRENT-SHA"}
       

   If you want to pull Triangulum into a Babashka script, you can do so with babashka.deps as follows:

   (require '[babashka.deps :refer [add-deps]])
   
   (add-deps '{:deps {sig-gis/triangulum {:git/url "https://github.com/sig-gis/triangulum"
                                          :git/sha "REPLACE-WITH-CURRENT-SHA"}}})
       

2. Define one or more aliases for Triangulum’s utility scripts in the :aliases map in your project’s deps.edn file.

   NOTE: You can find examples of these aliases in the deps.example.edn file in the root directory of this repository.

3. Configure the Triangulum features you want to use in a config.edn file in the root directory of your repository.

   Two syntaxes are provided for this file, one with nested keywords and one with namespaced keywords. You can see examples of the available features in config.nested-example.edn and config.namespaced-example.edn within this repository.

Running Tests

To launch the test suite, run:

clojure -M:test

Web Framework

triangulum.server

This namespace provides a batteries-included Ring Jetty web server, an nrepl server for connecting to the live web application, a timestamped file logging system, and worker functions for non-HTTP-related tasks.

1. In config.edn:

   :server   {:http-port         8080
              :https-port        8443      ; Only if you also include the keystore fields below
              :nrepl-bind        127.0.0.1 ; Must be paired with either :nrepl or :cider-repl below
              :nrepl-port        5555      ; Must be paired with either :nrepl or :cider-repl below
              :nrepl             false     ; For a vanilla nrepl
              :cider-nrepl       true      ; If your editor supports CIDER middleware
              :mode              "dev"     ; or prod
              :log-dir           "logs"    ; or "" for stdout
              :handler           product-ns.routing/handler
              :workers           {:scheduler {:start product-ns.jobs/start-scheduled-jobs!
                                              :stop  product-ns.jobs/stop-scheduled-jobs!}}
              :response-type     :json ; #{:json :edn :transit}
              :session-key       "changeme12345678"
              :bad-tokens        #{".php"} ; Reject URLs containing these strings
              :keystore-file     "keystore.pkcs12" ; Contains your SSL certificate(s)
              :keystore-type     "pkcs12"
              :keystore-password "foobar"}
       

2. At the command line:

   Start the server:

   clojure -M:server start
       

   Stop the server (requires :nrepl or :cider-nrepl):

   clojure -M:server stop
       

   Reload your namespaces into the running JVM (requires :nrepl or :cider-nrepl):

   clojure -M:server reload
       

   NOTES:

   • To use :https-port, you must also specify :keystore-file, :keystore-type, and :keystore-password.
   • You may not use both :nrepl and :cider-nrepl at the same time.
   • The :start functions for all :workers will be run before the web server is started.
   • The :stop functions for all :workers will be run after the web server is stopped. They will receive the outputs of the :start functions.
   • Before starting the server, you must set :handler to a namespace-qualified symbol bound to a Ring handler function.
   • If you set :mode to “dev”, then the wrap-reload middleware will be added to your handler function. This will automatically reload all of your namespaces into the running JVM on each HTTP request.

triangulum.handler

The triangulum.handler namespace provides core request handling and middleware composition for Triangulum applications. It sets up a Ring handler stack that includes various middlewares, such as request/response logging, exception handling, and request parameter parsing. Optional middlewares like wrap-ssl-redirect and wrap-reload can be applied based on your config.edn settings.

Usage

If you need to provide a symbol that is bound to a handler function to figwheel-main, you can use triangulum.handler/development-app to load in your project’s handler function from config.edn with the standard Triangulum middlewares added.

Functions

triangulum.views

This namespace provides functions for rendering pages and handling resources in Triangulum. It defines functions for reading asset files, generating HTML, and handling various types of responses.

Usage

1. Require the namespace in your project.
2. Use render-page to generate a handler that will return an HTML response with a standard template for React/Reagent web apps.

Example

(ns my-app.views
  (:require [triangulum.views :refer [render-page]]))

(def my-page-handler (render-page "/my-page"))

Functions

render-page

[uri]

Returns a function that takes a request and generates the HTML for the specified URI using the request’s parameters and session data. The generated HTML includes the necessary head and body sections.

Example usage: (def my-page (render-page “/my-page”))

Caveat

In JavaScript projects, we assign the relative path (from the project root) to the main component JSX file to the :js-init key. This file should export a function called pageInit that expects two arguments: params and session. You only need to set this key for the development mode to work, which enables Vite hot reload. In production, we rely on a manifest file generated by the bundling process to find the entry point. However, we still need to define pageInit and export it in the main entry point file.

In ClojureScript projects, we need to assign the namespaced symbol of the init function to the :cljs-init key, which accepts params and session as arguments, for both production and development environments.

The session map will also contain the :client-keys that were added in Triangulum’s config.edn.

;; nested config
{:app {:client-keys {:token "client-token" }}}

;; namespaced config
{:triangulum.views/client-keys {:token "client-token" }}}

triangulum.git

You can provide :tags-url, which is a url to the git tags page of your repository. Triangulum will extract all tags beginning with “prod”, sort them lexicographically, and return the last entry. If you use tags of the form “prod-YYYY.MM.DD-HASH”, then this will return the one with the latest date.

This tag label will be passed to the browser code in the :session map under the :versionDeployed key.

Utility Scripts

triangulum.build-db

Required Prerequisites

• Postgresql (version 12)

To set up the folder and file structure for use with build-db, use the following directory structure:

src/
|___clj/
| |___<project namespace>
|
|___cljs/
| |___<project namespace>
|
|___sql/
  |___create_db.sql
  |___changes/
  |___default_data/
  |___dev_data/
  |___functions/
  |___tables/

You may also run this command in your project root directory: mkdir -p src/sql/{changes,default_data,dev_data,functions,tables}

Postgresql needs to be installed on the machine that will be hosting this website. This installation task is system specific and is beyond the scope of this README, so please follow the instructions for your operating system and Postgresql version. However, please ensure that the database server’s superuser account is named “postgres” and that you know its database connection password before proceeding.

Once the Postgresql database server is running on your machine, you should navigate to the top level directory (i.e., the directory containing this README) and add the following alias to your deps.edn file:

{:aliases {:build-db {:main-opts ["-m" "triangulum.build-db"]}}}

Then run the database build command as follows:

clojure -M:build-db build-all -d database [-u user] [-p admin password]

This will call ./src/sql/create_db.sql, stored in the individual project repository. A variable database is set for the command line call to create_db.sql. This allows your project to generate the project database with a different name, depending on your deployment. To use this variable type :database in create_db.sql where needed. You can check out Collect Earth Online to view an example.

A handy use of the build-db command is to backup and restore your database. Calling

clojure -M:build-db backup -f somefile.dump

will create a .dump backup file using pg_dump.

To restore your database from a .dump file you will need a .dump file containg a copy of a database downloaded locally. Assuming you have a copy of a database, you can then run:

clojure -M:build-db restore -f somefile.dump

This will copy the database from the .dump file into your local Postgres database of the same name as the one in the .dump file. Note that you will be prompted with a password after running this command. You should enter the Postgres master password that you first created when running Postgres after installing. Depending on the size of your .dump file, this command may take a couple of minutes. Note that if you are working on a development branch and your .dump file contains a copy of a production database you may also need to apply some of the SQL changes from the ./sql/changes directory. Assuming your database doesn’t have any of the change files on development applied to it, you can apply all of them at once using the following command:

for filename in ./src/sql/changes/*.sql; do psql -U <db-name> -f $filename; done

triangulum.build-db can also be configured through config.edn. It uses the same configuration as triangulum.database (see above).

triangulum.config

To make organizing an application’s configurations simpler, create a config.edn file in the project’s root directory. The file is just a hashmap that is similar to:

;; config.edn
{:database {:host           "localhost"
            :port           5432
            :dbname         "dbname"
            :user           "user"
            :password       "super-secret-password"}
 :mail     {:host           "smtp.gmail.com"
            :user           "test@example.com"
            :pass           "3492734923742"
            :port           587}
 :server   {:host           "smtp.gmail.com"
            :user           ""
            :pass           ""
            :tls            true
            :port           587
            :base-url       "https://my.domain/"
            :auto-validate? false}
 ...}

You can find an up-to-date example in config.nested-example.edn file. It can be used as a configuration template for your project.

Add config.edn to your .gitignore file to keep sensitive information out of the git history.

To validate the config.edn file, run:

clojure -M:config validate [-f FILE]

To retrieve a configuration, use get-config. You can supply nested configuration keys as follows:

(triangulum.config/get-config :database) ;; -> {:user "triangulum" :pass "..."}
(triangulum.config/get-config :database :user) ;; -> "triangulum"

(triangulum.config/get-config :server) ;; -> {:http-port 8080 :mode "dev"}
(triangulum.config/get-config :server :http-port) ;; -> 8080

See each section below for an example configuration if one is required for use.

triangulum.deploy

To build a JAR file from your repository and deploy it to clojars.org, run:

env CLOJARS_USERNAME=$YOUR_USERNAME CLOJARS_PASSWORD=$YOUR_CLOJARS_TOKEN clojure -M:deploy $GROUP_ID $ARTIFACT_ID

NOTE: As of 2020-06-27, Clojars will no longer accept your Clojars password when deploying. You will have to use a token instead. Please read more about this here

triangulum.https

Required Prerequisites

• certbot
• openssl

If you have not already created a SSL certificate, you must start a server without a https port specified. (e.g. clojure -M:run-server).

Add the following alias to your deps.edn file:

{:aliases {:https {:main-opts ["-m" "triangulum.https"]}}}

To automatically create an SSL certificate signed by Let’s Encrypt, simply run the following command from your shell:

sudo clojure -M:https certbot-init -d mydomain.com [-p certbot-dir] [--cert-only]

The certbot creation process will run automatically and silently.

Note: If your certbot installation stores its config files in a directory other than /etc/letsencrypt, you should specify it with the optional certbot-dir argument to certbot-init.

Certbot runs as a background task every 12 hours and will renew any certificate that is set to expire in 30 days or less. Each time the certificate is renewed, any script in /etc/letsencrypt/renewal-hooks/deploy will be run automatically to repackage the updated certificate into the correct format.

Default Renewal Hook

If certbot runs successfully and –cert-only is not specified, then a shell script [mydomain].sh will be created in the certbot deploy hooks folder. This script will run clojure -M:https package-cert. Scripts in this folder will run automatically when a new certificate is created.

While there should be no need to do so, if you ever want to perform this repackaging step manually, simply run this command from your shell:

sudo clojure -M:https package-cert -d mydomain.com [-p certbot-dir]

Custom Renewal Hook

Create a shell script in /etc/letsencrypt/renewal-hooks/deploy and update permissions.

sudo nano /etc/letsencrypt/renewal-hooks/deploy/custom.sh
sudo chmod +x /etc/letsencrypt/renewal-hooks/deploy/custom.sh

triangulum.systemd

To make sure your application starts up on system reboot, you can use Triangulum to create a systemd user .service file by adding the following to your :aliases section in the deps.edn file:

{:aliases {:systemd {:main-opts ["-m" "triangulum.systemd"]}}}

Modify your app code to call (triangulum.notify/ready!) after all of your application’s services are started:

(ns <app>.server
  (:require [triangulum.notify :as notify]))
...

(defn app-start []
  (reset! db (jdbc/connect!))
  (reset! queues (q/start!))
  (reset! server (ring/start-server!)
  (when (notify/available?) (notify/ready!))))

And then run:

clojure -M:systemd enable -r $REPO [-p $HTTP_PORT] [-P $HTTPS_PORT] [-d $REPO_DIRECTORY] [-A $EXTRA_ALIASES]

This will install a file named cljweb-<repo>.service into the ~~/.config/systemd/user/~ directory, reload the systemctl daemon, and enable your service. By default, the current directory will be used in the service as the working directory. To supply an alternative, you can use -d. This will look for a Clojure project in that directory.

The server will always be started using clojure -M:server start unless the --extra-aliases option is passed. In that case, it will run with clojure -M${EXTRA_ALIASES}:server start.

To enable your user services to start on system reboot, you will need to run:

sudo loginctl enable-linger "$USER"

Now your service will be enabled at startup. You can also start, stop, and restart your service with the following commands:

clojure -M:systemd start -r <REPO>
clojure -M:systemd stop -r <REPO>
clojure -M:systemd restart -r <REPO>

Utility Namespaces

triangulum.cli

The triangulum.cli namespace provides a command-line interface (CLI) for Triangulum applications. It includes functions for parsing command-line options, displaying usage information, and checking for errors in the provided arguments.

Usage

Use get-cli-options to parse command-line arguments and return the user’s options.

Example

(def cli-options {...})

(def cli-actions {...})
(def alias-str "...")

(get-cli-options command-line-args cli-options cli-actions alias-str)

Functions

get-cli-options

Takes the command-line arguments, a map of CLI options, a map of CLI actions, an alias string, and an optional config map. Checks for valid CLI calls and returns the user’s options.

triangulum.errors

The triangulum.errors namespace provides error handling utilities for the Triangulum application. It includes functions and macros to handle exceptions and log errors.

Functions

init-throw

Takes a message string as input and throws an exception with the provided message.

Example

(init-throw "Error: Invalid input")

try-catch-throw

Takes a function try-fn and a message string as input. Executes the function and, if it throws an exception, catches the exception, logs the error, and then throws an exception with the augmented input message.

Example

(try-catch-throw (fn [] (throw (ex-info "Initial error" {}))) "Augmented error message")

Public Macros

nil-on-error

Catches any exceptions thrown within its body and returns nil if an exception occurs. If no exception occurs, it returns the result of the body’s evaluation.

Example

(nil-on-error (/ 1 0)) ; Returns nil
(nil-on-error (+ 2 3)) ; Returns 5

triangulum.response

You can set :response-type to configure the data-response function’s default return type (:json, :edn, :transit).

triangulum.utils

The triangulum.utils namespace provides a collection of utility functions for various purposes, such as text parsing, shell command execution, response building, and operations on maps and namespaces.

Functions

Text Parsing

kebab->snake

Converts a kebab-cased string to a snake_cased string.

kebab->camel

Converts a kebab-cased string to a camelCased string.

format-str

Formats a string with placeholders (e.g., “%s”) replaced by the provided arguments.

parse-as-sh-cmd

Splits a string into an array for use with clojure.java.shell/sh.

end-with

Appends a specified string to the end of another string, if it is not already there.

remove-end

Removes a specified string from the end of another string, if it is there.

Shell Commands

shell-wrapper

A wrapper around babashka.process/shell that logs the output and errors. Accepts an optional opts map as the first argument, followed by the command and its arguments. The :log? key in the opts map can be used to control logging (default is true).

DEPRECATED: sh-wrapper

Runs a set of bash commands in a specified directory and environment. Parses the output, creating an array as described in parse-as-sh-cmd.

Response Building

DEPRECATED: data-response

Use ‘triangulum.response/data-response’ instead. Creates a response object with a specified body, status, content type, and session.

Operations on Maps

mapm

Applies a function to each MapEntry of a map, returning a new map.

filterm

Filters a map based on a predicate applied to each MapEntry, returning a new map.

reverse-map

Reverses the key-value pairs in a given map.

Equality Checking

find-missing-keys

Checks if the keys of one map are a subset of another map’s keys, including nested maps.

Namespace Operations

resolve-foreign-symbol

Attempts to require a namespace-qualified symbol’s namespace and resolve the symbol within that namespace to a value.

delete-recursively

Recursively delete all files and directories under the given directory.Traverses the directory tree in reverse depth-first order.

triangulum.type-conversion

The triangulum.type-conversion namespace provides a collection of functions for converting between different data types and formats, including conversions between numbers, booleans, JSON, and PostgreSQL data types.

Functions

Converting Numbers

val->int

Converts a value to a Java Integer. Default value for failed conversion is -1.

val->long

Converts a value to a Java Long. Default value for failed conversion is -1.

val->float

Converts a value to a Java Float. Default value for failed conversion is -1.0. Note that Postgres real is equivalent to Java Float.

val->double

Converts a value to a Java Double. Default value for failed conversion is -1.0. Note that Postgres float is equivalent to Java Double.

Converting Booleans

val->bool

Converts a value to a Java Boolean. Default value for failed conversion is false.

JSON Conversions

json->clj

Converts a JSON string to its Clojure equivalent.

jsonb->json

Converts a PostgreSQL jsonb object to a JSON string.

jsonb->clj

Converts a PostgreSQL jsonb object to its Clojure equivalent.

clj->json

Converts a Clojure value to a JSON string.

PostgreSQL Conversions

str->pg

Converts a string to a PostgreSQL object of a specified type.

json->jsonb

Converts a JSON string to a PostgreSQL jsonb object.

clj->jsonb

Converts a Clojure value to a PostgreSQL jsonb object.

triangulum.sockets

The triangulum.sockets namespace provides functionality for creating and managing client and server sockets. It includes functions for opening and checking socket connections, sending messages to the server, and starting/stopping socket servers with custom request handlers. This namespace enables communication between distributed systems and allows you to implement networked applications.

Functions

Client Socket Functions

socket-open?

Checks if the socket at the specified host and port is open.

send-to-server!

Attempts to send a socket message. Returns :success if successful.

Server Socket Functions

stop-socket-server!

Stops the running socket server.

start-socket-server!

Starts a socket server at the specified port with a custom request handler.

triangulum.notify

The triangulum.notify namespace provides functions to interact with systemd for process management and notifications. It utilizes the SDNotify Java library to send notifications and check the availability of the current process. The functions in this namespace allow you to check if the process is managed by systemd, send “ready,” “reloading,” and “stopping” messages, and send custom status messages. These functions can be helpful when integrating your application with systemd for better process supervision and management.

Functions

available?

Checks if this process is a process managed by systemd.

ready!

Sends a ready message to systemd. Systemd file must include Type=notify to be used.

reloading!

Sends a reloading message to systemd. Must call send-notify! once reloading has been completed.

stopping!

Sends a stopping message to systemd.

send-status!

Sends a custom status message to systemd. (e.g. (send-status! "READY=1")).

triangulum.email

Triangulum provides some functionality for sending email from an SMTP server. Given the configuration inside :mail. :base-url is used to configure the host url, used when sending links in emails. :auto-validate can be used in development mode, for example, to skip sending emails, which has to be configured.

triangulum.logging

To send a message to the logger use log or log-str. log can take an optional argument to specify not default behavior. The default values are shown below. log-str always uses the default values.

(log "Hello world" {:newline? true :pprint? false :force-stdout? false})
(log-str "Hello" "world")

By default the above will log to standard out. If you would like to have the system log to YYYY-DD-MM.log, set a log path. You can either specify a path relative to the toplevel directory of the main project repository or an absolute path on your filesystem. The logger will keep the 10 most recent logs (where a new log is created every day at midnight). To stop the logging server set path to “”.

(set-log-path "logs")
(set-log-path "")

triangulum.database

To use triangulum.database, first add your database connection configurations to a config.edn file in your project’s root directory.

For example:

;; config.edn
{:database {:host     "localhost"
            :port     5432
            :dbname   "pyregence"
            :user     "pyregence"
            :password "pyregence"}}

To run a postgres sql command use call-sql. Currently call-sql only works with postgres. With the second parameter can be an optional settings map (default values shown below).

(call-sql "function" {:log? true :use-vec? false} "param1" "param2" ... "paramN")

To run a sqllite3 sql command use call-sqlite. An existing sqllite3 database must be provided.

(call-sqlite "select * from table" "path/db-file")

To insert new rows or update existing rows use insert-rows! and update-rows!. If fields are not provided, the first row will be assumed to be the field names.

(insert-rows! table-name rows-vector fields-map)
(update-rows! table-name rows-vector column-to-update fields-map)

Triangulum.worker

The triangulum.worker namespace is responsible for the management of worker lifecycle within the :server context, specifically those defined under the :workers key. This namespace furnishes functions to initiate and terminate workers, maintaining their current state within an atom.

Functions

start-workers!

Starts a set of workers based on the provided configuration. The workers parameter can be either a map (for nested workers) or a vector (for namespaced workers).

For nested workers, the map keys are worker names and values are maps with :start (a symbol representing the start function) and :stop keys. The start function is called to start the worker.

For namespaced workers, the vector elements are maps with ::name (the worker name), ::start (a symbol representing the start function), and ::stop keys. The start function is called to start each worker.

Arguments:

• workers: a map or vector representing the workers to be started.

stop-workers!

Stops a set of currently running workers. The workers to stop are determined based on the current state of the `workers` atom. If the `workers` atom contains a map, it’s assumed to be holding nested workers. If it contains a vector, it’s assumed to be holding namespaced workers.

The stop function is called with the value to stop each worker.

There are no arguments for this function.

License

Copyright © 2021-2023 Spatial Informatics Group, LLC.

Triangulum is distributed by Spatial Informatics Group, LLC. under the terms of the Eclipse Public License version 2.0 (EPLv2). See LICENSE.txt in this directory for more information.