About add_data.sh

TODO(Anthony): Move the following to their own wiki pages.

Quick way to test python/add_data.sh during development

docker cp python/add_data.sh opendrr-api_python-opendrr_1:/usr/src/app/add_data.sh && \
  docker compose up python-opendrr

docker compose up --build python-opendrr is helpful too, but it cannot be used to update the add_data.sh that was previously copied to an existing volume.

---

FAQ

Where are Elasticsearch data stored? There is no docker volume associated with it?

TODO

Regarding RUN mapfile -t ARRAY_NAME < <(jq -r '......' input.json) (Issue #111)

First of all, this is related to ShellCheck https://github.com/koalaman/shellcheck/wiki/SC2207 "Prefer mapfile or read -a to split command output (or quote to avoid splitting)". Code like array=( $(mycommand) ) is considered problematic, and mapfile -t array < <(mycommand) is offered as as correct code for Bash 4.x and up.

1. Is the syntax RUN mapfile -t ARRAY_NAME < <(jq -r '......' input.json) truly working and safe?

In particular, ss using the RUN function here OK?

The answer is: Yes! Even though the log only records mapfile -t ARRAY_NAME and nothing after the < operator, the redirection is fully functional.

2. jq errors did not stop the build. Is that good?

The <(command_list) syntax is known as process substitution. It does not stop at errors. set -o pipefail does not affect this behaviour. And the use of our custom RUN function has nothing to do with it either.

For more information and workarounds (e.g. to get the error code), see the discussion "How to detect an error using process substitution" at https://unix.stackexchange.com/questions/376114/how-to-detect-an-error-using-process-substitution

For our purposes though, I think jq errors not stopping the build is good. :-) It is probably also good to keep a log file and have add_data.sh automatically show warnings and errors at the end.

Notes on optimization, refactoring, testing, debugging, etc.

docker compose logs python-opendrr | egrep '\[add_data|maxresident'

docker compose logs pygeoapi-opendrr | grep '| NotFoundError' | uniq