notebook_3_best_practices.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Best Practices\n",
    "\n",
    "These practices are for writing GRASS tools or scripts which can be used in the same way. For general, standalone scripts, most of these simply don't apply. However, the concepts and techniques might still be useful.\n",
    "\n",
    "The assumption is that the tools is written in Python. However, the advice applies to other scripting languages as well, although, when the Python API is not available, the underlying mechanism such as environment variables may need to be used directly. The same best practices apply to tools written in C in terms of behavior, but the way how to achieve the desired behavior might be different."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use Standard Options in Interface\n",
    "\n",
    "As mentioned before, a GRASS tools must use the GRASS parser to handle its command line parameters, so that it works with the other components such as GUI.\n",
    "\n",
    "To make writing parameters simpler and the interfaces more unified, use standard options. See [Parser standard options](https://grass.osgeo.org/grass82/manuals/parser_standard_options.html). For example, use this:\n",
    "\n",
    "```python\n",
    "# %option G_OPT_V_INPUT\n",
    "# %end\n",
    "# %option G_OPT_R_OUTPUT\n",
    "# %end\n",
    "```\n",
    "\n",
    "If needed, override values which need to be different:\n",
    "\n",
    "```python\n",
    "# %option G_OPT_V_INPUT\n",
    "# % key: point_input\n",
    "# % label: Name of input vector map with points\n",
    "# % description: Points which used for sampling the raster input\n",
    "# %end\n",
    "# %option G_OPT_R_OUTPUT\n",
    "# % key: raster_input\n",
    "# % label: Name of sampled raster map\n",
    "# % description: Raster map which will be sampled by the points\n",
    "# %end\n",
    "```\n",
    "\n",
    "Don't repeat the values when a standard option defines them. For example, don't use this if possible:\n",
    "\n",
    "```python\n",
    "# Example of where standard option would work better:\n",
    "# %option\n",
    "# % key: input\n",
    "# % type: string\n",
    "# % required: yes\n",
    "# % multiple: no\n",
    "# % key_desc: name\n",
    "# % label: Name of input vector map\n",
    "# % description: Or data source for direct OGR access\n",
    "# % gisprompt: old,vector,vector\n",
    "# %end\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Consider both Flags and Options to Modify Behavior\n",
    "\n",
    "If the tool's behavior can be modified by users in some way, e.g., when its handling of nulls (no data) can be modified, a flag can be used to ask for that alternative behavior. Flags are like options which are booleans with default false with names which are only one character. They are defined using:\n",
    "\n",
    "```python\n",
    "# %flag\n",
    "# % key: n\n",
    "# % description: Consider zeros to be null values\n",
    "# %end\n",
    "```\n",
    "\n",
    "If the tool is used in Python, the flag would be used in the _flags_ parameter of _run_command_:\n",
    "\n",
    "```python\n",
    "gs.run_command(..., flags=\"n\", ...)\n",
    "```\n",
    "\n",
    "In the command line, the flag would be used with dash as `-n`.\n",
    "\n",
    "However, options are often better because they improve readability, clarify the default behavior, and allow for extension of the interface.\n",
    "\n",
    "Consider a tool which produces text output which by default produces human-readable plain text output. Then you add JSON output which is enabled by a flag `j`. Later, you decide to add YAML output. This now needs to be flag `y` which needs to be exclusive with flag `j`. Soon, you have several related flags each exclusive with all the others. Using an option instead of flag from the beginning allows the interface to accommodate more formats. In this example, an option named `format` can have default value `plain` and `json` from JSON output. When you later add YAML, you simply add `yaml` to possible values without a need for additional options or flags. The interface definition for the example would look like:\n",
    "\n",
    "```python\n",
    "# %option\n",
    "# % key: format\n",
    "# % type: string\n",
    "# % required: yes\n",
    "# % options: plain,json,yaml\n",
    "# % label: Output format\n",
    "# % descriptions: plain;Plain text output;json;JSON output;yaml;YAML output\n",
    "# % answer: plain\n",
    "# %end\n",
    "```\n",
    "\n",
    "Other typical cases where this applies include handling of computational region (e.g., `i` versus `extent=input`) or the aforementioned null handling (`n` for NULLs being zeros versus `nulls=zeros`, `nulls=nulls`, and `nulls=9999`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Input and Output Geospatial Data Format\n",
    "\n",
    "A tool should read and write geospatial data as GRASS raster or vector maps. The tools should generally use input geospatial data which are in the current GRASS location. Importing data from other formats should generally be left to dedicated import tools, e.g., _v.import_. The same applies to outputs and export of data. The obvious exceptions are import and export of data, e.g., _r.in.xyz_.\n",
    "\n",
    "The processing and analytical tools can then use simple names referring to the data in GRASS location instead of file paths. Here is an example of using existing _boundaries_ vector data and outputting new _boundaries_ raster data:\n",
    "\n",
    "```\n",
    "v.to.rast input=boundaries output=boundaries use=val\n",
    "```\n",
    "\n",
    "This follows _separation of concerns_: Format conversion and CRS transformations are separate from analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Geospatial Inputs Versus Outputs\n",
    "\n",
    "Generally, inputs should not be modified and the results of computation should go into newly created outputs. This rule usually holds for raster data processing and more often than not for vector data processing, too.\n",
    "\n",
    "If the tools adds to existing data or modifies a specific, especially auxiliary part of existing data, it may modify the existing input data instead of creating a copy with modifications. Good examples are adding an attribute column or modifying a color table."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overwriting Existing Data\n",
    "\n",
    "A tool should not overwrite existing data unless specified by the user using the `--overwrite` flag. For most cases, this is managed automatically by the  GRASS command line parser. For raster and vector maps and files in general, the parser automatically checks their existence and ends the tool execution with a proper error message in case the output already exists. If the flag is set by the user (`--overwrite` in command line, `overwrite=True` in Python), the parser enables the overwriting for the whole tool.\n",
    "\n",
    "The `--overwrite` flag can be globally enabled in GUI (_Settings > Preferences > Tools > Allow output files to overwrite existing files_), in Python scripts by `os.environ[\"GRASS_OVERWRITE\"] = \"1\"`, or in general by setting environment variable `GRASS_OVERWRITE` to `1`. Notably, the GRASS session from _grass.jupyter_ sets `GRASS_OVERWRITE` to `1` to enable re-running of the cells and notebooks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mapsets\n",
    "\n",
    "Output data should be always written to the current mapset. This is ensured by build-in GRASS mechanisms, so there is nothing which needs to be done in the tool.\n",
    "\n",
    "If a tool modifies inputs, rules for outputs apply, i.e., the input must be in the current mapset.\n",
    "\n",
    "If the tool is not modifying the inputs, the tool should accept inputs from any mapset in the current location. Tools should rely on existing GRASS mechanisms to determine in which mapset the data is in. If the user-provided names are simply passed to other GRASS tools, there is nothing to do in the tool itself. \n",
    "\n",
    "The user-provided name may or may not include mapset name. If the name is used to create, e.g., column names, the mapset needs to be resolved explicitly and possibly separated from the rest of the name. In Python, this can be done using the following code:\n",
    "\n",
    "```python\n",
    "file_info = gs.find_file(user_provided_name, element=\"cell\")  # Cell means raster here.\n",
    "full_name = file_info[\"fullname\"]\n",
    "name = file_info[\"name\"]\n",
    "mapset = file_info[\"mapset\"]\n",
    "```\n",
    "\n",
    "Good reasons to use mapsets explicitly in a tool include parallel processing - individual processes running separately in temporary mapsets - bulk processing, and, obviously, managing mapsets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "## Computational Region\n",
    "\n",
    "Tools should not change the computational region. This is done by specific tools, especially, by _g.region_.\n",
    "\n",
    "Raster processing tools should respect the current computational region. Vector processing tools may use the current computational region, e.g., to selected subset of the input data.\n",
    "\n",
    "Users should be able to re-run a command or workflow with different computational regions to, e.g., test processing in a small are and then move to a larger one.\n",
    "\n",
    "One exception to respecting the computational region rule are imports where respecting of the region is optional. The usual expectation is that the data is respected. Respecting of the region may be implemented as an optional feature. This is to avoid, e.g., importing data under finer resolution than the native resolution of the data. The tools should behave appropriately to the input data, for example, importing only the extent based on the current region may be appropriate for import of a global dataset.\n",
    "\n",
    "Another exception is raster processing where alignment of the cells plays a crucial role and there is a clear answer to how the alignment should be done. In that case, the tool may change the resolution.\n",
    "\n",
    "Some tools, such as _r.mapcalc_, opt for providing additional computation region handling policies. However, these require custom implementation and the general practice is to simply leave computation region handling out of the tool.\n",
    "\n",
    "Finally, some operations are meant use all the data, e.g., creating metadata, these operations should not use the current computational region.\n",
    "\n",
    "If a tool needs to change the computational region for part of the computation, temporary region in Python API is a simplest way to do that:\n",
    "\n",
    "```python\n",
    "gs.use_temp_region()  # From now on, use a separate region in the script.\n",
    "# Set the computational region with g.region as needed.\n",
    "```\n",
    "\n",
    "This makes any changes done in the tool local for the tool without influencing other tools running in the same session.\n",
    "\n",
    "The ultimate tool to change the computational region in a Python tool is the `GRASS_REGION` environment variable which is passed to subprocesses. (This generally works for any script, but not for tools which are using C libraries.) Python API has functions which help with the setup:\n",
    "\n",
    "```python\n",
    "os.environ[\"GRASS_REGION\"] = gs.region_env(raster=input_raster)\n",
    "```\n",
    "\n",
    "If different subprocesses need different regions, use different environments:\n",
    "\n",
    "```python\n",
    "env = os.environ.copy()\n",
    "env[\"GRASS_REGION\"] = gs.region_env(raster=input_raster)\n",
    "gs.run_command(\"r.slope.aspect\", elevation=input_raster, slope=slope, env=env)\n",
    "```\n",
    "\n",
    "This approach makes the computational region completely safe for parallel processes as no region-related files are modified."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mask\n",
    "\n",
    "GRASS GIS has a global mask managed by the _r.mask_ tool and represented by a raster called `MASK`. Tools should not set or remove the global mask. Raster tools called as subprocess will automatically respect the globally set mask when reading the data. For outputs, respecting of the mask is optional.\n",
    "\n",
    "If the tool can't avoid setting the mask internally, it should check for presence of the mask and fail if the mask is present. The tools should not remove and later restore the original mask because that creates confusing behavior for interactive use and breaks parallel processing.\n",
    "\n",
    "In addition to the global mask, tools may implement additional mask inputs, e.g., to limit interpolation of points. Interaction of the additional masking with the global mask should be documented. \n",
    "\n",
    "Generally, any mask behavior should be documented unless it is the standard case where masked cells don't participate in the computation and are represented as NULL cells (no data) in the output.\n",
    "\n",
    "The future versions of GRASS GIS will include improved global mask handling for use in tools and parallel processing and a tool to determine mask status (see PRs [2390](https://github.com/OSGeo/grass/pull/2390) and [2392](https://github.com/OSGeo/grass/pull/2392))."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional Parameters for Vector Data\n",
    "\n",
    "A tool with GRASS interface which takes vector map as an input should have at least input and layer parameters:\n",
    "\n",
    "```python\n",
    "# %option G_OPT_V_INPUT\n",
    "# %end\n",
    "# %option G_OPT_V_FIELD\n",
    "# %end\n",
    "```\n",
    "\n",
    "A layer number selects a subset of a vector map in GRASS GIS which is specified by its name. Most cases are covered by the default which is `1`, but multiple layers can be present in one vector map which allows for creation of complex data structures. The layer can also have an associated database link which links an attribute table. Where only geometry is used and attributes are not present or ignored, `-1` is used to denote all layers.\n",
    "\n",
    "Additionally, the presence of layer, covers cases where OGR-readable data in matching CRS are used directly using the OGR pseudo-mapset:\n",
    "\n",
    "```bash\n",
    "grass8 ~/grassdata/nc_basic_spm_grass7/foss4g --exec \\\n",
    "    v.to.rast input=\"~/data/project_shapefiles@OGR\" layer=all_sites output=sites use=val\n",
    "```\n",
    "\n",
    "When there is more than one input, all the additional parameters should be included for all if applicable and all names should be changed to avoid duplication.\n",
    "\n",
    "Sometimes, layer is needed for output, but usually it is not.\n",
    "\n",
    "Additionally, if it is possible, e.g., the underlying tools support it, the input vector should ideally also have:\n",
    "- type specifying geometry types (`G_OPT_V_TYPE`)\n",
    "- cats specifying category numbers (identifiers) of features to select (`G_OPT_V_CATS`)\n",
    "- where specifying SQL WHERE clause expression (`G_OPT_DB_WHERE`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Temporary Maps\n",
    "\n",
    "Using temporary map is preferred over using temporary mapsets. This follows the rule that writing should be done only to the current mapset. Some users may have write permissions only for their mapsets, but not for creating other mapsets.\n",
    "\n",
    "The following script creates a temporary name using _append_node_pid_ which is using node (computer) name and process identifier to create unique, but identifiable name. The temporary maps are removed when scripts ends that's to adding the removal function to exit procedures using _atexit.register_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile buffered_vector_to_raster.py\n",
    "#!/usr/bin/env python\n",
    "\n",
    "# %module\n",
    "# % description: Converts vector data to raster data\n",
    "# %end\n",
    "# %option G_OPT_V_INPUT\n",
    "# %end\n",
    "# %option G_OPT_V_FIELD\n",
    "# %end\n",
    "# %option G_OPT_R_OUTPUT\n",
    "# %end\n",
    "# %option\n",
    "# % key: buffer\n",
    "# % type: double\n",
    "# % required: yes\n",
    "# % description: Buffer around vector features\n",
    "# %end\n",
    "\n",
    "import atexit\n",
    "import subprocess\n",
    "import sys\n",
    "\n",
    "import grass.script as gs\n",
    "import grass.script.setup\n",
    "\n",
    "\n",
    "def remove(name):\n",
    "    gs.run_command(\"g.remove\", type=\"vector\", name=name, flags=\"f\", quiet=True, errors=\"ignore\")\n",
    "\n",
    "\n",
    "def main():\n",
    "    options, flags = gs.parser()\n",
    "    vector_input = options[\"input\"]\n",
    "    vector_layer = options[\"layer\"]\n",
    "    raster_output = options[\"output\"]\n",
    "    buffer = options[\"buffer\"]\n",
    "\n",
    "    temporary = gs.append_node_pid(\"tmp_buffer\")\n",
    "    atexit.register(remove, temporary)\n",
    "\n",
    "    gs.run_command(\"v.buffer\", input=vector_input, layer=vector_layer, output=temporary, use=\"val\")\n",
    "    gs.run_command(\"v.to.rast\", input=temporary, layer=vector_layer, output=raster_output, use=\"val\")\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Processing History\n",
    "\n",
    "Tools should record processing history to the output data.\n",
    "\n",
    "For vectors:\n",
    "\n",
    "```python\n",
    "gs.vector_history(output)\n",
    "```\n",
    "\n",
    "For rasters:\n",
    "\n",
    "```python\n",
    "gs.raster_history(output, overwrite=True)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile vector_to_raster_with_history.py\n",
    "#!/usr/bin/env python\n",
    "\n",
    "# %module\n",
    "# % description: Converts vector data to raster data\n",
    "# %end\n",
    "# %option G_OPT_V_INPUT\n",
    "# %end\n",
    "# %option G_OPT_V_FIELD\n",
    "# %end\n",
    "# %option G_OPT_R_OUTPUT\n",
    "# %end\n",
    "\n",
    "import subprocess\n",
    "import sys\n",
    "\n",
    "import grass.script as gs\n",
    "import grass.script.setup\n",
    "\n",
    "def main():\n",
    "    options, flags = gs.parser()\n",
    "    vector_input = options[\"input\"]\n",
    "    vector_layer = options[\"layer\"]\n",
    "    raster_output = options[\"output\"]\n",
    "    \n",
    "    gs.run_command(\"v.to.rast\", input=vector_input, layer=vector_layer, output=raster_output, use=\"val\")\n",
    "    gs.raster_history(raster_output, overwrite=True)\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We set executable permissions:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set execute permissions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!chmod u+x ./vector_to_raster_with_history.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the tool:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!grass ~/grassdata/nc_basic_spm_grass7/foss4g --exec ./vector_to_raster_with_history.py input=firestations output=stations_with_history --o"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show the history metadata information:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!grass ~/grassdata/nc_basic_spm_grass7/foss4g --exec r.info -h stations_with_history"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compare with the history of a raster we created earlier without managing the history:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!grass ~/grassdata/nc_basic_spm_grass7/foss4g --exec r.info -h stations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Memory and cores\n",
    "\n",
    "If the underlying tools support `memory` or `nprocs` parameters, the tool should expose those in the interface and pass the values to the underlying tools.\n",
    "\n",
    "If possible, whole raster should not be loaded into memory to avoid limiting the data size by machine memory. If the size of the processed chunks can vary, the memory consumption should be driven by a `memory` parameter. For Python tools, the proper behavior is usually taken care of by the underlying tools.\n",
    "\n",
    "The standard option `G_OPT_M_NPROCS` should be used to specify maximum number of cores (processes, threads) the tool will use. By default, only one one core should be used, or more precisely, the standard option `G_OPT_M_NPROCS` should be used with its default value. Clearly limiting the cores avoids taking more resources than the user expects."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}