notebook_4_install.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Document, Compile, Install, and Distribute\n",
    "\n",
    "There is few more steps to make the tool better for users and easy to distribute. Some of these steps are required for inclusion into the grass-addons repository.\n",
    "\n",
    "## Addons Repository\n",
    "\n",
    "- Community-maintained tools (addons aka extensions aka plugins)\n",
    "- Separate from the main repository, but only one repository\n",
    "- A repository with the source code, not just a registry\n",
    "- Best of both worlds:\n",
    "  * Broader community of contributors (including one-time contributors)\n",
    "  * Single repository maintained by the core community similarly to the main repository"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Name\n",
    "\n",
    "Tool name should follow the existing conventions for naming GRASS tools. Tools are organized into categories (families) based on their function. The categories are distinguished by prefixes:\n",
    "\n",
    "| Prefix | Function | Example |\n",
    "| --- | --- | --- |\n",
    "| r | raster processing | r.mapcalc: map algebra |\n",
    "| v | vector processing | v.clean: topological cleaning |\n",
    "| i | imagery processing | i.segment: object recognition |\n",
    "| db | database management | db.select: select values from table |\n",
    "| r3 | 3D raster processing | r3.stats: 3D raster statistics |\n",
    "| t | temporal data processing | t.rast.aggregate: temporal aggregation |\n",
    "| g | general data management | g.rename: renames map |\n",
    "| d | display | d.rast: display raster map |\n",
    "| ps | PostScript rendering | ps.map: create map compositions |\n",
    "| m | miscellaneous | m.proj: convert coordinates |\n",
    "\n",
    "The name of the module helps to understand its function, for example v.in.lidar starts with v so it deals with vector maps, the name follows with in which indicates that the module is for importing the data into GRASS GIS Spatial Database and finally lidar indicates that it deals with lidar point clouds. \n",
    "\n",
    "Generally, the idea is to include only one or two dots. All core tools comply with this rule, but some addons break it. Sometimes, this suggests further grouping. For example, tools staring with `v.net` deal with vector network analysis and tools starting with `g.gui` are opening GUI applications.\n",
    "\n",
    "Tools with non-compliant names will generally work, but may not make full use of some tools such as the GUI which uses the naming scheme to recognize GRASS tools in some contexts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Directory Structure\n",
    "\n",
    "The tool is in a directory named the same as the tool. The Python file has the name and `.py` extension.\n",
    "\n",
    "So far we have only a Python file, but all other files are in the directory as well. There can be subdirectories for special purposes; for example, tests are in a subdirectory.\n",
    "\n",
    "Now, we create the directory and rename our script from the previous notebook to comply with the rules:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p v.buffered.raster\n",
    "!cp vector_to_raster.py v.buffered.raster/v.buffered.raster.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Keywords\n",
    "\n",
    "Keywords are part of the basic description of the tools and its interface. The first two keywords are special. First keyword is the tool family of the tool, so, e.g., for vector tools, which has names starting with `v.`, the keyword is `vector`. Second keyword is a topic which is more highlighted in the documentation than other keywords. If possible, tools should use one of [existing topics](https://grass.osgeo.org/grass82/manuals/topics.html). Tool should have at least one other keyword. These can include other data types the tool works with, the name of the specific process it implements, or synonyms for the terms used in its name and description. Keywords can contain more than one word and can be understood as general labels or tags as long as they are adding to identification of the tool in searches. Reuse of [existing keywords](https://grass.osgeo.org/grass82/manuals/keywords.html) is encouraged. Keywords in Python are specified using as follows:\n",
    "\n",
    "```python\n",
    "# %module\n",
    "# % description: Converts vector data to raster data\n",
    "# % keyword: vector\n",
    "# % keyword: conversion\n",
    "# % keyword: raster\n",
    "# % keyword: rasterization\n",
    "# %end\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Documentation\n",
    "\n",
    "A file with documentation which uses simple HTML syntax must be provided. This documentation is then distributed with the addon and it is also automatically available online ([GRASS GIS Addons Manual pages](https://grass.osgeo.org/grass82/manuals/addons/)). A template with the main sections follows (the syntax highlighting does not work in notebook in JupyterLab, only in a separate editor tab)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile v.buffered.raster/v.buffered.raster.html\n",
    "<h2>DESCRIPTION</h2>\n",
    "\n",
    "A long description with details about the method, implementation, usage or whatever is appropriate.\n",
    "\n",
    "<h2>NOTES</h2>\n",
    "\n",
    "Random notes, tricks, and quirks which don't fit above.\n",
    "\n",
    "<h2>EXAMPLES</h2>\n",
    "\n",
    "Examples of how the tool can be used alone or in combination with other tools.\n",
    "Possibly using the GRASS North Carolina State Plane Metric sample Location.\n",
    "At least one screenshot (PNG format) of the result should provided to show the user what to expect.\n",
    "\n",
    "<h2>REFERENCES</h2>\n",
    "\n",
    "Reference or references to paper related to the tool or references which algorithm the tool is based on.\n",
    "\n",
    "<h2>SEE ALSO</h2>\n",
    "\n",
    "List of related or similar GRASS tools or tools used together with this tools as well as any related websites, or\n",
    "related pages at the GRASS GIS User wiki.\n",
    "\n",
    "<h2>AUTHORS</h2>\n",
    "\n",
    "List of author(s), their organizations and funding sources."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The name of the HTML file should be the name of the tool with `.html` extension.\n",
    "\n",
    "It is a good idea to include one ore more images to enhance the documentation. These should be PNGs, GIFs, or JPEGs, but if there are original files used to generate the images, these should be included as well (SVGs, scripts, notebooks). PNGs are preferred. GIFs are for animations. JPEGs for photographs.\n",
    "\n",
    "The image files should be named uniquely. The best practice is to use the name of the tool, but use underscores instead of dots, and an optional suffix. A PNG named like this without any additional suffix, may be used as an image representing the tool (this is currently done for tools in the main repository). All extensions should be lowercase (e.g., `.png`). The extension recognized for JPEG is `.jpg`.\n",
    "\n",
    "Optionally, a _README.md_ file can be included if some files or other aspects of the tool need more explanation which does not fit into any of the other files, e.g., when extra instructions are needed for re-creating the images or for maintenance of the code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Formal requirements\n",
    "\n",
    "Tools included in the grass-addons repository, must be under the GNU GPL license, version 2 or later (SPDX: GPL-2.0-or-later). There is a specified way how the first comment in tool's main file should look like. Here is a template for the first lines of a file:\n",
    "\n",
    "```python\n",
    "#!/usr/bin/env python\n",
    "\n",
    "##############################################################################\n",
    "# MODULE:    vector_to_raster\n",
    "#\n",
    "# AUTHORS:   Alice Doe <email AT some domain>\n",
    "#            Bob Doe <email AT some domain>\n",
    "#\n",
    "# PURPOSE:   Describe your script here from maintainer perspective\n",
    "#\n",
    "# COPYRIGHT: (C) 2022 Alice Doe and the GRASS Development Team\n",
    "#\n",
    "#            This program is free software under the GNU General Public\n",
    "#            License (>=v2). Read the file COPYING that comes with GRASS\n",
    "#            for details.\n",
    "##############################################################################\n",
    "\n",
    "\"\"\"Describe your script here from Python user perspective\"\"\"\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compilation\n",
    "\n",
    "Although Python is not a compiled language like C, we need to compile also the Python tools in order to include them into our GRASS installation, make them executable without the file extension, and to create HTML documentation. For this a `Makefile` needs to be written which follows a standard template as well. The included `Script.make` takes care of processing everything, given that the Python script, the HTML documentation and an optional screenshot(s) in PNG format are present in the same directory. Installed tools will show up in the GUI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile v.buffered.raster/Makefile\n",
    "MODULE_TOPDIR = ../..\n",
    "\n",
    "PGM = v.buffered.raster\n",
    "\n",
    "include $(MODULE_TOPDIR)/include/Make/Script.make\n",
    "\n",
    "default: script"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To compile, either a low level `make` command can be used, but it is easier to make use of installation mechanism of _g.extension_ which compiles and installs on Linux and macOS (and all unix-like systems).\n",
    "\n",
    "On Linux and macOS:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!grass --tmp-location XY --exec g.extension v.buffered.raster url=v.buffered.raster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!grass --tmp-location XY --exec which v.buffered.raster"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The best way to get the tools to Windows users is to include them in the grass-addons repository. (Experimentally, it is also possible to setup a private institution-specific repository like the grass-addons repository.)\n",
    "\n",
    "Code can be hosted on GitHub or other platform. _g.extension_ supports installation from many sources, but it needs the compilation tools which are not available on Windows, so this works only for Linux and macOS."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submitting to the GRASS GIS Addons Repository\n",
    "\n",
    "Create a pull request to the [grass-addons repository](https://github.com/OSGeo/grass-addons/) (instructions are there). Wait for someone to review it or convince someone to do that. When issues from the review are addressed, the reviewer will merge it.\n",
    "\n",
    "Finally, check the [Submitting Guidelines](https://trac.osgeo.org/grass/wiki/Submitting) (both for updating your files and updating the guidelines themselves).\n",
    "\n",
    "PR reviews are time consuming, so make it easier for the reviewer by checking the best practices yourself. And yes, you can become a reviewer and get access to the grass-addons repository too. That's actually much simpler than getting write access to the main repository."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing your code\n",
    "\n",
    "One way to speed up the review process is to include tests of your tool. This not only demonstrates that the tool works, but it makes it also easier to maintain the code in the future.\n",
    "\n",
    "### grass.gunittest tests\n",
    "\n",
    "- Based on highly customized extension of the standard Python _unittest_ package.\n",
    "- Code runs in an existing GRASS session.\n",
    "- Can assume the sample NC SPM location is the current location.\n",
    "- Many specialized functions for GRASS GIS, especially specialized asserts.\n",
    "\n",
    "The readily available test real-world data and assert functions specialized for GRASS GIS, make _grass.gunittest_ a great tool for tests of data processing tools.\n",
    "\n",
    "### pytest tests\n",
    "\n",
    "- Use _pytest_ as is.\n",
    "- There are no specialized functions for GRASS GIS yet.\n",
    "- Fixtures and comparisons need to be written using basic functions.\n",
    "- No GRASS session or data.\n",
    "\n",
    "The lack of any setup may work well for tests of tools which are not doing standard processing. Increasing number of project and people migrate to _pytest_, so you may simply prefer that."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Grouping Related Tools\n",
    "\n",
    "### Naming\n",
    "\n",
    "Just use names which start with the same prefix or contain the same words, but keep thinks separate.\n",
    "\n",
    "### Common Directory\n",
    "\n",
    "If it makes sense to use each tool individually, the tools can be in separate directories.\n",
    "\n",
    "One directory with multiple tools in subdirectories works well even for mix of Python and C tools. There should be an additional HTML documentation in the top directory. The name of the directory should be the common prefix in the name of the tools. If there is no common prefix, the tools should likely be separate.\n",
    "\n",
    "However, a step further, including common libraries to this structure, is more complicated and the functionality is not as stable as it should be. Creating a common library is not worth the trouble for a couple of short functions shared over couple of the tools, but it may be better for maintenance of a large library of broadly used functions. Note that the tools can also call each other possibly avoiding needs for Python imports. \n",
    "\n",
    "### Experimental Toolboxes\n",
    "\n",
    "#### Addon Toolbox\n",
    "\n",
    "Multiple tools can be listed together in an XML file and _g.extension_ can show and install this toolbox.\n",
    "\n",
    "#### GUI Toolbox\n",
    "\n",
    "Multiple tools can be listed together in an XML file which is stored in user configuration directory. The GUI adds these toolboxes to the tree in the _Tools_ tab."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}