notebook_1_scripts.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Developing Custom GRASS Tools - FOSS4G 2022 Workshop\n",
    "\n",
    "Learn how to develop custom tools (aka addons or modules) for GRASS GIS in Python and, if you like, in C.\n",
    "\n",
    "Python scripting is powerful, but what is even more powerful is turning a script into a GRASS tool with just a few tricks and tweaks we will cover in this workshop. When you develop a GRASS tool (aka module), you get a graphical user interface (GUI), command line interface, and convenience you and your users will appreciate. Such tools can be published in a community-maintained addon repository which helps not only to distribute the tool, but also to maintain the code in the long term.\n",
    "\n",
    "We will focus on Python, but we will cover tools written in C, too, because even compiled tools in C and C++ can be in this community-maintained repository and distributed to users.\n",
    "\n",
    "## Authors\n",
    "\n",
    "### Vaclav Petras\n",
    "\n",
    "Vaclav (Vashek) Petras is a research software engineer, open source developer, and open science advocate. He received his masters in Geoinformatics from the Czech Technical University and PhD in Geospatial Analytics from the North Carolina State University. Vaclav is a member of the GRASS GIS Development Team and Project Steering Committee.\n",
    "\n",
    "### Anna Petrasova\n",
    "\n",
    "Anna is a geospatial research software engineer with PhD in Geospatial Analytics. She develops spatio-temporal models of urbanization and pest spread across landscape. As a member of the OSGeo Foundation and the GRASS GIS Project Steering Committee, Anna advocates the use of open source software in research and education.\n",
    "\n",
    "Thanks for providing feedback goes to: Bernardo Santos\n",
    "\n",
    "## Related talks\n",
    "\n",
    "* _Take-Home Messages from Adding Code Quality Measures to GRASS GIS_\n",
    "* _Tips for parallelization in GRASS GIS in the context of land change modeling_\n",
    "* _Using GRASS GIS in Jupyter Notebooks: An Introduction to grass.jupyter_\n",
    "* _State of GRASS GIS_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Outline\n",
    "\n",
    "- This notebook\n",
    "  * Python script structure\n",
    "  * Running Python scripts\n",
    "  * Running GRASS GIS\n",
    "- Tools for GRASS GIS in Python\n",
    "- Best practices for writing GRASS tools\n",
    "- Best practices for writing GRASS tools\n",
    "- Tools for GRASS GIS in C"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Workshop Software Setup\n",
    "\n",
    "The workshop material assumes it runs in the prepared Binder environment which is running Ubuntu and GRASS GIS is already installed there.\n",
    "\n",
    "The Binder is set up with development version of GRASS GIS 8.3, but the notebooks will work with 8.2 as well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting GRASS GIS Ready\n",
    "\n",
    "If you are running the notebook in the prepared Binder, there is nothing to do. If you are compiling GRASS GIS yourself, start JupyterLab in a way that your compiled GRASS GIS is the first _grass_ command on the path, e.g.,:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "PATH=~/grass/code/bin.x86_64-pc-linux-gnu/:$PATH jupyter lab\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For other cases, please refer to [GRASS GIS Jupyter notebooks wiki page](https://grasswiki.osgeo.org/wiki/GRASS_GIS_Jupyter_notebooks#Running_a_Jupyter_notebook_locally).\n",
    "\n",
    "Check that GRASS GIS is running and that you get the expected version:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!grass --version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting Data Ready\n",
    "\n",
    "The Binder setup already has the [full North Carolina sample dataset](https://grass.osgeo.org/sampledata/north_carolina/nc_spm_08_grass7.zip) included.\n",
    "\n",
    "To do our test runs in an isolated environment, we will create a new mapset (aka subproject) called _foss4g_:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!grass -e -c ~/grassdata/nc_basic_spm_grass7/foss4g"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To start over later on, you can use a different mapset or delete this one using:\n",
    "\n",
    "```bash\n",
    "rm -r ~/grassdata/nc_basic_spm_grass7/foss4g\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python Script\n",
    "\n",
    "Let's start with a basic Python script which uses GRASS GIS. GRASS Python packages are usually not on Python path, so we will use GRASS command line interface to get the path to these packages before we import them.\n",
    "\n",
    "The script starts a GRASS session and uses the mapset we created above. Then, it prints the current mapset name. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%python\n",
    "# Import standard Python packages we need.\n",
    "import subprocess\n",
    "import sys\n",
    "\n",
    "# Ask GRASS GIS where its Python packages are.\n",
    "sys.path.append(\n",
    "    subprocess.check_output([\"grass\", \"--config\", \"python_path\"], text=True).strip()\n",
    ")\n",
    "\n",
    "# Import the GRASS GIS packages we need.\n",
    "import grass.script as gs\n",
    "import grass.script.setup  # Needed only in 8.2 and older.\n",
    "\n",
    "# Use GRASS session as a context manager.\n",
    "with grass.script.setup.init(\"~/grassdata/nc_basic_spm_grass7/foss4g\") as session:\n",
    "    print(gs.read_command(\"g.mapset\", flags=\"p\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running Python Scripts from Command Line\n",
    "\n",
    "For testing a script and for integrating it in GRASS GIS, it is advantageous to see how a script is executed from command line or generally as a subprocess.\n",
    "\n",
    "Before, we used IPython kernel cell magic `%%python` to run a cell as a separate Python script. Now, we will use `%%writefile` cell magic to create a Python file which we will execute in the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile mapset_print_script.py\n",
    "import subprocess\n",
    "import sys\n",
    "\n",
    "sys.path.append(\n",
    "    subprocess.check_output([\"grass\", \"--config\", \"python_path\"], text=True).strip()\n",
    ")\n",
    "\n",
    "import grass.script as gs\n",
    "import grass.script.setup\n",
    "\n",
    "with grass.script.setup.init(\"~/grassdata/nc_basic_spm_grass7/foss4g\") as session:\n",
    "    print(gs.read_command(\"g.mapset\", flags=\"p\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use _python_ to execute the script. It's name (or path) are provided as parameter:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python mapset_print_script.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Full Python Script Structure\n",
    "\n",
    "The best practice for Python scripts is to use a _main_ function which is called from the so-called \"if name equals main\" block. The name of the of the _main_ function is not import, although it usually is _main_, while the syntax of the if-name-equals-main block is. Generally, all code should be in the _main_ function or called from it. This is the structure we will use from now on:\n",
    "\n",
    "```python\n",
    "def main():\n",
    "    pass\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our script, combined with the new structure, now looks like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile mapset_print_main.py\n",
    "import subprocess\n",
    "import sys\n",
    "\n",
    "sys.path.append(\n",
    "    subprocess.check_output([\"grass\", \"--config\", \"python_path\"], text=True).strip()\n",
    ")\n",
    "\n",
    "import grass.script as gs\n",
    "import grass.script.setup\n",
    "\n",
    "\n",
    "def main():\n",
    "    with grass.script.setup.init(\"~/grassdata/nc_basic_spm_grass7/foss4g\") as session:\n",
    "        print(gs.read_command(\"g.mapset\", flags=\"p\"))\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python mapset_print_main.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Executable Scripts and Shebang\n",
    "\n",
    "On unix-like systems (Linux, macOS, ...), specifying the Python interpreter can be avoided when the script has execute permissions and the first line of the script called shebang specifies which interpreter to use for the given script. A minimal script then looks like this:\n",
    "\n",
    "```python\n",
    "#!/usr/bin/env python\n",
    "\n",
    "def main():\n",
    "    pass\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    main()\n",
    "```\n",
    "\n",
    "The first line now caries very special meaning, but for Python it is just a comment, although some helper tools may recognize it.\n",
    "\n",
    "Let's add shebang to our script:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile mapset_print_executable.py\n",
    "#!/usr/bin/env python\n",
    "import subprocess\n",
    "import sys\n",
    "\n",
    "sys.path.append(\n",
    "    subprocess.check_output([\"grass\", \"--config\", \"python_path\"], text=True).strip()\n",
    ")\n",
    "\n",
    "import grass.script as gs\n",
    "import grass.script.setup\n",
    "\n",
    "\n",
    "def main():\n",
    "    with grass.script.setup.init(\"~/grassdata/nc_basic_spm_grass7/foss4g\") as session:\n",
    "        print(gs.read_command(\"g.mapset\", flags=\"p\"))\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Permissions are managed using _chmod_. `chmod u+x` makes a file executable for the user:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!chmod u+x mapset_print_executable.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Script can then run without specifying the Python interpreter:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!./mapset_print_executable.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note `./` which says the script is in the current directory. The path is always mandatory in this case even if it is the current directory. Installed GRASS tools are _on path_, i.e., are where the operating system looks for executables, so for installed tools, no path needs to be specified.\n",
    "\n",
    "The executable mechanism on Windows is different and GRASS GIS does number of steps to ensure that the scripts can be executed and right Python is used."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running in GRASS GIS\n",
    "\n",
    "GRASS tools are different from Python scripts which are using GRASS GIS in the way that they are not setting up their own GRASS session. The tools are already running in a session which was previously set up by the user in some interactive or automated way, e.g., using GUI in a desktop environment or from a Python script.\n",
    "\n",
    "The following script assumes that it runs in a GRASS session. Because we separated the concern about the GRASS session, the script is simpler: There is no need to set up path to GRASS packages and initialize GRASS session with a mapset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile mapset_print_tool.py\n",
    "#!/usr/bin/env python\n",
    "\n",
    "import subprocess\n",
    "import sys\n",
    "\n",
    "import grass.script as gs\n",
    "\n",
    "\n",
    "def main():\n",
    "    print(gs.read_command(\"g.mapset\", flags=\"p\"))\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The script can then run in an interactive GRASS session (from GUI or shell) or it can be executed using the `--exec` interface:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!grass ~/grassdata/nc_basic_spm_grass7/foss4g --exec python ./mapset_print_tool.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make the script executable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!chmod u+x mapset_print_tool.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the executable script, we can leave out `python`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!grass ~/grassdata/nc_basic_spm_grass7/foss4g --exec ./mapset_print_tool.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Command Line Parameters\n",
    "\n",
    "Scripts like all other programs, can take command line parameters. This is a crucial feature for developing general scripts and GRASS tools.\n",
    "\n",
    "Here is a simple script which prints parameters received on the command line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile command_line_print.py\n",
    "#!/usr/bin/env python\n",
    "\n",
    "import sys\n",
    "\n",
    "\n",
    "def main():\n",
    "    print(f\"Parameters are: {sys.argv}\")\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make the script executable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!chmod u+x command_line_print.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try different combinations of parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!./command_line_print.py abc xyz 1 2 3 \"dd ee ff\" '44 55 66'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The script works just the same with `grass ... --exec` where parameters go after the script:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!grass ~/grassdata/nc_basic_spm_grass7/foss4g --exec ./command_line_print.py abc \"dd ee ff\""
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}