04_Testing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Testing\n",
    "\n",
    "We now have a fully automated script! 🎉👏🏻🦄\n",
    "\n",
    "😕 Such a shame we still cannot guarantee the results are correct... or that there are no bugs.\n",
    "\n",
    "The next step is to include **tests**... in fact testing should be a core part of our development process. In fact all of our **reproducible workflows** are analogous to experimental design in the scientific world"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![science](./assets/the_difference.png)\n",
    "\n",
    "<small> https://xkcd.com/242/ </small>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "There are various approaches to test software:\n",
    "- **Assertions**: 🦄 == 🦄\n",
    "- **Exceptions**: (within the code) serve as warnings ⚠️\n",
    "- **Unit tests**: investigate the behaviour of units of code (e.g functions)\n",
    "- **Regression tests**: defends against 🐛\n",
    "- **Integration tests**: ⚙️ checks that the pieces work together as expected"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Exceptions \n",
    "Remember when you tried to run `02_visualize-wines.py`? It would not work unless you had created a figures directory beforehand.\n",
    "\n",
    "We can catch this kinds of errors by adding this piece of code:\n",
    "```python\n",
    "try:\n",
    "        # try to save the figure\n",
    "        fig.savefig(fname, bbox_inches = 'tight')\n",
    "    except OSError as e:\n",
    "        # wowza! the directory does not exist\n",
    "        os.makedirs('figures')\n",
    "        print('Creating figures directory')\n",
    "        fig.savefig(fname, bbox_inches='tight')\n",
    "```\n",
    "Now our `runall` should work!!! 🎉🎉\n",
    "\n",
    "```\n",
    "$ python src.runall-wine-analysis\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Unit testing\n",
    "Open `03_country-subset.py` and add the following function:\n",
    "    \n",
    "```python \n",
    "def get_mean_price(filename):\n",
    "    \"\"\" function to get the mean price of the wines\n",
    "    rounded to 4 decimals\"\"\"\n",
    "    wine = pd.read_csv(filename)\n",
    "    mean_price = wine['price'].mean()\n",
    "    return round(mean_price, 4)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "And we will modify `get_country` too, so it return the data frame:\n",
    "```python\n",
    "def get_country(filename, country):\n",
    "    # Load table\n",
    "    wine = pd.read_csv(filename)\n",
    "\n",
    "    # Use the country name to subset data\n",
    "    subset_country = wine[wine['country'] == country ].copy()\n",
    "\n",
    "    # Constructing the fname\n",
    "    today = datetime.datetime.today().strftime('%Y-%m-%d')\n",
    "    fname = f'data/processed/{today}-winemag_{country}.csv'\n",
    "\n",
    "    # Saving the csv\n",
    "    subset_country.to_csv(fname)\n",
    "    print(fname)  # print the fname from here\n",
    "\n",
    "    return(subset_country)  #returns the data frame\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Now we are going to create our testing suite. \n",
    "To run the tests we are going to use **pytest**.\n",
    "You can find more information in the following resources:\n",
    "- Pytest usage examples can be found [here](http://doc.pytest.org/en/latest/usage.html)\n",
    "- Rules for [test discovery](http://doc.pytest.org/en/latest/goodpractices.html)\n",
    "- Great [posts about testing and pytest](http://pythontesting.net/start-here/)\n",
    "\n",
    "Now we can create our tests:\n",
    "```\n",
    "$ mkdir tests                     # Create tests directory\n",
    "$ touch tests/__init__.py         # Help find the test\n",
    "$ touch test_03_country_subset.py # Create our first test\n",
    "```\n",
    "⭐ Your test scripts name must start with: `test`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Modifying <code>test_03_country_subset.py</code>\n",
    "``` python\n",
    "import importlib\n",
    "\n",
    "country = importlib.import_module('.data.03_country-subset', 'src')\n",
    "\n",
    "# you might need to change the date so that it matches today's date\n",
    "processed_data = \"data/processed/2018-05-09-winemag_Chile.csv\"\n",
    "\n",
    "def test_get_mean_price():\n",
    "    mean_price = country.get_mean_price(processed_data)\n",
    "    assert mean_price == 20.7865\n",
    "```\n",
    "\n",
    "And you can run it from the shell using:\n",
    "```\n",
    "$ pytest\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### What if you want all the decimal numbers?\n",
    "\n",
    "``` python\n",
    "import importlib\n",
    "import numpy.testing as npt\n",
    "\n",
    "country = importlib.import_module('.data.03_country-subset', 'src')\n",
    "\n",
    "processed_data = \"data/processed/2018-05-09-winemag_Chile.csv\"\n",
    "\n",
    "def test_get_mean_price():\n",
    "    mean_price = country.get_mean_price(processed_data)\n",
    "    assert mean_price == 20.7865\n",
    "    npt.assert_allclose(country.get_mean_price(processed_data) , 20.787, rtol = 0.01)\n",
    "```\n",
    "\n",
    "The `numpy.testing.assert_allclose` allows you to set a tolerance "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### What else could go wrong?\n",
    "\n",
    "What if we created a data set and we want to make sure that my interim or raw data has not changed? -> Thus my dataframes have not changes either?\n",
    "\n",
    "```python \n",
    "import pandas.testing as pdt\n",
    "import pandas as pd\n",
    "\n",
    "\n",
    "interim_data = \"data/interim/2018-05-09-winemag_priceGBP.csv\"\n",
    "processed_data = \"data/processed/2018-05-09-winemag_Chile.csv\"\n",
    "\n",
    "def test_get_country():\n",
    "    # call the function\n",
    "    df = country.get_country(interim_data, 'Chile')\n",
    "    \n",
    "    # load my previous dataset\n",
    "    base = pd.read_csv(processed_data)\n",
    "    \n",
    "    # check if I am getting a dataframe\n",
    "    assert isinstance(df, pd.DataFrame)\n",
    "    assert isinstance(base, pd.DataFrame)\n",
    "    \n",
    "    # check that they are the same dataframes\n",
    "    pdt.assert_frame_equal(df, base)\n",
    "```    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Pytest tells us which tests passed and which did not:\n",
    "\n",
    "```python\n",
    " {message}\n",
    "    [left]:  {left}\n",
    "    [right]: {right}\"\"\".format(obj=obj, message=message, left=left, right=right)\n",
    "\n",
    "        if diff is not None:\n",
    "            msg += \"\\n[diff]: {diff}\".format(diff=diff)\n",
    "\n",
    ">       raise AssertionError(msg)\n",
    "E       AssertionError: DataFrame are different\n",
    "E\n",
    "E       DataFrame shape mismatch\n",
    "E       [left]:  (4472, 6)\n",
    "E       [right]: (4472, 7)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We now know what kind of bugs we can encounter.\n",
    "Let's fix this, open `03_subset-country.py` and add the lines to reset index of `subset_country` and remove the index when saving to the csv file.\n",
    "\n",
    "```python\n",
    "def get_country(filename, country):\n",
    "    # Load table\n",
    "    wine = pd.read_csv(filename)\n",
    "\n",
    "    # Use the country name to subset data\n",
    "    subset_country = wine[wine['country'] == country ].copy()\n",
    "    subset_country.reset_index(drop=True, inplace=True) \n",
    "\n",
    "    # Constructing the fname\n",
    "    today = datetime.datetime.today().strftime('%Y-%m-%d')\n",
    "    fname = f'data/processed/{today}-winemag_{country}.csv'\n",
    "\n",
    "    # Saving the csv\n",
    "    subset_country.to_csv(fname, index=False)\n",
    "    print(fname)  # print the fname from here\n",
    "\n",
    "    return(subset_country)  #returns the data frame\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### See what we did in the previous steps?\n",
    "\n",
    "We tested each of the functions in our module...\n",
    "we did *unit testing*!\n",
    "Notice something in the functions we just wrote? \n",
    "- Set-up: `mean = country.get_mean(interim_data)`\n",
    "- Assertions: `assert mean_price == 20.786`\n",
    "\n",
    "Now don't forget to commit your code:\n",
    "```\n",
    "$ git add .\n",
    "$ git commit -m \"Add unit test suite\"\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Past as Truth (regression tests)\n",
    "\n",
    "Regression tests assume that the past is “correct”. They are great for letting developers know when and how a code base has changed. They are not great for letting anyone know why the change occurred. The change between what a code produces now and what it computed before is called a regression.\n",
    "\n",
    "** How many times have you tried to run a script or a notebook you found online just to realize it is broken?**\n",
    "\n",
    "Let's do some regression testing on the Jupyter notebook using [nbval](https://github.com/computationalmodelling/nbval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Testing Jupyter notebooks with nbval\n",
    "\n",
    "We first need to understand how a Jupyter notebook works. \n",
    "All the data is stored in a .json like format (organised key, data values)... this includes the results, code, and markdown.\n",
    "\n",
    "![json](assets/json.jpg)\n",
    "\n",
    "Nbval checks the stored values while doing a *mock run* on the notebook and compares the saved version of the notebook vs the results obtained from the mock run \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Try it on your shell \n",
    "\n",
    "```\n",
    "$ pytest --nbval src/data/00_explore-data.ipynb\n",
    "```\n",
    "\n",
    "What would happen if you were to have a cell like this one?\n",
    "```python\n",
    "import time\n",
    "print('This notebook was last run on: ' + time.strftime('%d/%m/%y') + ' at: ' + time.strftime('%H:%M:%S'))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we have a timestamp in the notebook (`time.strftime('%H:%M:%S'))`) and the test is performed at a different time, the tests will fail. \n",
    "\n",
    "We can avoid this by providing a sanitizing file `sanitize.cfg`:\n",
    "<pre><code>\n",
    "[regex1]\n",
    "regex: \\d{1,2}/\\d{1,2}/\\d{2,4}\n",
    "replace: DATE-STAMP\n",
    "[regex2]\n",
    "regex: \\d{2}:\\d{2}:\\d{2}\n",
    "replace: TIME-STAMP\n",
    "</pre></code>\n",
    "\n",
    "This will identify time and data stamp like data in your notebooks. Then you can re run the test using:\n",
    "<pre><code>\n",
    "py.test --nbval demo.ipynb --sanitize-with sanitize_nb.cfg\n",
    "</pre></code>\n",
    "\n",
    "If you are only interested in verifying if your notebooks <strong>are broken or not</strong> (check for exceptions)\n",
    "you can use `--nbval-lax` which runs notebooks and checks for errors, but only compares the outputs of cells with a `#NBVAL_CHECK_OUTPUT` marker comment.\n",
    "<pre><code>\n",
    "$ py.test --nbval-lax classify-demo.ipynb\n",
    "</pre></code>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TDD and tests first\n",
    "\n",
    "Have you ever heard about test-driven-development? It is a commonly used practice in which you write the tests for your scripts before or at the time of writing the actual code. \n",
    "\n",
    "Some of the advantages include early bug detection, better test coverage, and generally a higher code quality. It also helps you ensure you know what your code is doing at all times and makes it easier to add new features without major risk of breakdown. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Provenance\n",
    "\n",
    "Image you created a beautiful graph and some results that makes your research Nobel worthy. Of course you ran the workflow multiple times doing minimal changes every single time. But now, 6 months later you need that **one** plot for you Nobel!!\n",
    "\n",
    "We can use the package [recipy](https://github.com/recipy/recipy) to log each run of your code to a database, keeping track of the input files, output files and the version of your code, and then let you query this database to find out how you actually did create graph.png"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<div class='warn'>Make sure everything is commited to git before carrying on.</div>\n",
    "<br>\n",
    "Add the following line to your `runall-wine-analysis` script\n",
    "\n",
    "```python\n",
    "import recipy\n",
    "```\n",
    "Run the script again `python -m src.runall-wine-analysis`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "You can now track the provenance of your project. \n",
    "\n",
    "Try using `recipy latest` and `recipy gui`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<link href=\"https://fonts.googleapis.com/css?family=Didact+Gothic|Dosis:400,500,700\" rel=\"stylesheet\"><style>\n",
       "@font-face {\n",
       "  font-family: \"Computer Modern\";\n",
       "  src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');\n",
       "}\n",
       "/* div.cell{\n",
       "width:800px;\n",
       "margin-left:16% !important;\n",
       "margin-right:auto;\n",
       "} */\n",
       "h1 {\n",
       "  font-family: 'Dosis', \"Helvetica Neue\", Arial, sans-serif;\n",
       "  color: #0B132B;\n",
       "}\n",
       "h2 {\n",
       "  font-family: 'Dosis', sans-serif;\n",
       "  color: #1C2541;\n",
       "}\n",
       "h3{\n",
       "  font-family: 'Dosis', sans-serif;\n",
       "  margin-top:12px;\n",
       "  margin-bottom: 3px;\n",
       "  color: #40a8a6;\n",
       "}\n",
       "h4{\n",
       "  font-family: 'Dosis', sans-serif;\n",
       "  color: #40a8a6;\n",
       "}\n",
       "h5 {\n",
       "  font-family: 'Dosis', sans-serif;\n",
       "  color: #40a8a6;\n",
       "}\n",
       "div.text_cell_render{\n",
       "  font-family: 'Didact Gothic',Computer Modern, \"Helvetica Neue\", Arial, Helvetica,\n",
       "  Geneva, sans-serif;\n",
       "  line-height: 130%;\n",
       "  font-size: 110%;\n",
       "  /* width:600px; */\n",
       "  /* margin-left:auto;\n",
       "  margin-right:auto; */\n",
       "}\n",
       "\n",
       ".text_cell_render h1 {\n",
       "  font-weight: 200;\n",
       "  font-size: 30pt;\n",
       "  /* font-size: 50pt */\n",
       "  line-height: 100%;\n",
       "  color:#0B132B;\n",
       "  margin-bottom: 0.5em;\n",
       "  margin-top: 0.5em;\n",
       "  display: block;\n",
       "}\n",
       "\n",
       ".text_cell_render h2{\n",
       "  font-weight: 500;\n",
       "}\n",
       "\n",
       ".text_cell_render h3{\n",
       "  font-weight: 500;\n",
       "}\n",
       "\n",
       "\n",
       ".warning{\n",
       "  color: rgb( 240, 20, 20 )\n",
       "}\n",
       "\n",
       "div.warn {\n",
       "  background-color: #FF5A5F;\n",
       "  border-color: #FF5A5F;\n",
       "  border-left: 5px solid #C81D25;\n",
       "  padding: 0.5em;\n",
       "\n",
       "  color: #fff;\n",
       "  opacity: 0.8;\n",
       "}\n",
       "\n",
       "div.info {\n",
       "  background-color: #087E8B;\n",
       "  border-color: #087E8B;\n",
       "  border-left: 5px solid #0B3954;\n",
       "  padding: 0.5em;\n",
       "  color: #fff;\n",
       "  opacity: 0.8;\n",
       "}\n",
       "\n",
       "</style>\n",
       "<script>\n",
       "MathJax.Hub.Config({\n",
       "  TeX: {\n",
       "    extensions: [\"AMSmath.js\"]\n",
       "    },\n",
       "    tex2jax: {\n",
       "      inlineMath: [ ['$','$'], [\"\\\\(\",\"\\\\)\"] ],\n",
       "      displayMath: [ ['$$','$$'], [\"\\\\[\",\"\\\\]\"] ]\n",
       "      },\n",
       "      displayAlign: 'center', // Change this to 'center' to center equations.\n",
       "      \"HTML-CSS\": {\n",
       "        styles: {'.MathJax_Display': {\"margin\": 4}}\n",
       "      }\n",
       "      });\n",
       "      </script>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.core.display import HTML\n",
    "\n",
    "\n",
    "def css_styling():\n",
    "    styles = open(\"styles/custom.css\", \"r\").read()\n",
    "    return HTML(styles)\n",
    "css_styling()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}