03_ProcessData.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Adding the code that performs the analysis\n",
    "\n",
    "Now for the real work &ndash; writing the code that will perform out analysis.\n",
    "\n",
    "Imagine we want to do the following:\n",
    "- Create a Jupyter notebook for exploratory analysis\n",
    "- Generate the following outputs using python scripts:\n",
    "    - Generate a subset of `winemag-130k-v2.csv` containing only the following columns: `country, designation, points, price (in GBP)`. Save in a .csv file\n",
    "    - Generate and save a table of wines only produced in Chile\n",
    "    - Save a scatterplot of the wines points vs price and a distribution plot of wine scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Don't worry you do not have to generate all of the scripts... we have provided some scripts for you to get started.\n",
    "You should now have a directory called `SupportScripts`.\n",
    "\n",
    "You need to make sure that all scripts from the directory are in the appropriate directory inside your newly created project.\n",
    "- Noteboks -> move the Jupyter notebooks for the exploratory data analysis\n",
    "- src/data -> move the raw data here\n",
    "- src/visualization (this should be left empty)\n",
    "\n",
    "Once this is done commit your changes to git.\n",
    "```bash\n",
    "$ git add .\n",
    "$ git commit -m \"Add processing scripts\"\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Now we are ready to start programming, analysing data, and testing. But let's face it.... there are going to be files\n",
    "**LOTS** of files. There are always lots of files generated along the way.\n",
    "\n",
    "![files](assets/allthefiles.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# The art of naming\n",
    "\n",
    "The three principles for (file) names:\n",
    "- **Machine readable **: regex and globbing friendly, deliberate use of delimiters *\n",
    "- **Human readable**: contains info on content, connects to concept of slug from semantic URLs\n",
    "- **Plays well with default ordering**: put something numeric first e.g. `01_data-cleaning.py`, use ISO 8601 for dates **YYYY-MM-DD**\n",
    "\n",
    "<small>* Avoid spaced, accented characters, files like 'foo' and 'Foo' </small>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![](./assets/dates_ISO.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## What works and what doesn't\n",
    "\n",
    "<table>\n",
    "  <tr>\n",
    "    <th>NO</th>\n",
    "    <th>YES</th>\n",
    "  </tr>\n",
    "  <tr>\n",
    "    <td>report.docx</td>\n",
    "    <td>2018-02-03_report-for-sla.docx</td>\n",
    "  </tr>\n",
    "  <tr>\n",
    "    <td>Joey's filename has spaces and punctuation.xlsx</td>\n",
    "    <td>joeys-filenames-are-getting-better.xlsx</td>\n",
    "  </tr>\n",
    "  <tr>\n",
    "    <td>fig 1.png</td>\n",
    "    <td>fig01_scatterplot-talk-length-vs-interest.png</td>\n",
    "  </tr>\n",
    "</table>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Running Jupyter lab\n",
    "\n",
    "We will be using [Jupyter lab](https://github.com/jupyterlab/jupyterlab) to write, execute, and modify our scripts and notebooks. \n",
    "\n",
    "You should have this installed already. We are going to start an instance by typing on the shell:\n",
    "```\n",
    "$ jupyter lab\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Metadata done right \n",
    "We are now going to use the [datapackage](https://github.com/frictionlessdata/datapackage-py) package to create some metadata for our dataset.\n",
    "\n",
    "Let's first inspect the dataset we have:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>description</th>\n",
       "      <th>designation</th>\n",
       "      <th>points</th>\n",
       "      <th>price</th>\n",
       "      <th>province</th>\n",
       "      <th>region_1</th>\n",
       "      <th>region_2</th>\n",
       "      <th>taster_name</th>\n",
       "      <th>taster_twitter_handle</th>\n",
       "      <th>title</th>\n",
       "      <th>variety</th>\n",
       "      <th>winery</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Italy</td>\n",
       "      <td>Aromas include tropical fruit, broom, brimston...</td>\n",
       "      <td>Vulkà Bianco</td>\n",
       "      <td>87</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Sicily &amp; Sardinia</td>\n",
       "      <td>Etna</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Kerin O’Keefe</td>\n",
       "      <td>@kerinokeefe</td>\n",
       "      <td>Nicosia 2013 Vulkà Bianco  (Etna)</td>\n",
       "      <td>White Blend</td>\n",
       "      <td>Nicosia</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Portugal</td>\n",
       "      <td>This is ripe and fruity, a wine that is smooth...</td>\n",
       "      <td>Avidagos</td>\n",
       "      <td>87</td>\n",
       "      <td>15.0</td>\n",
       "      <td>Douro</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Roger Voss</td>\n",
       "      <td>@vossroger</td>\n",
       "      <td>Quinta dos Avidagos 2011 Avidagos Red (Douro)</td>\n",
       "      <td>Portuguese Red</td>\n",
       "      <td>Quinta dos Avidagos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>US</td>\n",
       "      <td>Tart and snappy, the flavors of lime flesh and...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>87</td>\n",
       "      <td>14.0</td>\n",
       "      <td>Oregon</td>\n",
       "      <td>Willamette Valley</td>\n",
       "      <td>Willamette Valley</td>\n",
       "      <td>Paul Gregutt</td>\n",
       "      <td>@paulgwine</td>\n",
       "      <td>Rainstorm 2013 Pinot Gris (Willamette Valley)</td>\n",
       "      <td>Pinot Gris</td>\n",
       "      <td>Rainstorm</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>US</td>\n",
       "      <td>Pineapple rind, lemon pith and orange blossom ...</td>\n",
       "      <td>Reserve Late Harvest</td>\n",
       "      <td>87</td>\n",
       "      <td>13.0</td>\n",
       "      <td>Michigan</td>\n",
       "      <td>Lake Michigan Shore</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Alexander Peartree</td>\n",
       "      <td>NaN</td>\n",
       "      <td>St. Julian 2013 Reserve Late Harvest Riesling ...</td>\n",
       "      <td>Riesling</td>\n",
       "      <td>St. Julian</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>US</td>\n",
       "      <td>Much like the regular bottling from 2012, this...</td>\n",
       "      <td>Vintner's Reserve Wild Child Block</td>\n",
       "      <td>87</td>\n",
       "      <td>65.0</td>\n",
       "      <td>Oregon</td>\n",
       "      <td>Willamette Valley</td>\n",
       "      <td>Willamette Valley</td>\n",
       "      <td>Paul Gregutt</td>\n",
       "      <td>@paulgwine</td>\n",
       "      <td>Sweet Cheeks 2012 Vintner's Reserve Wild Child...</td>\n",
       "      <td>Pinot Noir</td>\n",
       "      <td>Sweet Cheeks</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    country                                        description  \\\n",
       "0     Italy  Aromas include tropical fruit, broom, brimston...   \n",
       "1  Portugal  This is ripe and fruity, a wine that is smooth...   \n",
       "2        US  Tart and snappy, the flavors of lime flesh and...   \n",
       "3        US  Pineapple rind, lemon pith and orange blossom ...   \n",
       "4        US  Much like the regular bottling from 2012, this...   \n",
       "\n",
       "                          designation  points  price           province  \\\n",
       "0                        Vulkà Bianco      87    NaN  Sicily & Sardinia   \n",
       "1                            Avidagos      87   15.0              Douro   \n",
       "2                                 NaN      87   14.0             Oregon   \n",
       "3                Reserve Late Harvest      87   13.0           Michigan   \n",
       "4  Vintner's Reserve Wild Child Block      87   65.0             Oregon   \n",
       "\n",
       "              region_1           region_2         taster_name  \\\n",
       "0                 Etna                NaN       Kerin O’Keefe   \n",
       "1                  NaN                NaN          Roger Voss   \n",
       "2    Willamette Valley  Willamette Valley        Paul Gregutt   \n",
       "3  Lake Michigan Shore                NaN  Alexander Peartree   \n",
       "4    Willamette Valley  Willamette Valley        Paul Gregutt   \n",
       "\n",
       "  taster_twitter_handle                                              title  \\\n",
       "0          @kerinokeefe                  Nicosia 2013 Vulkà Bianco  (Etna)   \n",
       "1            @vossroger      Quinta dos Avidagos 2011 Avidagos Red (Douro)   \n",
       "2           @paulgwine       Rainstorm 2013 Pinot Gris (Willamette Valley)   \n",
       "3                   NaN  St. Julian 2013 Reserve Late Harvest Riesling ...   \n",
       "4           @paulgwine   Sweet Cheeks 2012 Vintner's Reserve Wild Child...   \n",
       "\n",
       "          variety               winery  \n",
       "0     White Blend              Nicosia  \n",
       "1  Portuguese Red  Quinta dos Avidagos  \n",
       "2      Pinot Gris            Rainstorm  \n",
       "3        Riesling           St. Julian  \n",
       "4      Pinot Noir         Sweet Cheeks  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "wine = pd.read_csv('solutions/data/raw/winemag-data-130k-v2.csv', index_col=0)\n",
    "wine.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `datapackage` allows you to work with data packages, so we start by creating a blank data package like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datapackage\n",
    "package = datapackage.Package()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now add useful metadata by addding keys to the metadata attribute dictionary. We will start by adding the `name` key and the human-readable `title` key. For a list of the keys supported check the [DataPackage spec](https://frictionlessdata.io/specs/data-package/#metadata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'profile': 'data-package',\n",
       " 'name': 'winemag-reviews',\n",
       " 'title': 'Winemag wine reviews dataset'}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "package.descriptor['name'] = 'winemag-reviews'\n",
    "package.descriptor['title'] = 'Winemag wine reviews dataset'\n",
    "package.descriptor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inferring the data schema\n",
    "The next ste would then be to infer the data schema and generate additional metadata from our datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Some path manipulation might be needed... \n",
    "import pathlib\n",
    "import os "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'profile': 'tabular-data-package',\n",
       " 'resources': [{'path': 'solutions/data/chile.csv',\n",
       "   'profile': 'tabular-data-resource',\n",
       "   'name': 'chile',\n",
       "   'format': 'csv',\n",
       "   'mediatype': 'text/csv',\n",
       "   'encoding': 'windows-1252',\n",
       "   'schema': {'fields': [{'name': 'country',\n",
       "      'type': 'string',\n",
       "      'format': 'default'},\n",
       "     {'name': 'designation', 'type': 'string', 'format': 'default'},\n",
       "     {'name': 'points', 'type': 'integer', 'format': 'default'},\n",
       "     {'name': 'price', 'type': 'number', 'format': 'default'},\n",
       "     {'name': 'price_GBP', 'type': 'number', 'format': 'default'}],\n",
       "    'missingValues': ['']}},\n",
       "  {'path': 'solutions/data/raw/winemag-data-130k-v2.csv',\n",
       "   'profile': 'tabular-data-resource',\n",
       "   'name': 'winemag-data-130k-v2',\n",
       "   'format': 'csv',\n",
       "   'mediatype': 'text/csv',\n",
       "   'encoding': 'utf-8',\n",
       "   'schema': {'fields': [{'name': '', 'type': 'integer', 'format': 'default'},\n",
       "     {'name': 'country', 'type': 'string', 'format': 'default'},\n",
       "     {'name': 'description', 'type': 'string', 'format': 'default'},\n",
       "     {'name': 'designation', 'type': 'string', 'format': 'default'},\n",
       "     {'name': 'points', 'type': 'integer', 'format': 'default'},\n",
       "     {'name': 'price', 'type': 'number', 'format': 'default'},\n",
       "     {'name': 'province', 'type': 'string', 'format': 'default'},\n",
       "     {'name': 'region_1', 'type': 'string', 'format': 'default'},\n",
       "     {'name': 'region_2', 'type': 'string', 'format': 'default'},\n",
       "     {'name': 'taster_name', 'type': 'string', 'format': 'default'},\n",
       "     {'name': 'taster_twitter_handle', 'type': 'string', 'format': 'default'},\n",
       "     {'name': 'title', 'type': 'string', 'format': 'default'},\n",
       "     {'name': 'variety', 'type': 'string', 'format': 'default'},\n",
       "     {'name': 'winery', 'type': 'string', 'format': 'default'}],\n",
       "    'missingValues': ['']}}]}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "package.infer('./solutions/data/**/*.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(package.resources)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `infer` method has found all our files and inspected it to extract useful metadata like profile, encoding, format, Table Schema etc. Let's have a look at the resource:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'path': 'solutions/data/raw/winemag-data-130k-v2.csv',\n",
       " 'profile': 'tabular-data-resource',\n",
       " 'name': 'winemag-data-130k-v2',\n",
       " 'format': 'csv',\n",
       " 'mediatype': 'text/csv',\n",
       " 'encoding': 'utf-8',\n",
       " 'schema': {'fields': [{'name': '', 'type': 'integer', 'format': 'default'},\n",
       "   {'name': 'country', 'type': 'string', 'format': 'default'},\n",
       "   {'name': 'description', 'type': 'string', 'format': 'default'},\n",
       "   {'name': 'designation', 'type': 'string', 'format': 'default'},\n",
       "   {'name': 'points', 'type': 'integer', 'format': 'default'},\n",
       "   {'name': 'price', 'type': 'number', 'format': 'default'},\n",
       "   {'name': 'province', 'type': 'string', 'format': 'default'},\n",
       "   {'name': 'region_1', 'type': 'string', 'format': 'default'},\n",
       "   {'name': 'region_2', 'type': 'string', 'format': 'default'},\n",
       "   {'name': 'taster_name', 'type': 'string', 'format': 'default'},\n",
       "   {'name': 'taster_twitter_handle', 'type': 'string', 'format': 'default'},\n",
       "   {'name': 'title', 'type': 'string', 'format': 'default'},\n",
       "   {'name': 'variety', 'type': 'string', 'format': 'default'},\n",
       "   {'name': 'winery', 'type': 'string', 'format': 'default'}],\n",
       "  'missingValues': ['']}}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "package.descriptor['resources'][1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We might want to give this a better name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'path': 'solutions/data/raw/winemag-data-130k-v2.csv',\n",
       " 'profile': 'tabular-data-resource',\n",
       " 'name': 'winemag-reviews',\n",
       " 'format': 'csv',\n",
       " 'mediatype': 'text/csv',\n",
       " 'encoding': 'utf-8',\n",
       " 'schema': {'fields': [{'name': '', 'type': 'integer', 'format': 'default'},\n",
       "   {'name': 'country', 'type': 'string', 'format': 'default'},\n",
       "   {'name': 'description', 'type': 'string', 'format': 'default'},\n",
       "   {'name': 'designation', 'type': 'string', 'format': 'default'},\n",
       "   {'name': 'points', 'type': 'integer', 'format': 'default'},\n",
       "   {'name': 'price', 'type': 'number', 'format': 'default'},\n",
       "   {'name': 'province', 'type': 'string', 'format': 'default'},\n",
       "   {'name': 'region_1', 'type': 'string', 'format': 'default'},\n",
       "   {'name': 'region_2', 'type': 'string', 'format': 'default'},\n",
       "   {'name': 'taster_name', 'type': 'string', 'format': 'default'},\n",
       "   {'name': 'taster_twitter_handle', 'type': 'string', 'format': 'default'},\n",
       "   {'name': 'title', 'type': 'string', 'format': 'default'},\n",
       "   {'name': 'variety', 'type': 'string', 'format': 'default'},\n",
       "   {'name': 'winery', 'type': 'string', 'format': 'default'}],\n",
       "  'missingValues': ['']}}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "package.descriptor['resources'][1]['name'] = 'winemag-reviews'\n",
    "package.descriptor['resources'][1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because our resources are tabular we could read it as a tabular data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "package.get_resource('chile').read(keyed=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now save the contents of the descriptor into a `.json` file and a zip file. Every resource which content lives in the local filesystem will be copied to the zip file.\n",
    "The final structure of the zip file will be:\n",
    "```\n",
    "./datapackage.json\n",
    "./data/local.csv\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "package.save('solutions/data/datapackage.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "package.save('solutions/data/datapackage.zip')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to learn more about the `datapackage` package visit the GitHub repository [https://github.com/frictionlessdata/datapackage-py](https://github.com/frictionlessdata/datapackage-py)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# The scripts\n",
    "\n",
    "Let's start by checking the scripts and notebooks:\n",
    "- **00_explore-data.ipynb**: exploratory analysis \n",
    "- **01_subset-data-GBP.py**: subset of winemag-130k-v2.csv containing only the following columns: country, designation, points, price (in GBP). Save in a .csv file\n",
    "- **02_visualize-wines.py**\n",
    "- **03_country-subset.py**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First things first. We need to get a sense of what the data looks like and create some additional metadata for it.\n",
    "\n",
    "Open the `00_Explore-data.ipynb` notebook and run the cells."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "From the root of your file system you can run the scripts as follow (you might have to change `2018-05-09` to the current date):\n",
    "```\n",
    "$ python src/data/01_subset-data-GBP.py data/raw/winemag-data-130k-v2.csv \n",
    "$ python src/visualization/02_visualize-wines.py data/interim/2018-05-09-winemag_priceGBP.csv \n",
    "$ python src/data/03_country-subset.py data/interim/2018-05-09-winemag_priceGBP.csv Chile\n",
    "```\n",
    "\n",
    "😕 What problems did you encounter? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Documentation\n",
    "\n",
    "Documentation is an important part of a reproducible workflow.\n",
    "\n",
    "Take 5 minutes and identify which scripts/notebook have the best documentation. What makes it a good documentation?\n",
    "\n",
    "If you want to know more about documentation styles and Python style visit: [Google Python style guidelines](https://google.github.io/styleguide/pyguide.html#Comments)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![automate](assets/automate.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Packaging\n",
    "\n",
    "We used a modular approach here, so we can use and reuse the functions more efficiently. \n",
    "The next step it to make a `runall` script to minimize the user interaction. \n",
    "\n",
    "First, we need to make sure that Python recognizes our scripts as a package so we can call functions from the multiple modules.\n",
    "\n",
    "From the shell: \n",
    "```\n",
    "$ touch src/data/__init__.py  # Create empty file\n",
    "$ touch src/visualization/__init__.py  # Create empty file\n",
    "$ touch src/__init__.py\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Creating the **run all** script\n",
    "\n",
    "<div class='info'> \n",
    "    We will run everything from the root directory.<br>\n",
    "    As such all the paths will be relative to the top level of your project\n",
    "</div>\n",
    "\n",
    "Since our modules start with digits (i.e. `01`, `02`) we cannot do the import as we'd normally do\n",
    "```python \n",
    "from mypackage import myawesomemodule\n",
    "```\n",
    "Instead we need to do it like so:\n",
    "```python\n",
    "subset = importlib.import_module('.data.01_subset-data-GBP', 'src')\n",
    "plotwines = importlib.import_module('.visualization.02_visualize-wines', 'src')\n",
    "country_sub = importlib.import_module('.data.03_country-subset', 'src')\n",
    "```\n",
    "\n",
    "<div class='info'> Note that we need to make sure that the other subpackages are imported into the main package </div>\n",
    "\n",
    "So in the `src/__init__.py` you need to add:\n",
    "``` python\n",
    "from . import data\n",
    "from . import visualization\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TO DO:\n",
    "\n",
    "How would you do to run the analysis from step 01 (process the data) to 03 subset for a country and plot the results?\n",
    "\n",
    "\n",
    "Once you have done this and make sure you can run it from your shell and commit the changes to git.\n",
    "\n",
    "Note you might need to run this from the shell like so \n",
    "```\n",
    "python -m src.runall-wine-analysis\n",
    "``` \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Some additional notes\n",
    "\n",
    "We have created a base structure for this project. Thus you can leverage the use of such structure instead of running everything from the root folder. You'd need to adjust the paths bits in the `runall-wine-analysis.py` script.\n",
    "\n",
    "Also note that in the data processing scripts the interim/final produced data sets are named by using today's date e.g.\n",
    "```python\n",
    "# Constructing the fname\n",
    "today = datetime.datetime.today().strftime('%Y-%m-%d')\n",
    "fname = f'data/interim/{today}-winemag_priceGBP.csv'\n",
    "```\n",
    "\n",
    "This is **only** to demonstrate the use of appropriate naming conventions, but might be problematic at the time of testing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<link href=\"https://fonts.googleapis.com/css?family=Didact+Gothic|Dosis:400,500,700\" rel=\"stylesheet\"><style>\n",
       "@font-face {\n",
       "  font-family: \"Computer Modern\";\n",
       "  src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');\n",
       "}\n",
       "/* div.cell{\n",
       "width:800px;\n",
       "margin-left:16% !important;\n",
       "margin-right:auto;\n",
       "} */\n",
       "h1 {\n",
       "  font-family: 'Dosis', \"Helvetica Neue\", Arial, sans-serif;\n",
       "  color: #0B132B;\n",
       "}\n",
       "h2 {\n",
       "  font-family: 'Dosis', sans-serif;\n",
       "  color: #1C2541;\n",
       "}\n",
       "h3{\n",
       "  font-family: 'Dosis', sans-serif;\n",
       "  margin-top:12px;\n",
       "  margin-bottom: 3px;\n",
       "  color: #40a8a6;\n",
       "}\n",
       "h4{\n",
       "  font-family: 'Dosis', sans-serif;\n",
       "  color: #40a8a6;\n",
       "}\n",
       "h5 {\n",
       "  font-family: 'Dosis', sans-serif;\n",
       "  color: #40a8a6;\n",
       "}\n",
       "div.text_cell_render{\n",
       "  font-family: 'Didact Gothic',Computer Modern, \"Helvetica Neue\", Arial, Helvetica,\n",
       "  Geneva, sans-serif;\n",
       "  line-height: 130%;\n",
       "  font-size: 110%;\n",
       "  /* width:600px; */\n",
       "  /* margin-left:auto;\n",
       "  margin-right:auto; */\n",
       "}\n",
       "\n",
       ".text_cell_render h1 {\n",
       "  font-weight: 200;\n",
       "  font-size: 30pt;\n",
       "  /* font-size: 50pt */\n",
       "  line-height: 100%;\n",
       "  color:#0B132B;\n",
       "  margin-bottom: 0.5em;\n",
       "  margin-top: 0.5em;\n",
       "  display: block;\n",
       "}\n",
       "\n",
       ".text_cell_render h2{\n",
       "  font-weight: 500;\n",
       "}\n",
       "\n",
       ".text_cell_render h3{\n",
       "  font-weight: 500;\n",
       "}\n",
       "\n",
       "\n",
       ".warning{\n",
       "  color: rgb( 240, 20, 20 )\n",
       "}\n",
       "\n",
       "div.warn {\n",
       "  background-color: #FF5A5F;\n",
       "  border-color: #FF5A5F;\n",
       "  border-left: 5px solid #C81D25;\n",
       "  padding: 0.5em;\n",
       "\n",
       "  color: #fff;\n",
       "  opacity: 0.8;\n",
       "}\n",
       "\n",
       "div.info {\n",
       "  background-color: #087E8B;\n",
       "  border-color: #087E8B;\n",
       "  border-left: 5px solid #0B3954;\n",
       "  padding: 0.5em;\n",
       "  color: #fff;\n",
       "  opacity: 0.8;\n",
       "}\n",
       "\n",
       "</style>\n",
       "<script>\n",
       "MathJax.Hub.Config({\n",
       "  TeX: {\n",
       "    extensions: [\"AMSmath.js\"]\n",
       "    },\n",
       "    tex2jax: {\n",
       "      inlineMath: [ ['$','$'], [\"\\\\(\",\"\\\\)\"] ],\n",
       "      displayMath: [ ['$$','$$'], [\"\\\\[\",\"\\\\]\"] ]\n",
       "      },\n",
       "      displayAlign: 'center', // Change this to 'center' to center equations.\n",
       "      \"HTML-CSS\": {\n",
       "        styles: {'.MathJax_Display': {\"margin\": 4}}\n",
       "      }\n",
       "      });\n",
       "      </script>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.core.display import HTML\n",
    "\n",
    "\n",
    "def css_styling():\n",
    "    styles = open(\"styles/custom.css\", \"r\").read()\n",
    "    return HTML(styles)\n",
    "css_styling()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}