00A_preparation_and_install.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Welcome to the Dark Art of Coding:\n",
    "## Introduction to Data Science Fundamentals\n",
    "Preparation and installation guide\n",
    "\n",
    "<img src='images/logos.3.600.wide.png' height='250' width='300' style=\"float:right\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Main objectives\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By the end of this module, you will be able to:\n",
    "\n",
    "* Download the tools we will be using (conda, specific Python libraries)\n",
    "* Install the tools\n",
    "* Test them for successful installation\n",
    "* Open the Juptyer Notebooks that we will be using in class\n",
    "* Run the code samples found in the notebooks\n",
    "* Understand the importance of the tools for our tasks today\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Installing the Software You'll Need\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step Zero: Read through ALL the steps...\n",
    "\n",
    "We strongly recommend that you read through **ALL** the steps below, before you start to install, etc. For some more advanced practitioners, you **may** already have some tools installed OR available. \n",
    "\n",
    "IF you can successfully open the Jupyter Notebooks AND import the data libraries listed below in Step Three, then you shouldn't need to do anything. For folks who aren't sure OR for folks who are fairly new to Python/programming... these steps should get us to the point we need to be."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step One: Download and Install Miniconda\n",
    "\n",
    "Follow the instructions for your operating system in the **`miniconda`** quickstart guides.\n",
    "\n",
    "**Some warnings/cautions:**\n",
    "\n",
    "1. We **highly recommend** the use of `conda` as a package manager and virtual environment manager for this tutorial. This material has been tested using `conda` but has not been tested using `pip`, `virtualenv`, `pyenv`, etc.\n",
    "\n",
    "1. **IF you already have `conda`** installed via a previous `Anaconda` OR `miniconda` install, you should not need to reinstall. How can you tell? If you type `conda` on your command line and get a response similar to this, then you should be fine:\n",
    "\n",
    "    ```\n",
    "    my_macbook:my_folder chalmerlowe$ conda\n",
    "    usage: conda [-h] [-V] command ...\n",
    "\n",
    "    conda is a tool for managing and deploying applications, environments and packages.\n",
    "    .\n",
    "    .\n",
    "    .\n",
    "    ```\n",
    "1. Use a **Python 3.x** version of `miniconda` to install Python 3.x.\n",
    "1. Based on our experience in workshops, **the most common problems** we experience with installs is that a step got missed OR a command was typed incorrectly. It happens to all of us, so stay sharp, folks!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With that in mind, please choose the appropriate version and install `conda`\n",
    "\n",
    "**`conda` for Windows**:\n",
    "\n",
    "* Download the installer: [Miniconda installer for Windows.](https://conda.io/miniconda.html)\n",
    "* Double-click the .exe file.\n",
    "* Follow the instructions on the screen.\n",
    "* NOTE: If you are unsure about any setting, accept the defaults. You can change them later.\n",
    "* When installation is finished, from the **Start menu**, open the **Anaconda Prompt**.\n",
    "\n",
    "**`conda` for MacOS**:\n",
    "\n",
    "* Download the installer: [Miniconda installer for MacOS.](https://conda.io/miniconda.html)\n",
    "* In your Terminal window, run:\n",
    "    \n",
    "    ```bash\n",
    "    $ bash Miniconda3-latest-MacOSX-x86_64.sh\n",
    "    ```\n",
    "<br>\n",
    "* Follow the prompts on the installer screens.\n",
    "* NOTE: If you are unsure about any setting, accept the defaults. You can change them later.\n",
    "* **Close** and then **re-open** your Terminal window, to make the changes take effect.\n",
    "\n",
    "\n",
    "**`conda` for Linux**:\n",
    "\n",
    "* Download the installer: [Miniconda installer for Linux.](https://conda.io/miniconda.html)\n",
    "* In your Terminal window, run:\n",
    "    \n",
    "    ```bash\n",
    "    $ bash Miniconda3-latest-Linux-x86_64.sh\n",
    "    ```\n",
    "<br>\n",
    "* Follow the prompts on the installer screens.\n",
    "* If you are unsure about any setting, accept the defaults. You can change them later.\n",
    "* **Close** and then **re-open** your Terminal window, to make the changes take effect."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step Two: Confirm your conda install\n",
    "\n",
    "In a command prompt type `conda list`. If `conda` is installed properly, you will see a summary of the packages installed by `conda`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Troubleshooting\n",
    "\n",
    "Here's a list of error messages & how to fix them.\n",
    "\n",
    "- **`conda: Command not found.` **IF you see this, the most common reason is that your command shell is not yet aware of the installation of `conda`. The easiest fix is to simply **close** your terminal/command prompt & **reopen** your terminal/command prompt. If that doesn't fix it, ask for help."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step Three: Install Python, and other packages..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With `conda` installed, we want to ensure that we have a suitable version of Python installed and that we have the necessary libraries also installed.\n",
    "\n",
    "We will create a directory to hold our lesson content, for consistency, we will call this directory `stats` and then we will create a virtual environment and populate it with Python and our libraries.\n",
    "\n",
    "1. On your command prompt, make sure you are in a directory where you want your project folder to be located (many people put this in their `My Documents` OR `home` folder. From that directory, run the following command:\n",
    "\n",
    "    ```bash\n",
    "    chalmerlowe$ mkdir stats\n",
    "    ```\n",
    "    <br>\n",
    "1. Change directories into the new folder:\n",
    "\n",
    "    ```bash\n",
    "    chalmerlowe$ cd stats\n",
    "    ```\n",
    "    <br>\n",
    "1. Create a virtual environment with Python 3, using the following command (don't worry, we will explain this below):\n",
    "\n",
    "    ```bash\n",
    "    chalmerlowe$ conda create -n stats python=3\n",
    "    ```\n",
    "    <br>\n",
    "1. Activate your virtual environment using the command appropriate to your operating system. NOTICE your prompt will change to reflect the fact that you are now in a virtual environment:\n",
    "\n",
    "    **Mac/ Linux** \n",
    "\n",
    "    ```bash\n",
    "    chalmerlowe$ source activate stats\n",
    " (stats) chalmerlowe$    \n",
    "    ```\n",
    "    <br>\n",
    "    **Windows**\n",
    "\n",
    "    ```bat\n",
    "    C:\\> activate stats\n",
    "    (stats) C:\\>\n",
    "    ```\n",
    "    <br>\n",
    "\n",
    "1. Install the following additional packages to your virtual environment:\n",
    "\n",
    "    ```bash\n",
    "    (stats) chalmerlowe$ conda install jupyterlab matplotlib numpy pandas scipy \n",
    "    ```\n",
    "    <br> \n",
    "\n",
    "1. Test your installation, by typing the following on your commmand line/terminal:\n",
    "\n",
    "    ```bash\n",
    "    (stats) chalmerlowe$ jupyter lab \n",
    "    ```\n",
    "    <br>\n",
    "    \n",
    "    If your browser opens with a Jupyter Lab instance, you will know the install process succeeded. \n",
    "<br>\n",
    "1. Download the [class materials](https://github.com/chalmerlowe/jarvis_II/tree/master/datasci_basics) into your `stats` folder (If necessary, unzip the content).\n",
    "\n",
    "1. The files should then be visible in your Jupyter Lab environment, under the files tab on the left.\n",
    "\n",
    "<img src='images/jupyter_lab.png' width='600' style='float:left'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Done with commands for now!\n",
    "\n",
    "If you (if you're working in pairs, you and your partner) are done, then now you can put your green sticky up! This is how we know you're done with the commands.\n",
    "\n",
    "<img src='images/green_sticky.300px.png' width='200' style='float:left'>\n",
    "\n",
    "If you like reading, you can also keep reading this page."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Big Picture \n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is miniconda (conda) and why did we install it?\n",
    "\n",
    "Miniconda contains the `conda` package manager/virtual environment manager and `Python`. `conda` is language agnostic, so you can also use it to support work with languages besides Python. Once miniconda is installed, you will be able to: \n",
    "\n",
    "* create virtual environments and \n",
    "* manage separate installations of `Python` (including different versions) \n",
    "* manage Python packages/libraries\n",
    "* as well as manage packages in other languages ... packages that are fundamentally unmanageable by Python-only tools like `pip` & `virtualenv`.\n",
    "\n",
    "Whenever you work on a new project, you should create a separate environment for that project. `conda` lets you do this easily and efficiently. A later lesson will provide more details on both virtual environments and the use of the `conda` package manager.\n",
    "\n",
    "## And what is a virtual environment?\n",
    "\n",
    "When you create a virtual environment, `conda` will add subdirectories to a miniconda directory on your computer. Specifically it will create a directory that will contain:\n",
    "\n",
    "* a database and metadata about the virtualenv\n",
    "* software and libraries related to the project (i.e., Python and any modules you install in the virtualenv)\n",
    "\n",
    "NOTE: this virtualenv folder is **NOT** a duplicate of your project folder **NOR** does it contain your code/class material"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep dive\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is a virtual environment?\n",
    "\n",
    "As mentioned above, virtual environments (also called virtualenvs) are tools used to keep projects separate, especially in terms of keeping different software versions separate and different library versions separate. For example, virtualenvs prevent Python's site packages folder from getting filled with potentially conflicting versions of software AND thus prevents problems that arise when one project needs **version x.x** of a library but another project needs **version y.y** of the same library. At their core, virtualenvs are glorified directories that use scripts and metadata to organize and control the environment. You are allowed to have an essentially unlimited number of virtual environments. And as you saw above, they are very easy to create using various command line tools, such as `conda`.\n",
    "\n",
    "## When should we use a virtual environment?\n",
    "\n",
    "Anytime you have more than one project and there is a possibility of conflicts between your libraries, it is a good time to use a virtual environment. Having said that, many programmers use virtual environments for **all but the most trivial** programming tasks. Especially for beginners, using virtual environmentss early on in your learning career will build a valuable skill AND help prevent sneaky bugs related to version discrepancies. Bugs that can be hard to diagnose.\n",
    "\n",
    "## How do you create a virtual environment?\n",
    "\n",
    "While there are several programs or libraries that can generate virtual environments for today's lesson, we will be using the `conda` package manager, which includes the capability to simply and easily produce virtual environments.\n",
    "\n",
    "Presuming you have `conda` installed, these steps enable you to create and activate a virtual environment.\n",
    "\n",
    "```bash\n",
    "$ conda create -n stats python=3\n",
    "```\n",
    "\n",
    "Description:\n",
    "* `conda` runs the conda program.\n",
    "* `create` tells it to create a virtualenv\n",
    "* `-n` identifies the name of the virtualenv, in this case, `stats`\n",
    "* `python=3` tells conda that you want to install Python version 3 in this virtualenv\n",
    "\n",
    "**NOTE**: for other projects, you **can** use `python=2` or `python=3` and regardless which you choose, conda will default to the most recent version of Python within the version 2 OR version 3 family. If you need to select a specific minor version of python, use the following syntax:\n",
    "\n",
    "`python=3.2`\n",
    "\n",
    "When you execute the `conda create` command, `conda` prepares to install Python and any dependencies that Python relies upon. It will display output similar to the following. \n",
    "\n",
    "```bash\n",
    "my_macbook:my_folder chalmerlowe$ conda create -n stats python=3\n",
    "Fetching package metadata .......\n",
    "Solving package specifications: ..........\n",
    "\n",
    "Package plan for installation in environment /Users/chalmerlowe/miniconda3/envs/stats:\n",
    "\n",
    "The following packages will be downloaded:\n",
    "\n",
    "    package                    |            build\n",
    "    ---------------------------|-----------------\n",
    "    openssl-1.0.2k             |                1         3.0 MB\n",
    "    python-3.6.0               |                0        11.7 MB\n",
    "    setuptools-27.2.0          |           py36_0         523 KB\n",
    "    wheel-0.29.0               |           py36_0          87 KB\n",
    "    pip-9.0.1                  |           py36_1         1.7 MB\n",
    "    ------------------------------------------------------------\n",
    "                                            Total:        17.0 MB\n",
    "\n",
    "The following NEW packages will be INSTALLED:\n",
    "\n",
    "    openssl:    1.0.2k-1\n",
    "    pip:        9.0.1-py36_1\n",
    "    python:     3.6.0-0\n",
    "    readline:   6.2-2\n",
    "    setuptools: 27.2.0-py36_0\n",
    "    sqlite:     3.13.0-0\n",
    "    tk:         8.5.18-0\n",
    "    wheel:      0.29.0-py36_0\n",
    "    xz:         5.2.2-1\n",
    "    zlib:       1.2.8-3\n",
    "\n",
    "Proceed ([y]/n)?\n",
    "```\n",
    "\n",
    "To finish the creation of the virtualenv and install the software, press `y`.\n",
    "\n",
    "## Activating a virtualenv\n",
    "\n",
    "Once you have created a virtualenv, you will need to activate it. Activation has several side effects:\n",
    "\n",
    "* It temporarily changes your `$PATH` variable so calls to the `python` command (and similar commands) will look first in the virtual environment's `bin/` directory. \n",
    "* It temporarily changes your shell prompt to show which virtual environment you are using. Your prompt will likely look something like this, with the name of your virtual environment in parenthesis in front of the prompt:\n",
    "    * Mac/Linux: `(stats) chalmerlowe$`\n",
    "    * Windows: `(stats) C:\\>`\n",
    "\n",
    "To activate your virtual environment, run the appropriate command for your operating system:\n",
    "\n",
    "**Mac/Linux**\n",
    "\n",
    "```bash\n",
    "$ source activate stats\n",
    "```\n",
    "\n",
    "**Windows**\n",
    "\n",
    "```bat\n",
    "C:\\> activate stats\n",
    "```\n",
    "\n",
    "**Note:** If you are using **Power Shell**, `activate` won't work out of the box. Type `cmd` first to get a regular command prompt, *then* `activate stats`.\n",
    "\n",
    "### Adding software to your virtualenv \n",
    "\n",
    "To add more software to the virtualenv, you can use `conda` to install the software. The maintainers of conda provide access to many Python and non-Python libraries, but not all of them. If conda cannot install a particular library that you need, you can generally use `pip` or a similar package installation tool to install it instead (covering `pip` is outside the scope of this workshop).\n",
    "\n",
    "For example, to install IPython, you can use the following `conda` command:\n",
    "\n",
    "```\n",
    "conda install ipython\n",
    "```\n",
    "\n",
    "Conda will prepare to install IPython and any dependencies that IPython relies upon. It will display output similar to the following (truncated to save space).\n",
    "\n",
    "```bash\n",
    "Fetching package metadata .......\n",
    "Solving package specifications: ..........\n",
    "\n",
    "Package plan for installation in environment /Users/chalmerlowe/miniconda3:\n",
    "\n",
    "The following packages will be downloaded:\n",
    "\n",
    "    package                    |            build\n",
    "    ---------------------------|-----------------\n",
    "    conda-env-2.6.0            |                0          601 B\n",
    "    ...\n",
    "    ipython-5.3.0              |           py35_0        1021 KB\n",
    "    conda-4.3.14               |           py35_0         505 KB\n",
    "    ------------------------------------------------------------\n",
    "                                            Total:         3.8 MB\n",
    "\n",
    "The following NEW packages will be INSTALLED:\n",
    "\n",
    "    appnope:          0.1.0-py35_0\n",
    "    ...\n",
    "    wcwidth:          0.1.7-py35_0\n",
    "\n",
    "The following packages will be UPDATED:\n",
    "\n",
    "    conda:            4.1.11-py35_0 --> 4.3.14-py35_0\n",
    "    conda-env:        2.5.2-py35_0  --> 2.6.0-0\n",
    "    requests:         2.10.0-py35_0 --> 2.13.0-py35_0\n",
    "\n",
    "Proceed ([y]/n)?\n",
    "```\n",
    "\n",
    "To finish the installation of IPython and its dependencies, press `y`.\n",
    "\n",
    "### Multiple packages\n",
    "\n",
    "Multiple packages can be installed at the same time, by separating the package names with spaces:\n",
    "\n",
    "`conda install jupyterlab matplotlib numpy pandas scipy`\n",
    "\n",
    "**IF** there are special packages that you need to get from a specific repository channel (i.e. the conda-forge channel), you can designate a channel using the `-c` flag and the name of the channel (such as `conda-forge`) as shown here:\n",
    "\n",
    "`conda install -c conda-forge jupyterlab matplotlib numpy pandas scipy`\n",
    "\n",
    "### Leaving the virtualenv when you are done\n",
    "\n",
    "When you are done working in your virtualenv, you can deactivate it using the following command:\n",
    "\n",
    "**Mac/Linux**\n",
    "\n",
    "```bash\n",
    "(mytest) $ source deactivate\n",
    "$\n",
    "```\n",
    "\n",
    "**Windows**\n",
    "\n",
    "```bat\n",
    "(mytest) C:\\> deactivate\n",
    "C:\\>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Resources\n",
    "\n",
    "\n",
    "\n",
    "* [Using conda](http://conda.pydata.org/docs/using/index.html): A tutorial on how to use `conda`\n",
    "\n",
    "* [conda cheatsheet](https://conda.io/docs/_downloads/conda-cheatsheet.pdf): A cheatsheet of the most common `conda` commands\n",
    "\n",
    "* [conda myths and misconceptions](http://jakevdp.github.io/blog/2016/08/25/conda-myths-and-misconceptions/): Reasons why conda was created and how it differs from `pip`, `virtualenv`, etc.\n",
    "\n",
    "* [Python's `venv` and `virtualenv` can also create virtual environments.](http://stackoverflow.com/questions/41573587/what-is-the-difference-between-venv-pyvenv-pyenv-virtualenv-virtualenvwrappe)\n",
    "\n",
    "* [`pip` is Python's package manager.](https://en.wikipedia.org/wiki/Pip_(package_manager))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}