exercise-missing-values.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "42ea5692",
   "metadata": {
    "papermill": {
     "duration": 0.011008,
     "end_time": "2023-01-11T13:55:04.487527",
     "exception": false,
     "start_time": "2023-01-11T13:55:04.476519",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**This notebook is an exercise in the [Intermediate Machine Learning](https://www.kaggle.com/learn/intermediate-machine-learning) course.  You can reference the tutorial at [this link](https://www.kaggle.com/alexisbcook/missing-values).**\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5dbd950a",
   "metadata": {
    "papermill": {
     "duration": 0.007993,
     "end_time": "2023-01-11T13:55:04.504694",
     "exception": false,
     "start_time": "2023-01-11T13:55:04.496701",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Now it's your turn to test your new knowledge of **missing values** handling. You'll probably find it makes a big difference.\n",
    "\n",
    "# Setup\n",
    "\n",
    "The questions will give you feedback on your work. Run the following cell to set up the feedback system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ce231805",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-11T13:55:04.524147Z",
     "iopub.status.busy": "2023-01-11T13:55:04.523051Z",
     "iopub.status.idle": "2023-01-11T13:55:04.587188Z",
     "shell.execute_reply": "2023-01-11T13:55:04.584891Z"
    },
    "papermill": {
     "duration": 0.077592,
     "end_time": "2023-01-11T13:55:04.590484",
     "exception": false,
     "start_time": "2023-01-11T13:55:04.512892",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Setup Complete\n"
     ]
    }
   ],
   "source": [
    "# Set up code checking\n",
    "import os\n",
    "if not os.path.exists(\"../input/train.csv\"):\n",
    "    os.symlink(\"../input/home-data-for-ml-course/train.csv\", \"../input/train.csv\")  \n",
    "    os.symlink(\"../input/home-data-for-ml-course/test.csv\", \"../input/test.csv\") \n",
    "from learntools.core import binder\n",
    "binder.bind(globals())\n",
    "from learntools.ml_intermediate.ex2 import *\n",
    "print(\"Setup Complete\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c029f17",
   "metadata": {
    "papermill": {
     "duration": 0.009009,
     "end_time": "2023-01-11T13:55:04.608429",
     "exception": false,
     "start_time": "2023-01-11T13:55:04.599420",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "In this exercise, you will work with data from the [Housing Prices Competition for Kaggle Learn Users](https://www.kaggle.com/c/home-data-for-ml-course). \n",
    "\n",
    "![Ames Housing dataset image](https://i.imgur.com/lTJVG4e.png)\n",
    "\n",
    "Run the next code cell without changes to load the training and validation sets in `X_train`, `X_valid`, `y_train`, and `y_valid`.  The test set is loaded in `X_test`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4025d96d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-11T13:55:04.629871Z",
     "iopub.status.busy": "2023-01-11T13:55:04.629384Z",
     "iopub.status.idle": "2023-01-11T13:55:06.273985Z",
     "shell.execute_reply": "2023-01-11T13:55:06.272322Z"
    },
    "papermill": {
     "duration": 1.660577,
     "end_time": "2023-01-11T13:55:06.277954",
     "exception": false,
     "start_time": "2023-01-11T13:55:04.617377",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Read the data\n",
    "X_full = pd.read_csv('../input/train.csv', index_col='Id')\n",
    "X_test_full = pd.read_csv('../input/test.csv', index_col='Id')\n",
    "\n",
    "# Remove rows with missing target, separate target from predictors\n",
    "X_full.dropna(axis=0, subset=['SalePrice'], inplace=True)\n",
    "y = X_full.SalePrice\n",
    "X_full.drop(['SalePrice'], axis=1, inplace=True)\n",
    "\n",
    "# To keep things simple, we'll use only numerical predictors\n",
    "X = X_full.select_dtypes(exclude=['object'])\n",
    "X_test = X_test_full.select_dtypes(exclude=['object'])\n",
    "\n",
    "# Break off validation set from training data\n",
    "X_train, X_valid, y_train, y_valid = train_test_split(X, y, train_size=0.8, test_size=0.2,\n",
    "                                                      random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba31c2c6",
   "metadata": {
    "papermill": {
     "duration": 0.010974,
     "end_time": "2023-01-11T13:55:06.297891",
     "exception": false,
     "start_time": "2023-01-11T13:55:06.286917",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Use the next code cell to print the first five rows of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5449fd2d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-11T13:55:06.317872Z",
     "iopub.status.busy": "2023-01-11T13:55:06.317331Z",
     "iopub.status.idle": "2023-01-11T13:55:06.353399Z",
     "shell.execute_reply": "2023-01-11T13:55:06.351728Z"
    },
    "papermill": {
     "duration": 0.049371,
     "end_time": "2023-01-11T13:55:06.356065",
     "exception": false,
     "start_time": "2023-01-11T13:55:06.306694",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>OverallQual</th>\n",
       "      <th>OverallCond</th>\n",
       "      <th>YearBuilt</th>\n",
       "      <th>YearRemodAdd</th>\n",
       "      <th>MasVnrArea</th>\n",
       "      <th>BsmtFinSF1</th>\n",
       "      <th>BsmtFinSF2</th>\n",
       "      <th>...</th>\n",
       "      <th>GarageArea</th>\n",
       "      <th>WoodDeckSF</th>\n",
       "      <th>OpenPorchSF</th>\n",
       "      <th>EnclosedPorch</th>\n",
       "      <th>3SsnPorch</th>\n",
       "      <th>ScreenPorch</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>619</th>\n",
       "      <td>20</td>\n",
       "      <td>90.0</td>\n",
       "      <td>11694</td>\n",
       "      <td>9</td>\n",
       "      <td>5</td>\n",
       "      <td>2007</td>\n",
       "      <td>2007</td>\n",
       "      <td>452.0</td>\n",
       "      <td>48</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>774</td>\n",
       "      <td>0</td>\n",
       "      <td>108</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>260</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>2007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>871</th>\n",
       "      <td>20</td>\n",
       "      <td>60.0</td>\n",
       "      <td>6600</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>1962</td>\n",
       "      <td>1962</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>308</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>93</th>\n",
       "      <td>30</td>\n",
       "      <td>80.0</td>\n",
       "      <td>13360</td>\n",
       "      <td>5</td>\n",
       "      <td>7</td>\n",
       "      <td>1921</td>\n",
       "      <td>2006</td>\n",
       "      <td>0.0</td>\n",
       "      <td>713</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>432</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>44</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>818</th>\n",
       "      <td>20</td>\n",
       "      <td>NaN</td>\n",
       "      <td>13265</td>\n",
       "      <td>8</td>\n",
       "      <td>5</td>\n",
       "      <td>2002</td>\n",
       "      <td>2002</td>\n",
       "      <td>148.0</td>\n",
       "      <td>1218</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>857</td>\n",
       "      <td>150</td>\n",
       "      <td>59</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>2008</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>303</th>\n",
       "      <td>20</td>\n",
       "      <td>118.0</td>\n",
       "      <td>13704</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>2001</td>\n",
       "      <td>2002</td>\n",
       "      <td>150.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>843</td>\n",
       "      <td>468</td>\n",
       "      <td>81</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2006</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 36 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     MSSubClass  LotFrontage  LotArea  OverallQual  OverallCond  YearBuilt  \\\n",
       "Id                                                                           \n",
       "619          20         90.0    11694            9            5       2007   \n",
       "871          20         60.0     6600            5            5       1962   \n",
       "93           30         80.0    13360            5            7       1921   \n",
       "818          20          NaN    13265            8            5       2002   \n",
       "303          20        118.0    13704            7            5       2001   \n",
       "\n",
       "     YearRemodAdd  MasVnrArea  BsmtFinSF1  BsmtFinSF2  ...  GarageArea  \\\n",
       "Id                                                     ...               \n",
       "619          2007       452.0          48           0  ...         774   \n",
       "871          1962         0.0           0           0  ...         308   \n",
       "93           2006         0.0         713           0  ...         432   \n",
       "818          2002       148.0        1218           0  ...         857   \n",
       "303          2002       150.0           0           0  ...         843   \n",
       "\n",
       "     WoodDeckSF  OpenPorchSF  EnclosedPorch  3SsnPorch  ScreenPorch  PoolArea  \\\n",
       "Id                                                                              \n",
       "619           0          108              0          0          260         0   \n",
       "871           0            0              0          0            0         0   \n",
       "93            0            0             44          0            0         0   \n",
       "818         150           59              0          0            0         0   \n",
       "303         468           81              0          0            0         0   \n",
       "\n",
       "     MiscVal  MoSold  YrSold  \n",
       "Id                            \n",
       "619        0       7    2007  \n",
       "871        0       8    2009  \n",
       "93         0       8    2009  \n",
       "818        0       7    2008  \n",
       "303        0       1    2006  \n",
       "\n",
       "[5 rows x 36 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbb3288d",
   "metadata": {
    "papermill": {
     "duration": 0.008422,
     "end_time": "2023-01-11T13:55:06.373556",
     "exception": false,
     "start_time": "2023-01-11T13:55:06.365134",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "You can already see a few missing values in the first several rows.  In the next step, you'll obtain a more comprehensive understanding of the missing values in the dataset.\n",
    "\n",
    "# Step 1: Preliminary investigation\n",
    "\n",
    "Run the code cell below without changes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9cb7398c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-11T13:55:06.394757Z",
     "iopub.status.busy": "2023-01-11T13:55:06.393428Z",
     "iopub.status.idle": "2023-01-11T13:55:06.404852Z",
     "shell.execute_reply": "2023-01-11T13:55:06.403173Z"
    },
    "papermill": {
     "duration": 0.024337,
     "end_time": "2023-01-11T13:55:06.407423",
     "exception": false,
     "start_time": "2023-01-11T13:55:06.383086",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1168, 36)\n",
      "LotFrontage    212\n",
      "MasVnrArea       6\n",
      "GarageYrBlt     58\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# Shape of training data (num_rows, num_columns)\n",
    "print(X_train.shape)\n",
    "\n",
    "# Number of missing values in each column of training data\n",
    "missing_val_count_by_column = (X_train.isnull().sum())\n",
    "print(missing_val_count_by_column[missing_val_count_by_column > 0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcfee84d",
   "metadata": {
    "papermill": {
     "duration": 0.010519,
     "end_time": "2023-01-11T13:55:06.427508",
     "exception": false,
     "start_time": "2023-01-11T13:55:06.416989",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Part A\n",
    "\n",
    "Use the above output to answer the questions below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b6d305f5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-11T13:55:06.447963Z",
     "iopub.status.busy": "2023-01-11T13:55:06.447491Z",
     "iopub.status.idle": "2023-01-11T13:55:06.460250Z",
     "shell.execute_reply": "2023-01-11T13:55:06.458339Z"
    },
    "papermill": {
     "duration": 0.026227,
     "end_time": "2023-01-11T13:55:06.463353",
     "exception": false,
     "start_time": "2023-01-11T13:55:06.437126",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 1.0, \"interactionType\": 1, \"questionType\": 1, \"questionId\": \"1.1_InvestigateEquality\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Fill in the line below: How many rows are in the training data?\n",
    "num_rows = 1168\n",
    "\n",
    "# Fill in the line below: How many columns in the training data\n",
    "# have missing values?\n",
    "num_cols_with_missing = 3\n",
    "\n",
    "# Fill in the line below: How many missing entries are contained in \n",
    "# all of the training data?\n",
    "tot_missing = 276\n",
    "\n",
    "# Check your answers\n",
    "step_1.a.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b07ed521",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-11T13:55:06.485676Z",
     "iopub.status.busy": "2023-01-11T13:55:06.485166Z",
     "iopub.status.idle": "2023-01-11T13:55:06.491524Z",
     "shell.execute_reply": "2023-01-11T13:55:06.489807Z"
    },
    "papermill": {
     "duration": 0.022423,
     "end_time": "2023-01-11T13:55:06.494805",
     "exception": false,
     "start_time": "2023-01-11T13:55:06.472382",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#step_1.a.hint()\n",
    "#step_1.a.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6120dc36",
   "metadata": {
    "papermill": {
     "duration": 0.010309,
     "end_time": "2023-01-11T13:55:06.514878",
     "exception": false,
     "start_time": "2023-01-11T13:55:06.504569",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Part B\n",
    "Considering your answers above, what do you think is likely the best approach to dealing with the missing values?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "d0e35fa8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-11T13:55:06.537807Z",
     "iopub.status.busy": "2023-01-11T13:55:06.535855Z",
     "iopub.status.idle": "2023-01-11T13:55:06.547885Z",
     "shell.execute_reply": "2023-01-11T13:55:06.546672Z"
    },
    "papermill": {
     "duration": 0.026398,
     "end_time": "2023-01-11T13:55:06.550429",
     "exception": false,
     "start_time": "2023-01-11T13:55:06.524031",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 1.0, \"interactionType\": 1, \"questionType\": 4, \"questionId\": \"1.2_InvestigateThought\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct:</span> \n",
       "\n",
       "Since there are relatively few missing entries in the data (the column with the greatest percentage of missing values is missing less than 20% of its entries), we can expect that dropping columns is unlikely to yield good results.  This is because we'd be throwing away a lot of valuable data, and so imputation will likely perform better."
      ],
      "text/plain": [
       "Correct: \n",
       "\n",
       "Since there are relatively few missing entries in the data (the column with the greatest percentage of missing values is missing less than 20% of its entries), we can expect that dropping columns is unlikely to yield good results.  This is because we'd be throwing away a lot of valuable data, and so imputation will likely perform better."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Check your answer (Run this code cell to receive credit!)\n",
    "step_1.b.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a73aefae",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-11T13:55:06.570453Z",
     "iopub.status.busy": "2023-01-11T13:55:06.569388Z",
     "iopub.status.idle": "2023-01-11T13:55:06.576788Z",
     "shell.execute_reply": "2023-01-11T13:55:06.574958Z"
    },
    "papermill": {
     "duration": 0.021406,
     "end_time": "2023-01-11T13:55:06.580666",
     "exception": false,
     "start_time": "2023-01-11T13:55:06.559260",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "#step_1.b.hint()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "695e9007",
   "metadata": {
    "papermill": {
     "duration": 0.009654,
     "end_time": "2023-01-11T13:55:06.600481",
     "exception": false,
     "start_time": "2023-01-11T13:55:06.590827",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "To compare different approaches to dealing with missing values, you'll use the same `score_dataset()` function from the tutorial.  This function reports the [mean absolute error](https://en.wikipedia.org/wiki/Mean_absolute_error) (MAE) from a random forest model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "a4b49dfd",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-11T13:55:06.621340Z",
     "iopub.status.busy": "2023-01-11T13:55:06.620886Z",
     "iopub.status.idle": "2023-01-11T13:55:06.955574Z",
     "shell.execute_reply": "2023-01-11T13:55:06.954013Z"
    },
    "papermill": {
     "duration": 0.348472,
     "end_time": "2023-01-11T13:55:06.958693",
     "exception": false,
     "start_time": "2023-01-11T13:55:06.610221",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.metrics import mean_absolute_error\n",
    "\n",
    "# Function for comparing different approaches\n",
    "def score_dataset(X_train, X_valid, y_train, y_valid):\n",
    "    model = RandomForestRegressor(n_estimators=100, random_state=0)\n",
    "    model.fit(X_train, y_train)\n",
    "    preds = model.predict(X_valid)\n",
    "    return mean_absolute_error(y_valid, preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "983cd7c0",
   "metadata": {
    "papermill": {
     "duration": 0.009503,
     "end_time": "2023-01-11T13:55:06.978032",
     "exception": false,
     "start_time": "2023-01-11T13:55:06.968529",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Step 2: Drop columns with missing values\n",
    "\n",
    "In this step, you'll preprocess the data in `X_train` and `X_valid` to remove columns with missing values.  Set the preprocessed DataFrames to `reduced_X_train` and `reduced_X_valid`, respectively.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "0b757989",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-11T13:55:06.999818Z",
     "iopub.status.busy": "2023-01-11T13:55:06.999293Z",
     "iopub.status.idle": "2023-01-11T13:55:07.042861Z",
     "shell.execute_reply": "2023-01-11T13:55:07.041259Z"
    },
    "papermill": {
     "duration": 0.058241,
     "end_time": "2023-01-11T13:55:07.045967",
     "exception": false,
     "start_time": "2023-01-11T13:55:06.987726",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 1.0, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"2_DropMissing\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Fill in the line below: get names of columns with missing values\n",
    "cols = []\n",
    "for i in X.columns:\n",
    "    if X[i].isnull().any() == True:\n",
    "        cols.append(i)\n",
    "# Your code here\n",
    "\n",
    "# Fill in the lines below: drop columns in training and validation data\n",
    "reduced_X_train = X_train.drop(cols,axis=1)\n",
    "reduced_X_valid = X_valid.drop(cols,axis=1)\n",
    "\n",
    "# Check your answers\n",
    "step_2.check()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "222cfc4f",
   "metadata": {
    "papermill": {
     "duration": 0.010922,
     "end_time": "2023-01-11T13:55:07.067187",
     "exception": false,
     "start_time": "2023-01-11T13:55:07.056265",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Run the next code cell without changes to obtain the MAE for this approach."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "33571ef5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-11T13:55:07.091313Z",
     "iopub.status.busy": "2023-01-11T13:55:07.090859Z",
     "iopub.status.idle": "2023-01-11T13:55:08.337043Z",
     "shell.execute_reply": "2023-01-11T13:55:08.335265Z"
    },
    "papermill": {
     "duration": 1.262353,
     "end_time": "2023-01-11T13:55:08.340126",
     "exception": false,
     "start_time": "2023-01-11T13:55:07.077773",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAE (Drop columns with missing values):\n",
      "17837.82570776256\n"
     ]
    }
   ],
   "source": [
    "print(\"MAE (Drop columns with missing values):\")\n",
    "print(score_dataset(reduced_X_train, reduced_X_valid, y_train, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4539e679",
   "metadata": {
    "papermill": {
     "duration": 0.00963,
     "end_time": "2023-01-11T13:55:08.360565",
     "exception": false,
     "start_time": "2023-01-11T13:55:08.350935",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Step 3: Imputation\n",
    "\n",
    "### Part A\n",
    "\n",
    "Use the next code cell to impute missing values with the mean value along each column.  Set the preprocessed DataFrames to `imputed_X_train` and `imputed_X_valid`.  Make sure that the column names match those in `X_train` and `X_valid`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "5281b2fb",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-11T13:55:08.384629Z",
     "iopub.status.busy": "2023-01-11T13:55:08.384101Z",
     "iopub.status.idle": "2023-01-11T13:55:08.427503Z",
     "shell.execute_reply": "2023-01-11T13:55:08.426360Z"
    },
    "papermill": {
     "duration": 0.059022,
     "end_time": "2023-01-11T13:55:08.430486",
     "exception": false,
     "start_time": "2023-01-11T13:55:08.371464",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 1.0, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"3.1_ImputeCode\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.impute import SimpleImputer\n",
    "\n",
    "# Fill in the lines below: imputation\n",
    "my_imputer = SimpleImputer() # Your code here\n",
    "imputed_X_train = pd.DataFrame(my_imputer.fit_transform(X_train))\n",
    "imputed_X_valid = pd.DataFrame(my_imputer.transform(X_valid))\n",
    "\n",
    "# Fill in the lines below: imputation removed column names; put them back\n",
    "imputed_X_train.columns = X_train.columns\n",
    "imputed_X_valid.columns = X_valid.columns\n",
    "\n",
    "# Check your answers\n",
    "step_3.a.check()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9fe25e7",
   "metadata": {
    "papermill": {
     "duration": 0.011283,
     "end_time": "2023-01-11T13:55:08.452055",
     "exception": false,
     "start_time": "2023-01-11T13:55:08.440772",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Run the next code cell without changes to obtain the MAE for this approach."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "799a5335",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-11T13:55:08.474861Z",
     "iopub.status.busy": "2023-01-11T13:55:08.474393Z",
     "iopub.status.idle": "2023-01-11T13:55:09.873769Z",
     "shell.execute_reply": "2023-01-11T13:55:09.872473Z"
    },
    "papermill": {
     "duration": 1.413609,
     "end_time": "2023-01-11T13:55:09.876685",
     "exception": false,
     "start_time": "2023-01-11T13:55:08.463076",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAE (Imputation):\n",
      "18062.894611872147\n"
     ]
    }
   ],
   "source": [
    "print(\"MAE (Imputation):\")\n",
    "print(score_dataset(imputed_X_train, imputed_X_valid, y_train, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d810af2",
   "metadata": {
    "papermill": {
     "duration": 0.010573,
     "end_time": "2023-01-11T13:55:09.897916",
     "exception": false,
     "start_time": "2023-01-11T13:55:09.887343",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Part B\n",
    "\n",
    "Compare the MAE from each approach.  Does anything surprise you about the results?  Why do you think one approach performed better than the other?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "e11bbc48",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-11T13:55:09.922929Z",
     "iopub.status.busy": "2023-01-11T13:55:09.922458Z",
     "iopub.status.idle": "2023-01-11T13:55:09.933360Z",
     "shell.execute_reply": "2023-01-11T13:55:09.931702Z"
    },
    "papermill": {
     "duration": 0.026065,
     "end_time": "2023-01-11T13:55:09.936200",
     "exception": false,
     "start_time": "2023-01-11T13:55:09.910135",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 1.0, \"interactionType\": 1, \"questionType\": 4, \"questionId\": \"3.2_ImputeThought\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct:</span> \n",
       "\n",
       "Given that thre are so few missing values in the dataset, we'd expect imputation to perform better than dropping columns entirely.  However, we see that dropping columns performs slightly better!  While this can probably partially be attributed to noise in the dataset, another potential explanation is that the imputation method is not a great match to this dataset.  That is, maybe instead of filling in the mean value, it makes more sense to set every missing value to a value of 0, to fill in the most frequently encountered value, or to use some other method.  For instance, consider the `GarageYrBlt` column (which indicates the year that the garage was built).  It's likely that in some cases, a missing value could indicate a house that does not have a garage.  Does it make more sense to fill in the median value along each column in this case?  Or could we get better results by filling in the minimum value along each column?  It's not quite clear what's best in this case, but perhaps we can rule out some options immediately - for instance, setting missing values in this column to 0 is likely to yield horrible results!"
      ],
      "text/plain": [
       "Correct: \n",
       "\n",
       "Given that thre are so few missing values in the dataset, we'd expect imputation to perform better than dropping columns entirely.  However, we see that dropping columns performs slightly better!  While this can probably partially be attributed to noise in the dataset, another potential explanation is that the imputation method is not a great match to this dataset.  That is, maybe instead of filling in the mean value, it makes more sense to set every missing value to a value of 0, to fill in the most frequently encountered value, or to use some other method.  For instance, consider the `GarageYrBlt` column (which indicates the year that the garage was built).  It's likely that in some cases, a missing value could indicate a house that does not have a garage.  Does it make more sense to fill in the median value along each column in this case?  Or could we get better results by filling in the minimum value along each column?  It's not quite clear what's best in this case, but perhaps we can rule out some options immediately - for instance, setting missing values in this column to 0 is likely to yield horrible results!"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Check your answer (Run this code cell to receive credit!)\n",
    "step_3.b.check()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae0cb008",
   "metadata": {
    "papermill": {
     "duration": 0.01015,
     "end_time": "2023-01-11T13:55:09.957275",
     "exception": false,
     "start_time": "2023-01-11T13:55:09.947125",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Step 4: Generate test predictions\n",
    "\n",
    "In this final step, you'll use any approach of your choosing to deal with missing values.  Once you've preprocessed the training and validation features, you'll train and evaluate a random forest model.  Then, you'll preprocess the test data before generating predictions that can be submitted to the competition!\n",
    "\n",
    "### Part A\n",
    "\n",
    "Use the next code cell to preprocess the training and validation data.  Set the preprocessed DataFrames to `final_X_train` and `final_X_valid`.  **You can use any approach of your choosing here!**  in order for this step to be marked as correct, you need only ensure:\n",
    "- the preprocessed DataFrames have the same number of columns,\n",
    "- the preprocessed DataFrames have no missing values, \n",
    "- `final_X_train` and `y_train` have the same number of rows, and\n",
    "- `final_X_valid` and `y_valid` have the same number of rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "4531271e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-11T13:55:09.981046Z",
     "iopub.status.busy": "2023-01-11T13:55:09.980566Z",
     "iopub.status.idle": "2023-01-11T13:55:10.021132Z",
     "shell.execute_reply": "2023-01-11T13:55:10.019147Z"
    },
    "papermill": {
     "duration": 0.056695,
     "end_time": "2023-01-11T13:55:10.024321",
     "exception": false,
     "start_time": "2023-01-11T13:55:09.967626",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 1.0, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"4.1_PredsCodeA\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Preprocessed training and validation features\n",
    "final_imputer = SimpleImputer(strategy='median')\n",
    "final_X_train = pd.DataFrame(final_imputer.fit_transform(X_train))\n",
    "final_X_valid = pd.DataFrame(final_imputer.transform(X_valid))\n",
    "\n",
    "# Imputation removed column names; put them back\n",
    "final_X_train.columns = X_train.columns\n",
    "final_X_valid.columns = X_valid.columns\n",
    "\n",
    "# Check your answers\n",
    "step_4.a.check()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed71eeb8",
   "metadata": {
    "papermill": {
     "duration": 0.010245,
     "end_time": "2023-01-11T13:55:10.046346",
     "exception": false,
     "start_time": "2023-01-11T13:55:10.036101",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Run the next code cell to train and evaluate a random forest model.  (*Note that we don't use the `score_dataset()` function above, because we will soon use the trained model to generate test predictions!*)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "a812f0ed",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-11T13:55:10.069208Z",
     "iopub.status.busy": "2023-01-11T13:55:10.068722Z",
     "iopub.status.idle": "2023-01-11T13:55:11.436697Z",
     "shell.execute_reply": "2023-01-11T13:55:11.435054Z"
    },
    "papermill": {
     "duration": 1.383228,
     "end_time": "2023-01-11T13:55:11.439892",
     "exception": false,
     "start_time": "2023-01-11T13:55:10.056664",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAE (Your approach):\n",
      "17791.59899543379\n"
     ]
    }
   ],
   "source": [
    "# Define and fit model\n",
    "model = RandomForestRegressor(n_estimators=100, random_state=0)\n",
    "model.fit(final_X_train, y_train)\n",
    "\n",
    "# Get validation predictions and MAE\n",
    "preds_valid = model.predict(final_X_valid)\n",
    "print(\"MAE (Your approach):\")\n",
    "print(mean_absolute_error(y_valid, preds_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "418211dc",
   "metadata": {
    "papermill": {
     "duration": 0.010654,
     "end_time": "2023-01-11T13:55:11.461724",
     "exception": false,
     "start_time": "2023-01-11T13:55:11.451070",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Part B\n",
    "\n",
    "Use the next code cell to preprocess your test data.  Make sure that you use a method that agrees with how you preprocessed the training and validation data, and set the preprocessed test features to `final_X_test`.\n",
    "\n",
    "Then, use the preprocessed test features and the trained model to generate test predictions in `preds_test`.\n",
    "\n",
    "In order for this step to be marked correct, you need only ensure:\n",
    "- the preprocessed test DataFrame has no missing values, and\n",
    "- `final_X_test` has the same number of rows as `X_test`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "0965a8f9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-11T13:55:11.485872Z",
     "iopub.status.busy": "2023-01-11T13:55:11.485420Z",
     "iopub.status.idle": "2023-01-11T13:55:11.556689Z",
     "shell.execute_reply": "2023-01-11T13:55:11.555202Z"
    },
    "papermill": {
     "duration": 0.086363,
     "end_time": "2023-01-11T13:55:11.559568",
     "exception": false,
     "start_time": "2023-01-11T13:55:11.473205",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/sklearn/base.py:451: UserWarning: X does not have valid feature names, but RandomForestRegressor was fitted with feature names\n",
      "  \"X does not have valid feature names, but\"\n"
     ]
    },
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 1.0, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"4.2_PredsCodeB\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Fill in the line below: preprocess test data\n",
    "\n",
    "final_X_test = pd.DataFrame(final_imputer.transform(X_test))\n",
    "\n",
    "# Fill in the line below: get test predictions\n",
    "preds_test = model.predict(final_X_test)\n",
    "\n",
    "# Check your answers\n",
    "step_4.b.check()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ae1c721",
   "metadata": {
    "papermill": {
     "duration": 0.012435,
     "end_time": "2023-01-11T13:55:11.584673",
     "exception": false,
     "start_time": "2023-01-11T13:55:11.572238",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Run the next code cell without changes to save your results to a CSV file that can be submitted directly to the competition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "396e3e2e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-11T13:55:11.610597Z",
     "iopub.status.busy": "2023-01-11T13:55:11.610127Z",
     "iopub.status.idle": "2023-01-11T13:55:11.623533Z",
     "shell.execute_reply": "2023-01-11T13:55:11.621946Z"
    },
    "papermill": {
     "duration": 0.030105,
     "end_time": "2023-01-11T13:55:11.627046",
     "exception": false,
     "start_time": "2023-01-11T13:55:11.596941",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Save test predictions to file\n",
    "output = pd.DataFrame({'Id': X_test.index,\n",
    "                       'SalePrice': preds_test})\n",
    "output.to_csv('submission.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8dd16769",
   "metadata": {
    "papermill": {
     "duration": 0.011766,
     "end_time": "2023-01-11T13:55:11.651269",
     "exception": false,
     "start_time": "2023-01-11T13:55:11.639503",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "---\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "*Have questions or comments? Visit the [course discussion forum](https://www.kaggle.com/learn/intermediate-machine-learning/discussion) to chat with other learners.*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 19.239398,
   "end_time": "2023-01-11T13:55:12.588884",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2023-01-11T13:54:53.349486",
   "version": "2.3.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}