exercise-pipelines.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8220658f",
   "metadata": {
    "papermill": {
     "duration": 0.005107,
     "end_time": "2023-01-24T20:44:25.851101",
     "exception": false,
     "start_time": "2023-01-24T20:44:25.845994",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**This notebook is an exercise in the [Intermediate Machine Learning](https://www.kaggle.com/learn/intermediate-machine-learning) course.  You can reference the tutorial at [this link](https://www.kaggle.com/alexisbcook/pipelines).**\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cd06b9d",
   "metadata": {
    "papermill": {
     "duration": 0.003701,
     "end_time": "2023-01-24T20:44:25.858922",
     "exception": false,
     "start_time": "2023-01-24T20:44:25.855221",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "In this exercise, you will use **pipelines** to improve the efficiency of your machine learning code.\n",
    "\n",
    "# Setup\n",
    "\n",
    "The questions below will give you feedback on your work. Run the following cell to set up the feedback system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "137e47e1",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-24T20:44:25.869465Z",
     "iopub.status.busy": "2023-01-24T20:44:25.868822Z",
     "iopub.status.idle": "2023-01-24T20:44:26.900644Z",
     "shell.execute_reply": "2023-01-24T20:44:26.899274Z"
    },
    "papermill": {
     "duration": 1.041746,
     "end_time": "2023-01-24T20:44:26.904601",
     "exception": false,
     "start_time": "2023-01-24T20:44:25.862855",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Setup Complete\n"
     ]
    }
   ],
   "source": [
    "# Set up code checking\n",
    "import os\n",
    "if not os.path.exists(\"../input/train.csv\"):\n",
    "    os.symlink(\"../input/home-data-for-ml-course/train.csv\", \"../input/train.csv\")  \n",
    "    os.symlink(\"../input/home-data-for-ml-course/test.csv\", \"../input/test.csv\") \n",
    "from learntools.core import binder\n",
    "binder.bind(globals())\n",
    "from learntools.ml_intermediate.ex4 import *\n",
    "print(\"Setup Complete\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "069ec050",
   "metadata": {
    "papermill": {
     "duration": 0.004001,
     "end_time": "2023-01-24T20:44:26.914489",
     "exception": false,
     "start_time": "2023-01-24T20:44:26.910488",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "You will work with data from the [Housing Prices Competition for Kaggle Learn Users](https://www.kaggle.com/c/home-data-for-ml-course). \n",
    "\n",
    "![Ames Housing dataset image](https://i.imgur.com/lTJVG4e.png)\n",
    "\n",
    "Run the next code cell without changes to load the training and validation sets in `X_train`, `X_valid`, `y_train`, and `y_valid`.  The test set is loaded in `X_test`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9075628e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-24T20:44:26.924843Z",
     "iopub.status.busy": "2023-01-24T20:44:26.924222Z",
     "iopub.status.idle": "2023-01-24T20:44:27.120590Z",
     "shell.execute_reply": "2023-01-24T20:44:27.119411Z"
    },
    "papermill": {
     "duration": 0.204637,
     "end_time": "2023-01-24T20:44:27.123279",
     "exception": false,
     "start_time": "2023-01-24T20:44:26.918642",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Read the data\n",
    "X_full = pd.read_csv('../input/train.csv', index_col='Id')\n",
    "X_test_full = pd.read_csv('../input/test.csv', index_col='Id')\n",
    "\n",
    "# Remove rows with missing target, separate target from predictors\n",
    "X_full.dropna(axis=0, subset=['SalePrice'], inplace=True)\n",
    "y = X_full.SalePrice\n",
    "X_full.drop(['SalePrice'], axis=1, inplace=True)\n",
    "\n",
    "# Break off validation set from training data\n",
    "X_train_full, X_valid_full, y_train, y_valid = train_test_split(X_full, y, \n",
    "                                                                train_size=0.8, test_size=0.2,\n",
    "                                                                random_state=0)\n",
    "\n",
    "# \"Cardinality\" means the number of unique values in a column\n",
    "# Select categorical columns with relatively low cardinality (convenient but arbitrary)\n",
    "categorical_cols = [cname for cname in X_train_full.columns if\n",
    "                    X_train_full[cname].nunique() < 10 and \n",
    "                    X_train_full[cname].dtype == \"object\"]\n",
    "\n",
    "# Select numerical columns\n",
    "numerical_cols = [cname for cname in X_train_full.columns if \n",
    "                X_train_full[cname].dtype in ['int64', 'float64']]\n",
    "\n",
    "# Keep selected columns only\n",
    "my_cols = categorical_cols + numerical_cols\n",
    "X_train = X_train_full[my_cols].copy()\n",
    "X_valid = X_valid_full[my_cols].copy()\n",
    "X_test = X_test_full[my_cols].copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "76e85bff",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-24T20:44:27.133405Z",
     "iopub.status.busy": "2023-01-24T20:44:27.133013Z",
     "iopub.status.idle": "2023-01-24T20:44:27.161374Z",
     "shell.execute_reply": "2023-01-24T20:44:27.160317Z"
    },
    "papermill": {
     "duration": 0.035906,
     "end_time": "2023-01-24T20:44:27.163431",
     "exception": false,
     "start_time": "2023-01-24T20:44:27.127525",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>Street</th>\n",
       "      <th>Alley</th>\n",
       "      <th>LotShape</th>\n",
       "      <th>LandContour</th>\n",
       "      <th>Utilities</th>\n",
       "      <th>LotConfig</th>\n",
       "      <th>LandSlope</th>\n",
       "      <th>Condition1</th>\n",
       "      <th>Condition2</th>\n",
       "      <th>...</th>\n",
       "      <th>GarageArea</th>\n",
       "      <th>WoodDeckSF</th>\n",
       "      <th>OpenPorchSF</th>\n",
       "      <th>EnclosedPorch</th>\n",
       "      <th>3SsnPorch</th>\n",
       "      <th>ScreenPorch</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>619</th>\n",
       "      <td>RL</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Inside</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>Norm</td>\n",
       "      <td>Norm</td>\n",
       "      <td>...</td>\n",
       "      <td>774</td>\n",
       "      <td>0</td>\n",
       "      <td>108</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>260</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>2007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>871</th>\n",
       "      <td>RL</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Inside</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>PosN</td>\n",
       "      <td>Norm</td>\n",
       "      <td>...</td>\n",
       "      <td>308</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>93</th>\n",
       "      <td>RL</td>\n",
       "      <td>Pave</td>\n",
       "      <td>Grvl</td>\n",
       "      <td>IR1</td>\n",
       "      <td>HLS</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Inside</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>Norm</td>\n",
       "      <td>Norm</td>\n",
       "      <td>...</td>\n",
       "      <td>432</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>44</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>818</th>\n",
       "      <td>RL</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>CulDSac</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>Norm</td>\n",
       "      <td>Norm</td>\n",
       "      <td>...</td>\n",
       "      <td>857</td>\n",
       "      <td>150</td>\n",
       "      <td>59</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>2008</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>303</th>\n",
       "      <td>RL</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Corner</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>Norm</td>\n",
       "      <td>Norm</td>\n",
       "      <td>...</td>\n",
       "      <td>843</td>\n",
       "      <td>468</td>\n",
       "      <td>81</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2006</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 76 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    MSZoning Street Alley LotShape LandContour Utilities LotConfig LandSlope  \\\n",
       "Id                                                                             \n",
       "619       RL   Pave   NaN      Reg         Lvl    AllPub    Inside       Gtl   \n",
       "871       RL   Pave   NaN      Reg         Lvl    AllPub    Inside       Gtl   \n",
       "93        RL   Pave  Grvl      IR1         HLS    AllPub    Inside       Gtl   \n",
       "818       RL   Pave   NaN      IR1         Lvl    AllPub   CulDSac       Gtl   \n",
       "303       RL   Pave   NaN      IR1         Lvl    AllPub    Corner       Gtl   \n",
       "\n",
       "    Condition1 Condition2  ... GarageArea WoodDeckSF OpenPorchSF  \\\n",
       "Id                         ...                                     \n",
       "619       Norm       Norm  ...        774          0         108   \n",
       "871       PosN       Norm  ...        308          0           0   \n",
       "93        Norm       Norm  ...        432          0           0   \n",
       "818       Norm       Norm  ...        857        150          59   \n",
       "303       Norm       Norm  ...        843        468          81   \n",
       "\n",
       "    EnclosedPorch 3SsnPorch ScreenPorch PoolArea MiscVal MoSold YrSold  \n",
       "Id                                                                      \n",
       "619             0         0         260        0       0      7   2007  \n",
       "871             0         0           0        0       0      8   2009  \n",
       "93             44         0           0        0       0      8   2009  \n",
       "818             0         0           0        0       0      7   2008  \n",
       "303             0         0           0        0       0      1   2006  \n",
       "\n",
       "[5 rows x 76 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf3cba9f",
   "metadata": {
    "papermill": {
     "duration": 0.004248,
     "end_time": "2023-01-24T20:44:27.172267",
     "exception": false,
     "start_time": "2023-01-24T20:44:27.168019",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "The next code cell uses code from the tutorial to preprocess the data and train a model.  Run this code without changes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f4aba350",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-24T20:44:27.183147Z",
     "iopub.status.busy": "2023-01-24T20:44:27.182716Z",
     "iopub.status.idle": "2023-01-24T20:44:29.786050Z",
     "shell.execute_reply": "2023-01-24T20:44:29.784933Z"
    },
    "papermill": {
     "duration": 2.611808,
     "end_time": "2023-01-24T20:44:29.788462",
     "exception": false,
     "start_time": "2023-01-24T20:44:27.176654",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAE: 17861.780102739725\n"
     ]
    }
   ],
   "source": [
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.metrics import mean_absolute_error\n",
    "\n",
    "# Preprocessing for numerical data\n",
    "numerical_transformer = SimpleImputer(strategy='constant')\n",
    "\n",
    "# Preprocessing for categorical data\n",
    "categorical_transformer = Pipeline(steps=[\n",
    "    ('imputer', SimpleImputer(strategy='most_frequent')),\n",
    "    ('onehot', OneHotEncoder(handle_unknown='ignore'))\n",
    "])\n",
    "\n",
    "# Bundle preprocessing for numerical and categorical data\n",
    "preprocessor = ColumnTransformer(\n",
    "    transformers=[\n",
    "        ('num', numerical_transformer, numerical_cols),\n",
    "        ('cat', categorical_transformer, categorical_cols)\n",
    "    ])\n",
    "\n",
    "# Define model\n",
    "model = RandomForestRegressor(n_estimators=100, random_state=0)\n",
    "\n",
    "# Bundle preprocessing and modeling code in a pipeline\n",
    "clf = Pipeline(steps=[('preprocessor', preprocessor),\n",
    "                      ('model', model)\n",
    "                     ])\n",
    "\n",
    "# Preprocessing of training data, fit model \n",
    "clf.fit(X_train, y_train)\n",
    "\n",
    "# Preprocessing of validation data, get predictions\n",
    "preds = clf.predict(X_valid)\n",
    "\n",
    "print('MAE:', mean_absolute_error(y_valid, preds))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac11e0af",
   "metadata": {
    "papermill": {
     "duration": 0.004122,
     "end_time": "2023-01-24T20:44:29.797199",
     "exception": false,
     "start_time": "2023-01-24T20:44:29.793077",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "The code yields a value around 17862 for the mean absolute error (MAE).  In the next step, you will amend the code to do better.\n",
    "\n",
    "# Step 1: Improve the performance\n",
    "\n",
    "### Part A\n",
    "\n",
    "Now, it's your turn!  In the code cell below, define your own preprocessing steps and random forest model.  Fill in values for the following variables:\n",
    "- `numerical_transformer`\n",
    "- `categorical_transformer`\n",
    "- `model`\n",
    "\n",
    "To pass this part of the exercise, you need only define valid preprocessing steps and a random forest model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "291a5fed",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-24T20:44:29.807946Z",
     "iopub.status.busy": "2023-01-24T20:44:29.806923Z",
     "iopub.status.idle": "2023-01-24T20:44:29.818144Z",
     "shell.execute_reply": "2023-01-24T20:44:29.817048Z"
    },
    "papermill": {
     "duration": 0.018996,
     "end_time": "2023-01-24T20:44:29.820399",
     "exception": false,
     "start_time": "2023-01-24T20:44:29.801403",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 1.0, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"1.1_YourTurnModel\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Preprocessing for numerical data\n",
    "numerical_transformer = SimpleImputer(strategy='mean') # Your code here\n",
    "\n",
    "# Preprocessing for categorical data\n",
    "categorical_transformer = Pipeline(steps=[\n",
    "    ('imputer', SimpleImputer(strategy='most_frequent')),\n",
    "    ('onehot', OneHotEncoder(handle_unknown='ignore'))\n",
    "]) # Your code here\n",
    "\n",
    "# Bundle preprocessing for numerical and categorical data\n",
    "preprocessor = ColumnTransformer(\n",
    "    transformers=[\n",
    "        ('num', numerical_transformer, numerical_cols),\n",
    "        ('cat', categorical_transformer, categorical_cols)\n",
    "    ])\n",
    "\n",
    "# Define model\n",
    "model = RandomForestRegressor(n_estimators=110, random_state=0) # Your code here\n",
    "\n",
    "# Check your answer\n",
    "step_1.a.check()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b0b5b63",
   "metadata": {
    "papermill": {
     "duration": 0.004603,
     "end_time": "2023-01-24T20:44:29.829996",
     "exception": false,
     "start_time": "2023-01-24T20:44:29.825393",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Part B\n",
    "\n",
    "Run the code cell below without changes.\n",
    "\n",
    "To pass this step, you need to have defined a pipeline in **Part A** that achieves lower MAE than the code above.  You're encouraged to take your time here and try out many different approaches, to see how low you can get the MAE!  (_If your code does not pass, please amend the preprocessing steps and model in Part A._)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "9e2ac5d3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-24T20:44:29.841452Z",
     "iopub.status.busy": "2023-01-24T20:44:29.841088Z",
     "iopub.status.idle": "2023-01-24T20:44:32.438862Z",
     "shell.execute_reply": "2023-01-24T20:44:32.437635Z"
    },
    "papermill": {
     "duration": 2.606394,
     "end_time": "2023-01-24T20:44:32.441213",
     "exception": false,
     "start_time": "2023-01-24T20:44:29.834819",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAE: 17547.670236612703\n"
     ]
    },
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 1.0, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"1.2_YourTurnPredict\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Bundle preprocessing and modeling code in a pipeline\n",
    "my_pipeline = Pipeline(steps=[('preprocessor', preprocessor),\n",
    "                              ('model', model)\n",
    "                             ])\n",
    "\n",
    "# Preprocessing of training data, fit model \n",
    "my_pipeline.fit(X_train, y_train)\n",
    "\n",
    "# Preprocessing of validation data, get predictions\n",
    "preds = my_pipeline.predict(X_valid)\n",
    "\n",
    "# Evaluate the model\n",
    "score = mean_absolute_error(y_valid, preds)\n",
    "print('MAE:', score)\n",
    "\n",
    "# Check your answer\n",
    "step_1.b.check()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5b9cd73",
   "metadata": {
    "papermill": {
     "duration": 0.004783,
     "end_time": "2023-01-24T20:44:32.451083",
     "exception": false,
     "start_time": "2023-01-24T20:44:32.446300",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Step 2: Generate test predictions\n",
    "\n",
    "Now, you'll use your trained model to generate predictions with the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "5db7dabf",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-24T20:44:32.462442Z",
     "iopub.status.busy": "2023-01-24T20:44:32.462062Z",
     "iopub.status.idle": "2023-01-24T20:44:32.552455Z",
     "shell.execute_reply": "2023-01-24T20:44:32.551252Z"
    },
    "papermill": {
     "duration": 0.098938,
     "end_time": "2023-01-24T20:44:32.554816",
     "exception": false,
     "start_time": "2023-01-24T20:44:32.455878",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 1.0, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"2_TestPreds\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Preprocessing of test data, fit model\n",
    "preds_test = my_pipeline.predict(X_test_full) # Your code here\n",
    "\n",
    "# Check your answer\n",
    "step_2.check()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ecb505b",
   "metadata": {
    "papermill": {
     "duration": 0.005225,
     "end_time": "2023-01-24T20:44:32.565791",
     "exception": false,
     "start_time": "2023-01-24T20:44:32.560566",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Run the next code cell without changes to save your results to a CSV file that can be submitted directly to the competition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "391b2f9b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-24T20:44:32.579684Z",
     "iopub.status.busy": "2023-01-24T20:44:32.578660Z",
     "iopub.status.idle": "2023-01-24T20:44:32.590871Z",
     "shell.execute_reply": "2023-01-24T20:44:32.589757Z"
    },
    "papermill": {
     "duration": 0.021365,
     "end_time": "2023-01-24T20:44:32.593180",
     "exception": false,
     "start_time": "2023-01-24T20:44:32.571815",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Save test predictions to file\n",
    "output = pd.DataFrame({'Id': X_test.index,\n",
    "                       'SalePrice': preds_test})\n",
    "output.to_csv('submission.csv', index=False)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 15.908896,
   "end_time": "2023-01-24T20:44:33.319526",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2023-01-24T20:44:17.410630",
   "version": "2.3.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}