notebooks/parameter_tuning_randomized_search.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hyperparameter tuning by randomized-search\n",
    "\n",
    "In the previous notebook, we showed how to use a grid-search approach to\n",
    "search for the best hyperparameters maximizing the generalization performance\n",
    "of a predictive model.\n",
    "\n",
    "However, a grid-search approach has limitations. It does not scale well when\n",
    "the number of parameters to tune increases. Also, the grid imposes a\n",
    "regularity during the search which might miss better parameter\n",
    "values between two consecutive values on the grid.\n",
    "\n",
    "In this notebook, we present a different method to tune hyperparameters called\n",
    "randomized search."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Our predictive model\n",
    "\n",
    "Let us reload the dataset as we did previously:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "adult_census = pd.read_csv(\"../datasets/adult-census.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We extract the column containing the target."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_name = \"class\"\n",
    "target = adult_census[target_name]\n",
    "target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We drop from our data the target and the `\"education-num\"` column which\n",
    "duplicates the information with `\"education\"` columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = adult_census.drop(columns=[target_name, \"education-num\"])\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the dataset is loaded, we split it into a training and testing sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "data_train, data_test, target_train, target_test = train_test_split(\n",
    "    data, target, random_state=42\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create the same predictive pipeline as done for the grid-search section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.preprocessing import OrdinalEncoder\n",
    "from sklearn.compose import make_column_selector as selector\n",
    "\n",
    "categorical_columns_selector = selector(dtype_include=object)\n",
    "categorical_columns = categorical_columns_selector(data)\n",
    "\n",
    "categorical_preprocessor = OrdinalEncoder(\n",
    "    handle_unknown=\"use_encoded_value\", unknown_value=-1\n",
    ")\n",
    "preprocessor = ColumnTransformer(\n",
    "    [(\"cat_preprocessor\", categorical_preprocessor, categorical_columns)],\n",
    "    remainder=\"passthrough\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import HistGradientBoostingClassifier\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "model = Pipeline(\n",
    "    [\n",
    "        (\"preprocessor\", preprocessor),\n",
    "        (\n",
    "            \"classifier\",\n",
    "            HistGradientBoostingClassifier(random_state=42, max_leaf_nodes=4),\n",
    "        ),\n",
    "    ]\n",
    ")\n",
    "\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tuning using a randomized-search\n",
    "\n",
    "With the `GridSearchCV` estimator, the parameters need to be specified\n",
    "explicitly. We already mentioned that exploring a large number of values for\n",
    "different parameters quickly becomes untractable.\n",
    "\n",
    "Instead, we can randomly generate the parameter candidates. Indeed, such\n",
    "approach avoids the regularity of the grid. Hence, adding more evaluations can\n",
    "increase the resolution in each direction. This is the case in the frequent\n",
    "situation where the choice of some hyperparameters is not very important, as\n",
    "for the hyperparameter 2 in the figure below.\n",
    "\n",
    "![Randomized vs grid search](../figures/grid_vs_random_search.svg)\n",
    "\n",
    "Indeed, the number of evaluation points needs to be divided across the two\n",
    "different hyperparameters. With a grid, the danger is that the region of good\n",
    "hyperparameters may fall between lines of the grid. In the figure such region\n",
    "is aligned with the grid given that hyperparameter 2 has a weak influence.\n",
    "Rather, stochastic search samples the hyperparameter 1 independently from the\n",
    "hyperparameter 2 and find the optimal region.\n",
    "\n",
    "The `RandomizedSearchCV` class allows for such stochastic search. It is used\n",
    "similarly to the `GridSearchCV` but the sampling distributions need to be\n",
    "specified instead of the parameter values. For instance, we can draw\n",
    "candidates using a log-uniform distribution because the parameters we are\n",
    "interested in take positive values with a natural log scaling (.1 is as close\n",
    "to 1 as 10 is).\n",
    "\n",
    "<div class=\"admonition note alert alert-info\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Note</p>\n",
    "<p class=\"last\">Random search (with <tt class=\"docutils literal\">RandomizedSearchCV</tt>) is typically beneficial compared to\n",
    "grid search (with <tt class=\"docutils literal\">GridSearchCV</tt>) to optimize 3 or more hyperparameters.</p>\n",
    "</div>\n",
    "\n",
    "We now optimize 3 other parameters in addition to the ones we optimized in\n",
    "the notebook presenting the `GridSearchCV`:\n",
    "\n",
    "* `l2_regularization`: it corresponds to the strength of the regularization;\n",
    "* `min_samples_leaf`: it corresponds to the minimum number of samples required\n",
    "  in a leaf;\n",
    "* `max_bins`: it corresponds to the maximum number of bins to construct the\n",
    "  histograms.\n",
    "\n",
    "We recall the meaning of the 2 remaining parameters:\n",
    "\n",
    "* `learning_rate`: it corresponds to the speed at which the gradient-boosting\n",
    "  corrects the residuals at each boosting iteration;\n",
    "* `max_leaf_nodes`: it corresponds to the maximum number of leaves for each\n",
    "  tree in the ensemble.\n",
    "\n",
    "<div class=\"admonition note alert alert-info\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Note</p>\n",
    "<p class=\"last\"><tt class=\"docutils literal\">scipy.stats.loguniform</tt> can be used to generate floating numbers. To generate\n",
    "random values for integer-valued parameters (e.g. <tt class=\"docutils literal\">min_samples_leaf</tt>) we can\n",
    "adapt it as follows:</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import loguniform\n",
    "\n",
    "\n",
    "class loguniform_int:\n",
    "    \"\"\"Integer valued version of the log-uniform distribution\"\"\"\n",
    "\n",
    "    def __init__(self, a, b):\n",
    "        self._distribution = loguniform(a, b)\n",
    "\n",
    "    def rvs(self, *args, **kwargs):\n",
    "        \"\"\"Random variable sample\"\"\"\n",
    "        return self._distribution.rvs(*args, **kwargs).astype(int)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can define the randomized search using the different distributions.\n",
    "Executing 10 iterations of 5-fold cross-validation for random parametrizations\n",
    "of this model on this dataset can take from 10 seconds to several minutes,\n",
    "depending on the speed of the host computer and the number of available\n",
    "processors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "\n",
    "param_distributions = {\n",
    "    \"classifier__l2_regularization\": loguniform(1e-6, 1e3),\n",
    "    \"classifier__learning_rate\": loguniform(0.001, 10),\n",
    "    \"classifier__max_leaf_nodes\": loguniform_int(2, 256),\n",
    "    \"classifier__min_samples_leaf\": loguniform_int(1, 100),\n",
    "    \"classifier__max_bins\": loguniform_int(2, 255),\n",
    "}\n",
    "\n",
    "model_random_search = RandomizedSearchCV(\n",
    "    model,\n",
    "    param_distributions=param_distributions,\n",
    "    n_iter=10,\n",
    "    cv=5,\n",
    "    verbose=1,\n",
    ")\n",
    "model_random_search.fit(data_train, target_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we can compute the accuracy score on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy = model_random_search.score(data_test, target_test)\n",
    "\n",
    "print(f\"The test accuracy score of the best model is {accuracy:.2f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pprint import pprint\n",
    "\n",
    "print(\"The best parameters are:\")\n",
    "pprint(model_random_search.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lines_to_next_cell": 2
   },
   "source": [
    "We can inspect the results using the attributes `cv_results` as we did\n",
    "previously."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the parameter names\n",
    "column_results = [f\"param_{name}\" for name in param_distributions.keys()]\n",
    "column_results += [\"mean_test_score\", \"std_test_score\", \"rank_test_score\"]\n",
    "\n",
    "cv_results = pd.DataFrame(model_random_search.cv_results_)\n",
    "cv_results = cv_results[column_results].sort_values(\n",
    "    \"mean_test_score\", ascending=False\n",
    ")\n",
    "\n",
    "\n",
    "def shorten_param(param_name):\n",
    "    if \"__\" in param_name:\n",
    "        return param_name.rsplit(\"__\", 1)[1]\n",
    "    return param_name\n",
    "\n",
    "\n",
    "cv_results = cv_results.rename(shorten_param, axis=1)\n",
    "cv_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Keep in mind that tuning is limited by the number of different combinations of\n",
    "parameters that are scored by the randomized search. In fact, there might be\n",
    "other sets of parameters leading to similar or better generalization\n",
    "performances but that were not tested in the search. In practice, a randomized\n",
    "hyperparameter search is usually run with a large number of iterations. In\n",
    "order to avoid the computation cost and still make a decent analysis, we load\n",
    "the results obtained from a similar search with 500 iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# model_random_search = RandomizedSearchCV(\n",
    "#     model, param_distributions=param_distributions, n_iter=500,\n",
    "#     n_jobs=2, cv=5)\n",
    "# model_random_search.fit(data_train, target_train)\n",
    "# cv_results =  pd.DataFrame(model_random_search.cv_results_)\n",
    "# cv_results.to_csv(\"../figures/randomized_search_results.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv_results = pd.read_csv(\n",
    "    \"../figures/randomized_search_results.csv\", index_col=0\n",
    ")\n",
    "\n",
    "(\n",
    "    cv_results[column_results]\n",
    "    .rename(shorten_param, axis=1)\n",
    "    .sort_values(\"mean_test_score\", ascending=False)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case the top performing models have test scores with a high overlap\n",
    "between each other, meaning that indeed, the set of parameters leading to the\n",
    "best generalization performance is not unique."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "In this notebook, we saw how a randomized search offers a valuable alternative\n",
    "to grid-search when the number of hyperparameters to tune is more than two. It\n",
    "also alleviates the regularity imposed by the grid that might be problematic\n",
    "sometimes.\n",
    "\n",
    "In the following, we will see how to use interactive plotting tools to explore\n",
    "the results of large hyperparameter search sessions and gain some insights on\n",
    "range of parameter values that lead to the highest performing models and how\n",
    "different hyperparameter are coupled or not."
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "main_language": "python"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}