find_tabular_errors/find_tabular_errors.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zwfWqPeA1zX0"
   },
   "source": [
    "# Handling Mislabeled Tabular Data to Improve Your XGBoost Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "onqP2ipLDtK7"
   },
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/cleanlab/examples/blob/master/find_tabular_errors/find_tabular_errors.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IjVDHIWaDtK8"
   },
   "source": [
    "This article highlights data-centric AI techniques using [cleanlab](https://github.com/cleanlab/cleanlab) to improve the accuracy of an XGBoost classifier (reducing prediction errors by 70% on the noisy dataset considered here!). These techniques optimize the dataset itself rather than altering the model’s architecture or hyperparameters. As a result, it is possible to achieve further improvements in accuracy by fine-tuning the model with the newly enhanced data. Enhancements to the dataset are model-agnostic and therefore are transferable to other modeling and analytical endeavors, as opposed to being specific to a particular type of model.\n",
    "\n",
    "At a high level we will:\n",
    "- Establish a baseline XGBoost model accuracy trained on the original data.\n",
    "- Use cleanlab's `find_label_issues()` to highlight hundreds of mislabeled data points. \n",
    "- Remove the data with automatically-flagged label issues from the dataset, and then retrain the exact same XGBoost model. **This simple step reduces the error in model predictions by 36%!** The raw difference in accuracy values between the two XGBoost models is **8%**.\n",
    "- Introduce a **no-code** solution to efficiently fix the label errors in the dataset which **reduces the error in model predictions by 70%** from the baseline, identical XGBoost model!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "je7P55z4RwX_"
   },
   "source": [
    "## Setup and Data Processing\n",
    "\n",
    "Let’s take a look at our student grades tabular dataset. The data includes three exam scores (numerical features), a written note (categorical feature with missing values), and a (noisy) letter grade (categorical label). Our aim is to train a model to classify the grade for each student based on the other features, but 20% of the grade labels in this dataset are actually incorrect.\n",
    "\n",
    "We have access to the true letter grade each student should’ve received, which we use for evaluating both the underlying accuracy of model predictions and how well cleanlab detects which data are mislabeled. These true grades are only reserved for model evaluation and are manually validated, gold-standard labels. They are not present in any of the training procedures. We utilize a 75/25 split for out train/test data.\n",
    "\n",
    "In your noisily-labeled datasets, there will typically be no such ground truth, and therefore addressing label issues is even more important to facilitate proper model evaluation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please install the dependencies specified in this [requirements.txt](https://github.com/cleanlab/examples/blob/master/find_tabular_errors/requirements.txt) file before running the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "nQVmMBQOS43j"
   },
   "outputs": [],
   "source": [
    "from cleanlab.filter import find_label_issues\n",
    "from xgboost import XGBClassifier\n",
    "from sklearn import preprocessing\n",
    "from sklearn.model_selection import cross_val_predict\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import accuracy_score\n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"https://s.cleanlab.ai/student-grades-demo.csv\")\n",
    "df_c = df.copy()\n",
    "\n",
    "# Transform letter grades and notes to categorical numbers.\n",
    "# Necessary for XGBoost and cleanlab.\n",
    "df['letter_grade'] = preprocessing.LabelEncoder().fit_transform(df['letter_grade'])\n",
    "df['noisy_letter_grade'] = preprocessing.LabelEncoder().fit_transform(df['noisy_letter_grade'])\n",
    "df['notes'] = preprocessing.LabelEncoder().fit_transform(df[\"notes\"])\n",
    "df['notes'] = df['notes'].astype('category')\n",
    "\n",
    "# Split data for evaluation and set test data.\n",
    "df_train, df_test = train_test_split(df, random_state=0)\n",
    "df_train.reset_index(drop=True, inplace=True)\n",
    "df_test.reset_index(drop=True, inplace=True)\n",
    "test_data = df_test.drop(['stud_ID', 'letter_grade', 'noisy_letter_grade'], axis=1)\n",
    "test_labels = df_test['letter_grade']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dVH_iciASD9F"
   },
   "source": [
    "# Train and Evaluate XGBoost Classifier\n",
    "\n",
    "Now that we’ve seen what can be achieved with cleanlab, let’s take a look at how we get there.\n",
    "\n",
    "For our model of choice, we will use XGBoost, an implementation of gradient-boosting decision trees (GBDT), which are commonly used with tabular data. If our tabular data consisted solely of numerical and boolean values, we could potentially utilize a simpler model such as a nearest-neighbor or logistic regression. However, our data includes a notes column, which we will treat as a categorical feature. Fortunately, XGBoost (>v1.6) is able to handle mixed data types (numerical and categorical) by setting the `enable_categorical` parameter to `true`, thereby simplifying the modeling process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "gCS19IqJsQUL",
    "outputId": "321eba76-57c2-45ee-ab43-7eb7d623efa5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy with original data: 79.2%\n"
     ]
    }
   ],
   "source": [
    "# Train model on noisy labels.\n",
    "train_data = df_train.drop(['stud_ID', 'letter_grade', 'noisy_letter_grade'], axis=1)\n",
    "train_labels = df_train['noisy_letter_grade']\n",
    "\n",
    "# XGBoost(experimental) supports categorical data.\n",
    "# Here we use default hyperparameters for simplicity.\n",
    "model = XGBClassifier(tree_method=\"hist\", enable_categorical=True)\n",
    "model.fit(train_data, train_labels)\n",
    "\n",
    "# Evaluate model on test split with ground truth labels.\n",
    "preds = model.predict(test_data)\n",
    "acc_original = accuracy_score(preds, test_labels)\n",
    "print(f\"Accuracy with original data: {round(acc_original*100,1)}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lTQ_iB-JSWUl"
   },
   "source": [
    "Using the default hyperparameters, our baseline XGBoost model demonstrates an accuracy of 79.2% when trained on the noisy labels and predicting the test set. It appears that the presence of 20% label noise is significantly disrupting the model’s ability to accurately predict the labels on such a trivial task."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "klDe2ag8SZ2T"
   },
   "source": [
    "# Find Label Issues\n",
    "\n",
    "In order to use cleanlab, we need to obtain **out-of-sample** predicted probabilities for all of our training data in order to provide the `find_label_issues()` method with the necessary input. Getting the predicted probabilities can be achieved through the use of our `XGBClassifier` model with cross-validation, which can be implemented easily using the `cross_val_predict` function from scikit-learn.\n",
    "\n",
    "In just a few lines of code, we get a list of possible label issues! A few of the top results are shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 359
    },
    "id": "SfJ83uP-Xski",
    "outputId": "f3550ef1-5f43-464e-961f-54347080f966"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/pandas/util/_decorators.py:311: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  return func(*args, **kwargs)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-5f12789d-a614-4d7c-862b-3077d58f9e65\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>stud_ID</th>\n",
       "      <th>exam_1</th>\n",
       "      <th>exam_2</th>\n",
       "      <th>exam_3</th>\n",
       "      <th>notes</th>\n",
       "      <th>letter_grade</th>\n",
       "      <th>noisy_letter_grade</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>765</th>\n",
       "      <td>75ce98</td>\n",
       "      <td>91</td>\n",
       "      <td>89</td>\n",
       "      <td>81</td>\n",
       "      <td>NaN</td>\n",
       "      <td>B</td>\n",
       "      <td>F</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>744</th>\n",
       "      <td>3d0fdf</td>\n",
       "      <td>90</td>\n",
       "      <td>74</td>\n",
       "      <td>95</td>\n",
       "      <td>great participation +10</td>\n",
       "      <td>A</td>\n",
       "      <td>F</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>637</th>\n",
       "      <td>77c9c5</td>\n",
       "      <td>0</td>\n",
       "      <td>79</td>\n",
       "      <td>65</td>\n",
       "      <td>cheated on exam, gets 0pts</td>\n",
       "      <td>F</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>404</th>\n",
       "      <td>bb13f4</td>\n",
       "      <td>65</td>\n",
       "      <td>95</td>\n",
       "      <td>68</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>217</th>\n",
       "      <td>3c4cbb</td>\n",
       "      <td>94</td>\n",
       "      <td>62</td>\n",
       "      <td>66</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-5f12789d-a614-4d7c-862b-3077d58f9e65')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-5f12789d-a614-4d7c-862b-3077d58f9e65 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-5f12789d-a614-4d7c-862b-3077d58f9e65');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "    stud_ID  exam_1  exam_2  exam_3                       notes letter_grade  \\\n",
       "765  75ce98      91      89      81                         NaN            B   \n",
       "744  3d0fdf      90      74      95     great participation +10            A   \n",
       "637  77c9c5       0      79      65  cheated on exam, gets 0pts            F   \n",
       "404  bb13f4      65      95      68                         NaN            C   \n",
       "217  3c4cbb      94      62      66                         NaN            C   \n",
       "\n",
       "    noisy_letter_grade  \n",
       "765                  F  \n",
       "744                  F  \n",
       "637                  A  \n",
       "404                  A  \n",
       "217                  C  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get predicted probabilities through cross validation.\n",
    "model = XGBClassifier(tree_method=\"hist\", enable_categorical=True)\n",
    "pred_probs = cross_val_predict(model, train_data, train_labels, method='predict_proba')\n",
    "\n",
    "# Returns list of indices of label issues, sorted by self_confidence.\n",
    "issue_idx = find_label_issues(train_labels, pred_probs, return_indices_ranked_by='self_confidence')\n",
    "\n",
    "# Filter original data to show students with grade issues.\n",
    "issue_stud_id = df_train.iloc[issue_idx].stud_ID.values.tolist()\n",
    "issues_df = df_c[df_c['stud_ID'].isin(issue_stud_id)]\n",
    "issues_df.sort_values(by=\"stud_ID\", key=lambda column: column.map(lambda e: issue_stud_id.index(e)), inplace=True)\n",
    "\n",
    "# Show a few good examples.\n",
    "issues_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "326EP61IEuJC"
   },
   "source": [
    "Let’s take a look at a few of the label issues automatically identified in our dataset. Take a look at row 1, where the student got grades of 91, 89, and 81, which should result in a ‘B’ yet was accidentally labeled as an ‘F’. In row 2, the student had great participation resulting in an addition of 10 points to the overall average, receiving exam grades of 90, 74, and 95 (averages to 86.3, overall 96.3 with the bonus), which should result in a ‘A’ yet was accidentally labeled as an ‘F’.\n",
    "\n",
    "**Note: `find_label_issues` is able to determine that the given label is incorrect, without ever seeing the ground truth label `letter_grade`.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PrvJHkPzSq6Q"
   },
   "source": [
    "# How'd We Do?\n",
    "\n",
    "Let's go a step further and see how cleanlab did at automatically identifying which data points are mislabeled. If we take the intersection of the label errors identified by cleanlab and the true label errors, we see that cleanlab was able to identify 83% of the label errors correctly (based on predictions from a model that is only 79% accurate). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "9O2a6urWc1DA",
    "outputId": "b5c5add0-36b9-4e03-b3c4-4a7b22c438b0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Percentage of errors found: 82.9%\n"
     ]
    }
   ],
   "source": [
    "# Computing percentage of true errors identified. \n",
    "true_error_idx = df_train[df_train.letter_grade != df_train.noisy_letter_grade].index.values\n",
    "cl_acc = len(set(true_error_idx).intersection(set(issue_idx)))/len(true_error_idx)\n",
    "print(f\"Percentage of errors found: {round(cl_acc*100,1)}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YzxXoDOqSzn-"
   },
   "source": [
    "# Retraining for a More Robust Model\n",
    "\n",
    "Now that we have the indices of potential label errors let’s remove them from our data, retrain our model, and see what performance improvement we can gain.\n",
    "\n",
    "Keep in mind our baseline model from above, trained on the original data using the `noisy_letter_grade` as the prediction label, achieved an accuracy of 79%.\n",
    "\n",
    "Let’s use a very simple method to handle these label errors and just drop them entirely from the data and retrain our exact same `XGBClassifier`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "FsQFmy7xgSUa",
    "outputId": "eef2e102-8bae-442f-d6a9-a15a905caa8b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy with original data: 79.2%\n",
      "Accuracy with errors found by cleanlab removed: 86.9%\n",
      "Reduction in error: 36.7%\n"
     ]
    }
   ],
   "source": [
    "# Remove the label errors found by cleanlab.\n",
    "train_data_cl = df_train.drop(issue_idx)\n",
    "train_labels_cl = train_data_cl['noisy_letter_grade']\n",
    "train_data_cl = train_data_cl.drop(['stud_ID', 'letter_grade', 'noisy_letter_grade'], axis=1)\n",
    "\n",
    "# Train a more robust classifier with less erroneous data.\n",
    "model = XGBClassifier(tree_method=\"hist\", enable_categorical=True)\n",
    "model.fit(train_data_cl, train_labels_cl)\n",
    "\n",
    "# Evaluate model on test split with ground truth labels.\n",
    "preds = model.predict(test_data)\n",
    "acc_clean = accuracy_score(preds, test_labels)\n",
    "print(f\"Accuracy with original data: {round(acc_original*100, 1)}%\")\n",
    "print(f\"Accuracy with errors found by cleanlab removed: {round(acc_clean*100, 1)}%\")\n",
    "\n",
    "# Compute reduction in error.\n",
    "err = ((1-acc_original)-(1-acc_clean))/(1-acc_original)\n",
    "print(f\"Reduction in error: {round(err*100,1)}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9J9clVf1UzQZ"
   },
   "source": [
    "After removing the suspected label issues, our model's new accuracy is now 87%, which means we **reduced the error-rate of the model by 36%**. \n",
    "\n",
    "**Note: throughout this entire process, we never changed any code related to model architecture/hyperparameters, training, or data preprocessing! This improvement is strictly coming from increasing the quality of our data which leaves room for additional optimizations on the modeling side.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "znk8DligxAPs"
   },
   "source": [
    "# Fixing the Label Errors\n",
    "\n",
    "Instead of just dropping the potential label issues, the smarter (yet more complex) way to increase our data quality would be to correct the label issues by hand. This simultaneously removes a noisy data point and adds an accurate one, but making such corrections manually is cumbersome.\n",
    "\n",
    "[Cleanlab Studio](https://cleanlab.ai/studio) provides a user-friendly interface to make these changes without writing a single line of code. Simply upload your dataset and Studio computes everything we just did for you, so you can spend more time fixing the issues instead of just finding them.\n",
    "\n",
    "Here, we use the auto-fix feature on this dataset and replace the Studio-found label issues with the automatically-suggested label. From data upload to data export, the whole process took only 5 minutes without having to know any ML.  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "qRFpuKhHw-BR",
    "outputId": "1c46c5ad-0d53-453a-8e27-e9be4fa3beb3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy with original data: 79.2%\n",
      "Accuracy with errors found by cleanlab removed: 86.9%\n",
      "Accuracy with errors found by Studio fixed: 93.6%\n",
      "\n",
      "Reduction in error using cleanlab opensource over baseline: 36.7%\n",
      "Reduction in error using cleanlab Studio over opensource: 51.6%\n",
      "Reduction in error using Cleanlab Studio over baseline: 69.4%\n"
     ]
    }
   ],
   "source": [
    "# Get the export produced by Cleanlab Studio and split to match training subset.\n",
    "clean_df = pd.read_csv(\"https://s.cleanlab.ai/student-grades-demo-studio-export.csv\")\n",
    "train_students = df_train.stud_ID.values\n",
    "clean_df = clean_df[clean_df['stud_ID'].isin(train_students)]\n",
    "\n",
    "# Same pre-processing as above.\n",
    "clean_df['cleanlab_suggested_label'] = preprocessing.LabelEncoder().fit_transform(clean_df['cleanlab_suggested_label'])\n",
    "clean_df['notes'] = preprocessing.LabelEncoder().fit_transform(clean_df[\"notes\"])\n",
    "clean_df['notes'] = clean_df['notes'].astype('category')\n",
    "\n",
    "# Train a more robust classifier with less erroneous data.\n",
    "clean_labels = clean_df['cleanlab_suggested_label']\n",
    "clean_data = clean_df[['exam_1','exam_2','exam_3','notes']]\n",
    "model = XGBClassifier(tree_method=\"hist\", enable_categorical=True)\n",
    "model.fit(clean_data, clean_labels)\n",
    "\n",
    "# Evaluate model on test split with ground truth labels.\n",
    "preds = model.predict(test_data)\n",
    "acc_studio = accuracy_score(preds, test_labels)\n",
    "print(f\"Accuracy with original data: {round(acc_original*100, 1)}%\")\n",
    "print(f\"Accuracy with errors found by cleanlab removed: {round(acc_clean*100, 1)}%\")\n",
    "print(f\"Accuracy with errors found by Studio fixed: {round(acc_studio*100, 1)}%\")\n",
    "print()\n",
    "\n",
    "# Compute reductions in error.\n",
    "clos_err = ((1-acc_original)-(1-acc_clean))/(1-acc_original)\n",
    "studio_err = ((1-acc_clean)-(1-acc_studio))/(1-acc_clean)\n",
    "tot_err = ((1-acc_original)-(1-acc_studio))/(1-acc_original)\n",
    "print(f\"Reduction in error using cleanlab opensource over baseline: {round(clos_err*100,1)}%\")\n",
    "print(f\"Reduction in error using cleanlab Studio over opensource: {round(studio_err*100,1)}%\")\n",
    "print(f\"Reduction in error using Cleanlab Studio over baseline: {round(tot_err*100,1)}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-W-Lo82SVp7I"
   },
   "source": [
    "# Conclusion\n",
    "\n",
    "Cleanlab is an incredibly powerful and efficient tool for identifying and addressing label errors in your data that can be used to improve any ML model (not just XGBoost) for most types of data (not just tabular, but also images, text, audio, etc). By implementing just a few lines of open-source code, cleanlab can automatically detect and help you prioritize many potential issues within your data. With this insight, you'll be able to improve the quality of your data and ultimately achieve better model performance.\n",
    "\n",
    "For the student grades dataset, we found that **simply dropping identified label errors and retraining the model resulted in a 36% reduction in prediction error** on our classification problem (with accuracy improving from 79% to 87%). \n",
    "\n",
    "Going one step further, we used Cleanlab Studio to automatically **fix the incorrect labels,resulting in a 70% reduction in prediction error** (with accuracy improving from 79% to 94%).\n",
    "\n",
    "By using open-source libraries for data-centric AI like [cleanlab](https://github.com/cleanlab/cleanlab) to ensure the integrity of your data, you can mitigate costly labeling errors and boost the performance of your models.\n",
    "\n",
    "\n",
    "# Next Steps\n",
    "\n",
    "We would like to extend a special thanks to all of our open-source contributors. Your support and engagement have played a crucial role in the development and success of cleanlab. If you are interested in becoming a contributor to the cleanlab project and helping us build the standard open-source library for data-centric AI, please visit our [GitHub page](https://github.com/cleanlab/cleanlab) and [contributing guide](https://github.com/cleanlab/cleanlab/blob/master/CONTRIBUTING.md).\n",
    "\n",
    "If you interested in using cleanlab to improve your data-centric techniques and ML tasks, our comprehensive [tutorials](https://docs.cleanlab.ai/) provide a simple and efficient way to get started. In just 5 minutes, you can learn how to apply cleanlab to a variety of data types (text, tabular, image, audio, etc) and ML tasks (classification, entity recognition, image/document tagging, etc).\n",
    "\n",
    "We would love to connect with you, too!\n",
    "- Join our [Cleanlab Community Slack](https://cleanlab.ai/slack/)\n",
    "- Follow us on [LinkedIn](https://www.linkedin.com/company/cleanlab/)\n",
    "- Follow us on [Twitter](https://twitter.com/CleanlabAI)\n",
    "\n",
    "Bonus: Learn how cleanlab can also help improve training data in [Kaggle](https://www.kaggle.com/code/ulytkch/cleanlab-data-centric-ai-example-0-7703-python/notebook) competitions.\n"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}