transformer_sklearn/transformer_sklearn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d8bd96c9",
   "metadata": {},
   "source": [
    "# Training Transformer Models with Scikit-learn"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a60a726b",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/cleanlab/examples/blob/master/transformer_sklearn/transformer_sklearn.ipynb)     "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b13d8a2",
   "metadata": {},
   "source": [
    "Have you ever wanted to use handy scikit-learn functionalities with your neural networks, but could not because TensorFlow models are not compatible with the scikit-learn API? Here we introduce `KerasWrapperModel`, a one-line wrapper for TensorFlow/Keras models that enables you to use TensorFlow models within scikit-learn workflows that include features like Pipeline, GridSearch and more.\n",
    "\n",
    "To demonstrate `KerasWrapperModel`, we will train a classifier (fine-tuning a pretrained Bert model) to classify positive vs. negative text reviews of products via the steps below:\n",
    "\n",
    "- Tokenize our text data to be suitable for a pretrained Bert Transformer model from HuggingFace. \n",
    "- Define a (pretrained) Transformer network in Keras code and then swap out one line (`keras.Model` -> `KerasWrapperModel`) to make this model compatible with scikit-learn.\n",
    "- Conduct a grid search to find the best parameters for the model using sklearn's `GridSearchCV`\n",
    "- Using the optimal parameters found, train the classifier to classify the reviews\n",
    "- Use cleanlab's `CleanLearning` to train a more robust version of the same model, which again just involves one extra line of code. This allows you to automatically remove label issues in the dataset, and then retrain the same classifier to get improved predictions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56d44364",
   "metadata": {},
   "source": [
    "Please install the dependencies specified in this [requirements.txt](https://github.com/cleanlab/examples/blob/master/transformer_sklearn/requirements.txt) file before running the notebook. For better efficieny, please also consider running this notebook on GPU.\n",
    "\n",
    "Note that this notebook will only work with `tensorflow<2.11`. If using `tensorflow>=2.11`, please replace `tf.keras.optimizers` with `tf.keras.optimizers.legacy`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "cccf56fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ['TF_CPP_MIN_LOG_LEVEL'] = \"3\" \n",
    "os.environ['TOKENIZERS_PARALLELISM'] = \"false\"\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.base import BaseEstimator\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "import tensorflow as tf\n",
    "from transformers import AutoTokenizer, TFAutoModel\n",
    "from transformers import logging\n",
    "\n",
    "from cleanlab.models.keras import KerasWrapperModel\n",
    "from cleanlab.classification import CleanLearning\n",
    "\n",
    "logging.set_verbosity(40)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c89da37",
   "metadata": {},
   "source": [
    "## Loading and Preprocessing Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5bb18f5",
   "metadata": {},
   "source": [
    "Let's take a look at the Amazon Reviews text dataset we are using for this demonstration. Each example in our dataset is a magazine review obtained from Amazon that have been classified into two categories: positive (1) and negative (0). \n",
    "\n",
    "Here is a sample label and review from the train set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "623a78a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data = pd.read_csv(\"https://s.cleanlab.ai/amazon_reviews/train.csv\")\n",
    "test_data = pd.read_csv(\"https://s.cleanlab.ai/amazon_reviews/test.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "94f57cf4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example Label: 1\n",
      "Example Text: Excellent product! I love reading through the magazine and learning about the cool new products out there and the cool programs!\n"
     ]
    }
   ],
   "source": [
    "i = 0\n",
    "print(f\"Example Label: {train_data.iloc[i]['label']}\")\n",
    "print(f\"Example Text: {train_data.iloc[i]['review_text']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73867a2f",
   "metadata": {},
   "source": [
    "Before we train our classifier, we need to transform our text data into a format suitable as an input for a neural network. Here, we tokenize the text to be suitable for a pretrained Bert transformer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1e06f61",
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_NAME = \"bert-base-uncased\"\n",
    "tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)\n",
    "\n",
    "train_input = tokenizer(\n",
    "    train_data[\"review_text\"].to_list(),\n",
    "    padding=\"max_length\",\n",
    "    truncation=True,\n",
    "    max_length=30,\n",
    "    return_tensors=\"tf\",\n",
    ")\n",
    "\n",
    "test_input = tokenizer(\n",
    "    test_data[\"review_text\"].to_list(),\n",
    "    padding=\"max_length\",\n",
    "    truncation=True,\n",
    "    max_length=30,\n",
    "    return_tensors=\"tf\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e9ce3d1",
   "metadata": {},
   "source": [
    "The pretrained bert tokenizer outputs a dictionary containing input IDs, token type IDs and attention masks for each example, each containing separate pieces of information about the text. The input IDs map each token to a specific ID specified by the pretrained bert model, the token type IDs are used to differentiate between different token types in question answering tasks (and are not relevate for our current classification task), where the attention mask indicates which tokens are contextual tokens vs padding tokens. \n",
    "\n",
    "For our current task, the input IDs already contain all the information needed to represent the text data, hence we will extract the input IDs from the dictionary returned by the tokenizer, and convert out input data into numpy arrays that are suitable for inputs into scikit-learn functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "61623836",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_input_ids = np.array(train_input[\"input_ids\"])\n",
    "train_labels = np.array(train_data['label'])\n",
    "\n",
    "test_input_ids = np.array(test_input[\"input_ids\"])\n",
    "test_labels = np.array(test_data['label'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e6d6eed",
   "metadata": {},
   "source": [
    "## Define Neural Network Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45eb107d",
   "metadata": {},
   "source": [
    "The next step is to define the Keras function that builds the Transformer model we would normally use to classify the Amazon reviews. \n",
    "\n",
    "Here we are fine-tuning a pretrained Bert Transformer for classification, which requires the `input_id`, `token_type_ids` and `attention_mask` as inputs. Our model only takes the `input_ids` as an input however we can internally obtain the information required to construct the `token_type_ids` and `attention_mask` arrays to pass into the bert model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "50153876",
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_model(model_name:str, max_len:int, n_classes:int):\n",
    "    # define input ids, token type ids and attention mask\n",
    "    input_ids = tf.keras.layers.Input(shape=(max_len,), dtype='int32', name='input_ids')\n",
    "    token_type_ids = tf.keras.layers.Lambda(lambda x: x * 0, name='token_type_ids')(input_ids)\n",
    "    attention_mask = tf.keras.layers.Lambda(lambda x: tf.cast(x != 0, tf.int32), name=\"attention_mask\")(input_ids)\n",
    "\n",
    "    # get bert main layer and add it to the NN, passing in inputs\n",
    "    bert_layer = TFAutoModel.from_pretrained(model_name)\n",
    "    layer = bert_layer(input_ids=input_ids, token_type_ids=token_type_ids, attention_mask=attention_mask)[1]\n",
    "    output_layer = tf.keras.layers.Dense(n_classes, activation='softmax')(layer)\n",
    "\n",
    "    # model instance\n",
    "    model = tf.keras.Model(inputs=[input_ids], outputs=output_layer)\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01accb3d",
   "metadata": {},
   "source": [
    "Ordinarily you would instantiate this model using `keras.Model` with the above `build_model` function, but here we simply replace this with `KerasWrapperModel` instead. The resulting `model` object is now a Keras model that is scikit-learn compatible!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "8a27ba49",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = KerasWrapperModel(\n",
    "    model=build_model,\n",
    "    model_kwargs={\n",
    "        \"model_name\": MODEL_NAME,\n",
    "        \"max_len\": 30,\n",
    "        \"n_classes\": 2,\n",
    "    },\n",
    "    compile_kwargs= {\n",
    "      \"optimizer\":tf.keras.optimizers.Adam(2e-5),\n",
    "      \"loss\":tf.keras.losses.SparseCategoricalCrossentropy(),\n",
    "      \"metrics\":[\"accuracy\"],\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "940fe1b3",
   "metadata": {},
   "source": [
    "We can check out the summary of our wrapped Keras neural network by calling the `summary()` method, the same way we would for a regular Keras model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a3e28c2d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "__________________________________________________________________________________________________\n",
      " Layer (type)                   Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      " input_ids (InputLayer)         [(None, 30)]         0           []                               \n",
      "                                                                                                  \n",
      " attention_mask (Lambda)        (None, 30)           0           ['input_ids[0][0]']              \n",
      "                                                                                                  \n",
      " token_type_ids (Lambda)        (None, 30)           0           ['input_ids[0][0]']              \n",
      "                                                                                                  \n",
      " tf_bert_model (TFBertModel)    TFBaseModelOutputWi  109482240   ['input_ids[0][0]',              \n",
      "                                thPoolingAndCrossAt               'attention_mask[0][0]',         \n",
      "                                tentions(last_hidde               'token_type_ids[0][0]']         \n",
      "                                n_state=(None, 30,                                                \n",
      "                                768),                                                             \n",
      "                                 pooler_output=(Non                                               \n",
      "                                e, 768),                                                          \n",
      "                                 past_key_values=No                                               \n",
      "                                ne, hidden_states=N                                               \n",
      "                                one, attentions=Non                                               \n",
      "                                e, cross_attentions                                               \n",
      "                                =None)                                                            \n",
      "                                                                                                  \n",
      " dense (Dense)                  (None, 2)            1538        ['tf_bert_model[0][1]']          \n",
      "                                                                                                  \n",
      "==================================================================================================\n",
      "Total params: 109,483,778\n",
      "Trainable params: 109,483,778\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8e9cb51",
   "metadata": {},
   "source": [
    "## Conduct GridSearch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c23adb0d",
   "metadata": {},
   "source": [
    "Now that our Keras model is sklearn-compatible, we can use `GridSearchCV` to find the best hyperparameters for this classifier. The code to implement the grid search is the same as you would for any other sklearn model, and is shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f30aad23",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "params = {\n",
    "    'batch_size': [32, 64],\n",
    "    'epochs': [5, 10],\n",
    "}\n",
    "\n",
    "gs = GridSearchCV(model, params, refit=False, cv=3, verbose=2, scoring='accuracy', error_score='raise')\n",
    "gs.fit(train_input_ids, train_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "8eeedad6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "best score: 0.834, best params: {'batch_size': 64, 'epochs': 5}\n"
     ]
    }
   ],
   "source": [
    "print(\"best score: {:.3f}, best params: {}\".format(gs.best_score_, gs.best_params_))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e5081a6",
   "metadata": {},
   "source": [
    "After the grid search, we obtain the best parameters for the model and can use these parameters to train the final model. We then measure the accuracy of the model using a held-out test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5b56fb0",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.fit(train_input_ids, train_labels, epochs=5, batch_size=64)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab79369a",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_preds = model.predict(test_input_ids)\n",
    "model_accuracy = np.mean(model_preds == test_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "e23c8f3d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Base model accuracy = 0.917\n"
     ]
    }
   ],
   "source": [
    "print(f\"Base model accuracy = {model_accuracy}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "840e9c34",
   "metadata": {},
   "source": [
    "## Train a more robust model using CleanLearning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fa16593",
   "metadata": {},
   "source": [
    "We've already obtain a good model by hyperparameter tuning above. However there are other data-centric techniques we can use to improve the classifier's performance that do not alter the model's architecture or hyperparameters in any way. \n",
    "\n",
    "Instead, we would like to focus on the data quality of our training data. Most datasets contain label errors, which will negatively impact the ability of the model to learn and hence its performance. Here, we demonstrate how to use `CleanLearning` to automatically identify label errors and train a more robust model.\n",
    "\n",
    "`CleanLearning` is a wrapper than can be easily applied to any scikit-learn compatible model (which our `model` above is because we wrapped it with the scikit-learn compatible `KerasWrapperModel`!). Once wrapped, the resulting model can still be used in the same manner as a regular sklearn model, but it will now train more robustly if the data have noisy labels.\n",
    "\n",
    "Here, we wrap the our Keras model in a `CleanLearning` object and call `fit()` to train the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "e8327ebb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# define fresh model for CleanLearning\n",
    "model = KerasWrapperModel(\n",
    "    model=build_model,\n",
    "    model_kwargs={\n",
    "        \"model_name\": MODEL_NAME,\n",
    "        \"max_len\": 30,\n",
    "        \"n_classes\": 2,\n",
    "    },\n",
    "    compile_kwargs= {\n",
    "      \"optimizer\":tf.keras.optimizers.Adam(2e-5),\n",
    "      \"loss\":tf.keras.losses.SparseCategoricalCrossentropy(),\n",
    "      \"metrics\":[\"accuracy\"],\n",
    "    },\n",
    ")\n",
    "\n",
    "cl = CleanLearning(clf=model, cv_n_folds=3, verbose=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fbe88e58",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "cl.fit(train_input_ids, train_labels, clf_kwargs={\"epochs\": 5,  \"batch_size\": 64})  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60600e64",
   "metadata": {},
   "source": [
    "cleanlab's `CleanLearning` will automatically identify the label issues in the dataset and remove them, before training the final model on the remaining (clean) subset of the data, which will produce a more robust model.\n",
    "\n",
    "Next, we can check the performance of our newly trained `CleanLearning` model by measuring its accuracy on the same test set as above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "ad6bd1ac",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "32/32 [==============================] - 2s 24ms/step\n"
     ]
    }
   ],
   "source": [
    "cl_preds = cl.predict(test_input_ids)\n",
    "cl_accuracy = np.mean(cl_preds == test_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "e038205a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CleanLearning model accuracy = 0.945\n"
     ]
    }
   ],
   "source": [
    "print(f\"CleanLearning model accuracy = {cl_accuracy}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b9359b7",
   "metadata": {},
   "source": [
    "We see here that the test accuracy has improved after cleaning the label issues from the original dataset. \n",
    "\n",
    "We can also check out the label issues identified by cleanlab by calling the `get_label_issues()` method on the `CleanLearning` object. Here, we will print the index of the top 10 issues and take a closer look at a few of them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "c76db1ba",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>is_label_issue</th>\n",
       "      <th>label_quality</th>\n",
       "      <th>given_label</th>\n",
       "      <th>predicted_label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>False</td>\n",
       "      <td>0.729322</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>False</td>\n",
       "      <td>0.727811</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>False</td>\n",
       "      <td>0.729250</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>True</td>\n",
       "      <td>0.312260</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>False</td>\n",
       "      <td>0.730861</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4995</th>\n",
       "      <td>False</td>\n",
       "      <td>0.730561</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4996</th>\n",
       "      <td>False</td>\n",
       "      <td>0.724102</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4997</th>\n",
       "      <td>False</td>\n",
       "      <td>0.729231</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4998</th>\n",
       "      <td>False</td>\n",
       "      <td>0.724246</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4999</th>\n",
       "      <td>False</td>\n",
       "      <td>0.693937</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5000 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      is_label_issue  label_quality  given_label  predicted_label\n",
       "0              False       0.729322            1                1\n",
       "1              False       0.727811            1                1\n",
       "2              False       0.729250            1                1\n",
       "3               True       0.312260            1                0\n",
       "4              False       0.730861            1                1\n",
       "...              ...            ...          ...              ...\n",
       "4995           False       0.730561            1                1\n",
       "4996           False       0.724102            0                0\n",
       "4997           False       0.729231            1                1\n",
       "4998           False       0.724246            0                0\n",
       "4999           False       0.693937            0                0\n",
       "\n",
       "[5000 rows x 4 columns]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cl.get_label_issues()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "cd9d72f5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "index of the top 10 most likely errors: \n",
      " [3477 4560 4516 3731 1003 4330 2997 2689 3839 1075]\n"
     ]
    }
   ],
   "source": [
    "label_issues = cl.get_label_issues()\n",
    "lowest_quality_issues = label_issues[\"label_quality\"].argsort()[:10].to_numpy()\n",
    "\n",
    "print(f\"index of the top 10 most likely errors: \\n {lowest_quality_issues}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c2bf7dc",
   "metadata": {},
   "source": [
    "Let's see if cleanlab correctly identified these label errors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "47eb9766",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example Label: 0\n",
      "Example Text: Very satisfied. Love the magazine.l\n"
     ]
    }
   ],
   "source": [
    "i = 3477\n",
    "print(f\"Example Label: {train_data.iloc[i]['label']}\")\n",
    "print(f\"Example Text: {train_data.iloc[i]['review_text']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "3cb6397a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example Label: 0\n",
      "Example Text: I rec'd the first issue exactly one month from the day that I signed up\n",
      "for it. Excellent service. Very pleasantly surprised!!\n"
     ]
    }
   ],
   "source": [
    "i = 4560\n",
    "print(f\"Example Label: {train_data.iloc[i]['label']}\")\n",
    "print(f\"Example Text: {train_data.iloc[i]['review_text']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "2fce40b9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example Label: 0\n",
      "Example Text: By far the top auto magazine of both US and UK crop!  Love the in depth coverage, the industry insights, and the exclusive first coverage.  Simply the best!\n"
     ]
    }
   ],
   "source": [
    "i = 4516\n",
    "print(f\"Example Label: {train_data.iloc[i]['label']}\")\n",
    "print(f\"Example Text: {train_data.iloc[i]['review_text']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0dea5f77",
   "metadata": {},
   "source": [
    "All three examples above are clearly positive reviews which have been mislabeled as negative reviews (and have been correctly identified as label issues by cleanlab!)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "114e9f94",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3242eec9",
   "metadata": {},
   "source": [
    "We've demonstrated how handy `KerasWrapperModel` is to make any TensorFlow/Keras model compatible with scikit-learn. While we only demonstrated the use of this classifier with `GridSearchCV`, it is also compatible with a ton of other scikit-learn functionality such as Pipelines and more. \n",
    "\n",
    "By making your neural network sklearn-compatible, you can also easily use `CleanLearning` to identify label issues in your dataset and train a more robust version of the same model!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}