yelp_review_classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text Classification using Naive Bayes model\n",
    "\n",
    "Yelp review data is taken from the Kaggle competetion\n",
    "\n",
    "** Goal: ** Classify the review text from 1 - 5 star range.\n",
    "\n",
    "- Data is splitted into train and test data sets\n",
    "- Data is transformed to document-term matrices.\n",
    "- ** Multinomial Naive Bayes ** model is applied on the training data\n",
    "- ** Accuracy ** , ** confusion matrix ** and ** classification report ** is generated.\n",
    "- Predicted data is examined using False Positive, False Negatives etc.\n",
    "- Frequency of 1 star and 5 star of tokens is calculated to examine which tokens are contributing\n",
    "- most to 1 star and 5 start predictions.\n",
    "- Calculated ** Precision **, ** Recall/Sensitivity ** , ** F1 score ** and ** Support **."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "yelp = pd.read_csv('./data/yelp.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10000, 10)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "yelp.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 10000 entries, 0 to 9999\n",
      "Data columns (total 10 columns):\n",
      "business_id    10000 non-null object\n",
      "date           10000 non-null object\n",
      "review_id      10000 non-null object\n",
      "stars          10000 non-null int64\n",
      "text           10000 non-null object\n",
      "type           10000 non-null object\n",
      "user_id        10000 non-null object\n",
      "cool           10000 non-null int64\n",
      "useful         10000 non-null int64\n",
      "funny          10000 non-null int64\n",
      "dtypes: int64(4), object(6)\n",
      "memory usage: 781.3+ KB\n"
     ]
    }
   ],
   "source": [
    "# check the data types of columns\n",
    "yelp.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>business_id</th>\n",
       "      <th>date</th>\n",
       "      <th>review_id</th>\n",
       "      <th>stars</th>\n",
       "      <th>text</th>\n",
       "      <th>type</th>\n",
       "      <th>user_id</th>\n",
       "      <th>cool</th>\n",
       "      <th>useful</th>\n",
       "      <th>funny</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1090</th>\n",
       "      <td>v68yAIw0gpn15xwtcR_rKg</td>\n",
       "      <td>2012-05-28</td>\n",
       "      <td>uMTrZsAO-U1IXZeMDo-U5g</td>\n",
       "      <td>4</td>\n",
       "      <td>This Dutch Bros had some of the best customer ...</td>\n",
       "      <td>review</td>\n",
       "      <td>sYxD6VdrAaajp-gxim0I_Q</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5599</th>\n",
       "      <td>fP-BPL6iRu2tbcvlnjRshw</td>\n",
       "      <td>2012-10-21</td>\n",
       "      <td>9o9V6DK7LRHgwzYYA7vUfg</td>\n",
       "      <td>3</td>\n",
       "      <td>Super cool place I've been to 3-4 times now an...</td>\n",
       "      <td>review</td>\n",
       "      <td>5mvaOdV7nyKCP2Xct7yDTw</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3532</th>\n",
       "      <td>fJ46ok6poCuLGT1O2M3xBA</td>\n",
       "      <td>2011-03-08</td>\n",
       "      <td>18uXgzZnJRCfBeP3V3tKqQ</td>\n",
       "      <td>5</td>\n",
       "      <td>I had an amazing facial here! \\nThe space is c...</td>\n",
       "      <td>review</td>\n",
       "      <td>0o0VMEJeQY0pAAZ9nxErBA</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2527</th>\n",
       "      <td>sYZt3f1YFlg0ycDMyO-vJw</td>\n",
       "      <td>2010-03-25</td>\n",
       "      <td>uUr6meezb22Ig40LtvfYyA</td>\n",
       "      <td>2</td>\n",
       "      <td>What kind of sports bar doesn't serve SLIDERS?...</td>\n",
       "      <td>review</td>\n",
       "      <td>OksbhhgC71Ary3zNHMypeQ</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2567</th>\n",
       "      <td>5Feaj6aixO_QxKFnkHf3xg</td>\n",
       "      <td>2012-03-28</td>\n",
       "      <td>ExR82nFB6qg5xvc9kU0bnw</td>\n",
       "      <td>4</td>\n",
       "      <td>Stayed here with my girlfriend for a Radiohead...</td>\n",
       "      <td>review</td>\n",
       "      <td>vt0y5E2LUFeG0_PZ1b3d_Q</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 business_id        date               review_id  stars  \\\n",
       "1090  v68yAIw0gpn15xwtcR_rKg  2012-05-28  uMTrZsAO-U1IXZeMDo-U5g      4   \n",
       "5599  fP-BPL6iRu2tbcvlnjRshw  2012-10-21  9o9V6DK7LRHgwzYYA7vUfg      3   \n",
       "3532  fJ46ok6poCuLGT1O2M3xBA  2011-03-08  18uXgzZnJRCfBeP3V3tKqQ      5   \n",
       "2527  sYZt3f1YFlg0ycDMyO-vJw  2010-03-25  uUr6meezb22Ig40LtvfYyA      2   \n",
       "2567  5Feaj6aixO_QxKFnkHf3xg  2012-03-28  ExR82nFB6qg5xvc9kU0bnw      4   \n",
       "\n",
       "                                                   text    type  \\\n",
       "1090  This Dutch Bros had some of the best customer ...  review   \n",
       "5599  Super cool place I've been to 3-4 times now an...  review   \n",
       "3532  I had an amazing facial here! \\nThe space is c...  review   \n",
       "2527  What kind of sports bar doesn't serve SLIDERS?...  review   \n",
       "2567  Stayed here with my girlfriend for a Radiohead...  review   \n",
       "\n",
       "                     user_id  cool  useful  funny  \n",
       "1090  sYxD6VdrAaajp-gxim0I_Q     0       1      0  \n",
       "5599  5mvaOdV7nyKCP2Xct7yDTw     0       0      0  \n",
       "3532  0o0VMEJeQY0pAAZ9nxErBA     3       2      2  \n",
       "2527  OksbhhgC71Ary3zNHMypeQ     0       0      0  \n",
       "2567  vt0y5E2LUFeG0_PZ1b3d_Q     0       0      0  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Random 5 yelp reviews\n",
    "yelp.sample(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1     749\n",
       "2     927\n",
       "3    1461\n",
       "4    3526\n",
       "5    3337\n",
       "Name: stars, dtype: int64"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# class distribution\n",
    "yelp.stars.value_counts().sort_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First, take only two class - 5-start and 1-star reviews\n",
    "yelp.best_worst = yelp[(yelp.stars == 5) | (yelp.stars == 1)]\n",
    "\n",
    "# equivalently, 'loc' method\n",
    "yelp.best_worst = yelp.loc[(yelp.stars ==5) | (yelp.stars == 1), :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4086, 10)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Examine the shape\n",
    "yelp.best_worst.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define X and y\n",
    "X = yelp.best_worst.text\n",
    "y = yelp.best_worst.stars"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split into trianing and test data\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create document-term matrices\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.naive_bayes import MultinomialNB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3064, 16825)"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# define a count vector\n",
    "vect = CountVectorizer()\n",
    "\n",
    "# fit and transform train data set\n",
    "X_train_dtm = vect.fit_transform(X_train)\n",
    "X_train_dtm.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1022, 16825)"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# make document-term matrix\n",
    "X_test_dtm = vect.transform(X_test)\n",
    "X_test_dtm.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True)"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create an instance of naive bayes model and train the model\n",
    "nb = MultinomialNB()\n",
    "nb.fit(X_train_dtm, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can also use pipeline to perform in single step\n",
    "alt_model = make_pipeline(\n",
    "    CountVectorizer(), MultinomialNB()\n",
    ").fit(X_train, y_train)\n",
    "\n",
    "# Predict class for test data set revies\n",
    "alt_test_pred = nb.predict(X_test_dtm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Predict class for test data set revies\n",
    "test_pred = nb.predict(X_test_dtm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9187866927592955"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate accuracy\n",
    "from sklearn import metrics\n",
    "metrics.accuracy_score(y_test, test_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[126  58]\n",
      " [ 25 813]]\n"
     ]
    }
   ],
   "source": [
    "# confusion matrix\n",
    "print(metrics.confusion_matrix(y_test, test_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5    838\n",
       "1    184\n",
       "Name: stars, dtype: int64"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# exampine the class distribution of the test data set\n",
    "y_test.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5    0.819961\n",
       "Name: stars, dtype: float64"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate null accuracy for 5-star reviews\n",
    "# 81.9 % of reviews are 5-star reviews \n",
    "y_test.value_counts().head(1) / y_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2175    This has to be the worst restaurant in terms o...\n",
       "1781    If you like the stuck up Scottsdale vibe this ...\n",
       "2674    I'm sorry to be what seems to be the lone one ...\n",
       "9984    Went last night to Whore Foods to get basics t...\n",
       "3392    I found Lisa G's while driving through phoenix...\n",
       "8283    Don't know where I should start. Grand opening...\n",
       "2765    Went last week, and ordered a dozen variety. I...\n",
       "2839    Never Again,\\nI brought my Mountain Bike in (w...\n",
       "321     My wife and I live around the corner, hadn't e...\n",
       "1919                                         D-scust-ing.\n",
       "Name: text, dtype: object"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now lets observe predictions and see FP FN\n",
    "\n",
    "# first 10 false positives (1-star reviews incorrectly classified as 5-star reviews)\n",
    "X_test[y_test < test_pred].head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"If you like the stuck up Scottsdale vibe this is a good place for you. The food isn't impressive. Nice outdoor seating.\""
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Model is reacting to words good, impressive, nice\n",
    "yelp.loc[1781, \"text\"]\n",
    "X_test[1781]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'D-scust-ing.'"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# like this, model doesn't have enough data to classify such review text\n",
    "\n",
    "X_test[1919]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7148    I now consider myself an Arizonian. If you dri...\n",
       "4963    This is by far my favourite department store, ...\n",
       "6318    Since I have ranted recently on poor customer ...\n",
       "380     This is a must try for any Mani Pedi fan. I us...\n",
       "5565    I`ve had work done by this shop a few times th...\n",
       "3448    I was there last week with my sisters and whil...\n",
       "6050    I went to sears today to check on a layaway th...\n",
       "2504    I've passed by prestige nails in walmart 100s ...\n",
       "2475    This place is so great! I am a nanny and had t...\n",
       "241     I was sad to come back to lai lai's and they n...\n",
       "Name: text, dtype: object"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# first 10 false negatives (5-star reviews incorrectly classified as 1-star reviews)\n",
    "X_test[y_test > test_pred].head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'This is by far my favourite department store, hands down. I have had nothing but perfect experiences in this store, without exception, no matter what department I\\'m in. The shoe SA\\'s will bend over backwards to help you find a specific shoe, and the staff will even go so far as to send out hand-written thank you cards to your home address after you make a purchase - big or small. Tim & Anthony in the shoe salon are fabulous beyond words! \\n\\nI am not completely sure that I understand why people complain about the amount of merchandise on the floor or the lack of crowds in this store. Frankly, I would rather not be bombarded with merchandise and other people. One of the things I love the most about Barney\\'s is not only the prompt attention of SA\\'s, but the fact that they aren\\'t rushing around trying to help 35 people at once. The SA\\'s at Barney\\'s are incredibly friendly and will stop to have an actual conversation, regardless or whether you are purchasing something or not. I have also never experienced a \"high pressure\" sale situation here.\\n\\nAll in all, Barneys is pricey, and there is no getting around it. But, um, so is Neiman\\'s and that place is a crock. Anywhere that ONLY accepts American Express or their charge card and then treats you like scum if you aren\\'t carrying neither is no place that I want to spend my hard earned dollars. Yay Barneys!'"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# false negative: model is reacting to the words \"complain\", \"crowds\", \"rushing\", \"pricey\", \"scum\"\n",
    "X_test[4963]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16825"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Calculate which 10 tokens are most predictive\n",
    "\n",
    "# store the vocabulary of X_train\n",
    "X_train_tokens = vect.get_feature_names()\n",
    "len(X_train_tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "# store the number of times each token appears across each class\n",
    "one_star_token_count = nb.feature_count_[0, :]\n",
    "five_star_token_count = nb.feature_count_[1, :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a DataFrame of tokens with their separate one-star and five-star counts\n",
    "tokens = pd.DataFrame({\n",
    "    'token':X_train_tokens,\n",
    "    'one_star':one_star_token_count,\n",
    "    'five_star':five_star_token_count}\n",
    ").set_index('token')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add 1 to one-star and five-star counts to avoid dividing by 0\n",
    "tokens['one_star'] = tokens.one_star + 1\n",
    "tokens['five_star'] = tokens.five_star + 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 565., 2499.])"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# first number is one-star reviews, second number is five-star reviews\n",
    "nb.class_count_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert the one-star and five-star counts into frequencies\n",
    "tokens['one_star'] = tokens.one_star / nb.class_count_[0]\n",
    "tokens['five_star'] = tokens.five_star / nb.class_count_[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate the ratio of five-star to one-star for each token\n",
    "tokens['five_star_ratio'] = tokens.five_star / tokens.one_star"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>five_star</th>\n",
       "      <th>one_star</th>\n",
       "      <th>five_star_ratio</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>token</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>fantastic</th>\n",
       "      <td>0.077231</td>\n",
       "      <td>0.003540</td>\n",
       "      <td>21.817727</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>perfect</th>\n",
       "      <td>0.098039</td>\n",
       "      <td>0.005310</td>\n",
       "      <td>18.464052</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>yum</th>\n",
       "      <td>0.024810</td>\n",
       "      <td>0.001770</td>\n",
       "      <td>14.017607</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>favorite</th>\n",
       "      <td>0.138055</td>\n",
       "      <td>0.012389</td>\n",
       "      <td>11.143029</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>outstanding</th>\n",
       "      <td>0.019608</td>\n",
       "      <td>0.001770</td>\n",
       "      <td>11.078431</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>brunch</th>\n",
       "      <td>0.016807</td>\n",
       "      <td>0.001770</td>\n",
       "      <td>9.495798</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gem</th>\n",
       "      <td>0.016006</td>\n",
       "      <td>0.001770</td>\n",
       "      <td>9.043617</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mozzarella</th>\n",
       "      <td>0.015606</td>\n",
       "      <td>0.001770</td>\n",
       "      <td>8.817527</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pasty</th>\n",
       "      <td>0.015606</td>\n",
       "      <td>0.001770</td>\n",
       "      <td>8.817527</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>amazing</th>\n",
       "      <td>0.185274</td>\n",
       "      <td>0.021239</td>\n",
       "      <td>8.723323</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             five_star  one_star  five_star_ratio\n",
       "token                                            \n",
       "fantastic     0.077231  0.003540        21.817727\n",
       "perfect       0.098039  0.005310        18.464052\n",
       "yum           0.024810  0.001770        14.017607\n",
       "favorite      0.138055  0.012389        11.143029\n",
       "outstanding   0.019608  0.001770        11.078431\n",
       "brunch        0.016807  0.001770         9.495798\n",
       "gem           0.016006  0.001770         9.043617\n",
       "mozzarella    0.015606  0.001770         8.817527\n",
       "pasty         0.015606  0.001770         8.817527\n",
       "amazing       0.185274  0.021239         8.723323"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# sort the DataFrame by five_star_ratio (descending order), and examine the first 10 rows\n",
    "tokens.sort_values('five_star_ratio', ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>five_star</th>\n",
       "      <th>one_star</th>\n",
       "      <th>five_star_ratio</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>token</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>staffperson</th>\n",
       "      <td>0.0004</td>\n",
       "      <td>0.030088</td>\n",
       "      <td>0.013299</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>refused</th>\n",
       "      <td>0.0004</td>\n",
       "      <td>0.024779</td>\n",
       "      <td>0.016149</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>disgusting</th>\n",
       "      <td>0.0008</td>\n",
       "      <td>0.042478</td>\n",
       "      <td>0.018841</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>filthy</th>\n",
       "      <td>0.0004</td>\n",
       "      <td>0.019469</td>\n",
       "      <td>0.020554</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unprofessional</th>\n",
       "      <td>0.0004</td>\n",
       "      <td>0.015929</td>\n",
       "      <td>0.025121</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unacceptable</th>\n",
       "      <td>0.0004</td>\n",
       "      <td>0.015929</td>\n",
       "      <td>0.025121</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>acknowledge</th>\n",
       "      <td>0.0004</td>\n",
       "      <td>0.015929</td>\n",
       "      <td>0.025121</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ugh</th>\n",
       "      <td>0.0008</td>\n",
       "      <td>0.030088</td>\n",
       "      <td>0.026599</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fuse</th>\n",
       "      <td>0.0004</td>\n",
       "      <td>0.014159</td>\n",
       "      <td>0.028261</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>boca</th>\n",
       "      <td>0.0004</td>\n",
       "      <td>0.014159</td>\n",
       "      <td>0.028261</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                five_star  one_star  five_star_ratio\n",
       "token                                               \n",
       "staffperson        0.0004  0.030088         0.013299\n",
       "refused            0.0004  0.024779         0.016149\n",
       "disgusting         0.0008  0.042478         0.018841\n",
       "filthy             0.0004  0.019469         0.020554\n",
       "unprofessional     0.0004  0.015929         0.025121\n",
       "unacceptable       0.0004  0.015929         0.025121\n",
       "acknowledge        0.0004  0.015929         0.025121\n",
       "ugh                0.0008  0.030088         0.026599\n",
       "fuse               0.0004  0.014159         0.028261\n",
       "boca               0.0004  0.014159         0.028261"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# sort the DataFrame by five_star_ratio (ascending order),\n",
    "# and examine the first 10 rows, tokens contributing to 1-star reviews\n",
    "tokens.sort_values('five_star_ratio', ascending=True).head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's observe predicts for each class(5-class)\n",
    "# Multiclass classification\n",
    "# define X and y using the original DataFrame\n",
    "X = yelp.text\n",
    "y = yelp.stars"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1     749\n",
       "2     927\n",
       "3    1461\n",
       "4    3526\n",
       "5    3337\n",
       "Name: stars, dtype: int64"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check distribution of classes\n",
    "y.value_counts().sort_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "# split X and y into training and testing sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create document-term matrices using CountVectorizer\n",
    "X_train_dtm = vect.fit_transform(X_train)\n",
    "X_test_dtm = vect.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True)"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# fit a Multinomial Naive Bayes model\n",
    "nb.fit(X_train_dtm, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "# make class predictions\n",
    "test_pred = nb.predict(X_test_dtm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4712"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate the accuary\n",
    "metrics.accuracy_score(y_test, test_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4    0.3536\n",
       "Name: stars, dtype: float64"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate the null accuracy\n",
    "y_test.value_counts().head(1) / y_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 55,  14,  24,  65,  27],\n",
       "       [ 28,  16,  41, 122,  27],\n",
       "       [  5,   7,  35, 281,  37],\n",
       "       [  7,   0,  16, 629, 232],\n",
       "       [  6,   4,   6, 373, 443]], dtype=int64)"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# print the confusion matrix\n",
    "metrics.confusion_matrix(y_test, test_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "          1       0.54      0.30      0.38       185\n",
      "          2       0.39      0.07      0.12       234\n",
      "          3       0.29      0.10      0.14       365\n",
      "          4       0.43      0.71      0.53       884\n",
      "          5       0.58      0.53      0.55       832\n",
      "\n",
      "avg / total       0.46      0.47      0.43      2500\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# print the classification report\n",
    "print(metrics.classification_report(y_test, test_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.5445544554455446\n"
     ]
    }
   ],
   "source": [
    "# Of all the samples we classified as true how many are actually true\n",
    "# Precesion(P) = (correctly predicted Positive) / (total predicted Positive)\n",
    "# P = TP / TP + FP\n",
    "\n",
    "# calculate precision for class 1\n",
    "precision = 55 / float(55 + 28 + 5 + 7 + 6)\n",
    "print(precision)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.2972972972972973\n"
     ]
    }
   ],
   "source": [
    "# Recall: Of all the actual true samples how many did we classify as true\n",
    "# Recall( R) = (correctly predicted Positive) / (total correct Positive observation) = TP / TP + FN\n",
    "\n",
    "# calculate recall/Sensitivity\n",
    "recall = 55 / float(55 + 14 + 24 + 65 + 27)\n",
    "print(recall)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.38461538461538464\n"
     ]
    }
   ],
   "source": [
    "# F1 score is a weighted average of precision and recall.\n",
    "# manually calculate the F1 score for class 1\n",
    "f1 = 2 * (precision * recall) / (precision + recall)\n",
    "print(f1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "185\n"
     ]
    }
   ],
   "source": [
    "# support: How many observations exist for which a given class is the true class?\n",
    "# manually calculate the support for class 1\n",
    "support = 55 + 14 + 24 + 65 + 27\n",
    "print(support)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "tf",
   "language": "python",
   "name": "tf"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}