notebooks/Data Analysis & Prototype.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Auto Complete Server - Conversation Corpus Analysis & Prototype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal of this project is to build a relevant, robust, fast auto completion server that will be used to provide auto completion suggestions to customer service agents. The auto completion model will be trained on a provided set of customer/agent conversations.\n",
    "\n",
    "This notebook will be used to explore the data and start prototyping the future components of our auto-complete server."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 236,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import json\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import nltk\n",
    "import datrie\n",
    "import string\n",
    "import re\n",
    "import operator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Data\n",
    "\n",
    "The conversation json file is loaded into a dataframe where each row represents a message part of a given issue."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sample_file = '../data/sample_conversations.json'\n",
    " \n",
    "with open(sample_file, encoding='utf8') as f:\n",
    "    sample_json = json.loads(f.read())\n",
    "    \n",
    "df = pd.io.json.json_normalize(\n",
    "    data=sample_json,\n",
    "    record_path=['Issues', 'Messages'],\n",
    "    meta=[\n",
    "        ['Issues', 'CompanyGroupId'],\n",
    "        ['Issues', 'IssueId']\n",
    "    ]\n",
    ")\n",
    " \n",
    "df = df[['Issues.IssueId', 'Issues.CompanyGroupId', 'IsFromCustomer', 'Text']]\n",
    "df.rename(columns={'Issues.IssueId': 'IssueId', 'Issues.CompanyGroupId': 'CompanyGroupId'},\n",
    "          inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since our interest is in providing autocompletion suggestions to the customer service agent, we need to focus on their own messages, which are tagged with `IsFromCustomer = False`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>IssueId</th>\n",
       "      <th>CompanyGroupId</th>\n",
       "      <th>IsFromCustomer</th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>20383</th>\n",
       "      <td>13500001</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "      <td>OK let me update you account information.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>352</th>\n",
       "      <td>310001</td>\n",
       "      <td>40001</td>\n",
       "      <td>False</td>\n",
       "      <td>could you please provide me with your new addr...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7429</th>\n",
       "      <td>5050001</td>\n",
       "      <td>20001</td>\n",
       "      <td>False</td>\n",
       "      <td>Great. Have a safe flight</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5627</th>\n",
       "      <td>3860001</td>\n",
       "      <td>40001</td>\n",
       "      <td>False</td>\n",
       "      <td>Thank you for that information. I need to make...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11672</th>\n",
       "      <td>8270001</td>\n",
       "      <td>50001</td>\n",
       "      <td>False</td>\n",
       "      <td>Have you tried rebooting the system</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        IssueId  CompanyGroupId  IsFromCustomer  \\\n",
       "20383  13500001               1           False   \n",
       "352      310001           40001           False   \n",
       "7429    5050001           20001           False   \n",
       "5627    3860001           40001           False   \n",
       "11672   8270001           50001           False   \n",
       "\n",
       "                                                    Text  \n",
       "20383          OK let me update you account information.  \n",
       "352    could you please provide me with your new addr...  \n",
       "7429                           Great. Have a safe flight  \n",
       "5627   Thank you for that information. I need to make...  \n",
       "11672                Have you tried rebooting the system  "
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from_agent = df.IsFromCustomer == False\n",
    "df[from_agent].sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It will be useful later to split the messages text into sentences. This will enable us to easily measure sentence frequencies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Note: the creation of the sentences dataframe might not be scalable for a corpus of a billion messages\n",
    "# with this implementation. A clever usage of pandas stack() and joins might enable us to do it in a more\n",
    "# efficient fashion.\n",
    "tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')\n",
    "columns = ['IssueId', 'CompanyGroupId', 'IsFromCustomer', 'SentenceId', 'Sentence']\n",
    "df_sentences = pd.DataFrame(columns=columns)\n",
    "for message in df.itertuples():\n",
    "    sentences = tokenizer.tokenize(message.Text)\n",
    "    for i, sentence in enumerate(sentences):\n",
    "        new_row = [message.IssueId, message.CompanyGroupId, message.IsFromCustomer, i, sentence]\n",
    "        df_sentences = df_sentences.append(dict(zip(columns, new_row)), ignore_index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Is there anything else I can help you with today?    285\n",
       "Great.                                               268\n",
       "Okay.                                                204\n",
       "Thank you.                                           197\n",
       "Have a great day.                                    165\n",
       "You're welcome.                                      158\n",
       "Is there anything else I can help you with?          139\n",
       "Have a great day                                     138\n",
       "Is there anything else I can assist you with?        133\n",
       "Have a great day!                                     85\n",
       "Name: Sentence, dtype: int64"
      ]
     },
     "execution_count": 195,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from_agent = df_sentences.IsFromCustomer == False\n",
    "df_sentences[from_agent].Sentence.value_counts().head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How many sentences?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 307,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(16508, 5)"
      ]
     },
     "execution_count": 307,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_sentences[from_agent].shape`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## n-gram Frequencies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Unigrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 309,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')\n",
    "tokens = []\n",
    "for message in df.itertuples():\n",
    "    sentences = tokenizer.tokenize(message.Text)\n",
    "    for sentence in sentences:\n",
    "        tokens += nltk.word_tokenize(sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def ngram_freq(n):\n",
    "    ngrams = nltk.ngrams(tokens, n)\n",
    "    fdist = nltk.FreqDist(ngrams)\n",
    "    return fdist.most_common(5) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(('.',), 12362),\n",
       " (('I',), 10388),\n",
       " (('you',), 8612),\n",
       " (('the',), 5465),\n",
       " (('to',), 4748)]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ngram_freq(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vocabulary size?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 322,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6,416\n"
     ]
    }
   ],
   "source": [
    "unigrams = nltk.ngrams(tokens, 1)\n",
    "fdist = nltk.FreqDist(unigrams)\n",
    "v = fdist.B()\n",
    "print(\"{:,}\".format(v))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 324,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bigrams space size = 41,165,056\n",
      "Trigrams space size = 264,114,999,296\n"
     ]
    }
   ],
   "source": [
    "print(\"Bigrams space size = {:,}\".format(v**2))\n",
    "print(\"Trigrams space size = {:,}\".format(v**3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bigrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(('.', 'I'), 2629),\n",
       " (('I', 'can'), 1737),\n",
       " (('you', 'with'), 1355),\n",
       " (('Thank', 'you'), 1275),\n",
       " (('anything', 'else'), 1076)]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ngram_freq(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Trigrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(('there', 'anything', 'else'), 1041),\n",
       " (('else', 'I', 'can'), 950),\n",
       " (('anything', 'else', 'I'), 946),\n",
       " (('Is', 'there', 'anything'), 915),\n",
       " (('help', 'you', 'with'), 797)]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ngram_freq(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Those results should give a good idea of what the auto completion is expected to do. For example, if a representative types \"Is there\", the auto completion should suggest \"Is there anything else I can help you with?\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### n-grams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(('Is', 'there', 'anything', 'else', 'I', 'can', 'help', 'you', 'with'), 500),\n",
       " (('there', 'anything', 'else', 'I', 'can', 'help', 'you', 'with', 'today'),\n",
       "  326),\n",
       " (('.', 'Is', 'there', 'anything', 'else', 'I', 'can', 'help', 'you'), 312),\n",
       " (('anything', 'else', 'I', 'can', 'help', 'you', 'with', 'today', '?'), 305),\n",
       " (('Is', 'there', 'anything', 'else', 'I', 'can', 'assist', 'you', 'with'),\n",
       "  246)]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n = 9\n",
    "ngram_freq(n)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can observe that the frequencies logically decrease with the length of the n-gram. However, it might be more relevant to auto complete with a word with a full sentence than with the most frequent associated bigram. When mixing n-grams of different lengths, we need to find some way to adjust their scores - they cannot be compared on the raw frequency."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build Trie Data Structure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 199,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Is there anything else I can help you with today?    285\n",
       "Great.                                               268\n",
       "Okay.                                                204\n",
       "Thank you.                                           197\n",
       "Have a great day.                                    165\n",
       "You're welcome.                                      158\n",
       "Is there anything else I can help you with?          139\n",
       "Have a great day                                     138\n",
       "Is there anything else I can assist you with?        133\n",
       "Have a great day!                                     85\n",
       "Name: Sentence, dtype: int64"
      ]
     },
     "execution_count": 199,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentences_freq = df_sentences[from_agent].Sentence.value_counts()\n",
    "sentences_freq.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9845,)"
      ]
     },
     "execution_count": 200,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentences_freq.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's build the trie! The values of each sentence will correspond to frequency observed in the corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 221,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "trie = datrie.Trie(string.printable)\n",
    "for sentence, freq in sentences_freq.iteritems():\n",
    "    trie[sentence] = freq\n",
    "trie.save('sentences_freq.trie')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 253,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def generate_completions(prefix):\n",
    "    completions = trie.items(prefix)\n",
    "    sorted_d = sorted(\n",
    "        dict(completions).items(),\n",
    "        key=operator.itemgetter(1),\n",
    "        reverse=True\n",
    "    )\n",
    "    n_results = min(len(sorted_d), 5)\n",
    "    return sorted_d[:n_results]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 261,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 261,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "generate_completions(\"H\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quick and Dirty Performance Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 278,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "uname_result(system='Darwin', node='macgregor.local', release='17.7.0', version='Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64', machine='x86_64', processor='i386') i386 3.6.4 CPython\n"
     ]
    }
   ],
   "source": [
    "import platform\n",
    "print(\n",
    "    platform.uname(),\n",
    "    platform.processor(),\n",
    "    platform.python_version(),\n",
    "    platform.python_implementation()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 286,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100 loops, best of 7: 1.97 ms per loop\n"
     ]
    }
   ],
   "source": [
    "%%timeit -n 100 -r 7\n",
    "generate_completions(\"H\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 285,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100 loops, best of 7: 1.98 ms per loop\n"
     ]
    }
   ],
   "source": [
    "%%timeit -n 100 -r 7\n",
    "generate_completions(\"W\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 287,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100 loops, best of 7: 111 µs per loop\n"
     ]
    }
   ],
   "source": [
    "%%timeit -n 100 -r 7\n",
    "generate_completions(\"How\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}