ManualLabelledDataEvaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluate manually labeled data\n",
    "\n",
    "Author: Steeve Huang (黃功詳)\n",
    "\n",
    "This notebook demonstrates how good my algorithm is for automatic speech recognition generation.\n",
    "The 996 rows of data are manually labeled by myself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import os\n",
    "import re\n",
    "from sklearn.metrics import recall_score, precision_score, f1_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Data\n",
    "__gt_df__ is manually labeled by me.\n",
    "\n",
    "__2018-7-17-10-54__ is the predicted subtitle by my algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>subtitle</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>DTfHpK5W4M4-0001929-2.0</td>\n",
       "      <td>但不也是 挺好的嗎</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>QKYCPuoxj6U-0001075-2.0</td>\n",
       "      <td>白虎</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9cIr5pINQaY-0001757-2.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>aczYWgtYNYw-0002651-2.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>J7Cu0sIACKo-0000566-2.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        id   subtitle\n",
       "0  DTfHpK5W4M4-0001929-2.0  但不也是 挺好的嗎\n",
       "1  QKYCPuoxj6U-0001075-2.0         白虎\n",
       "2  9cIr5pINQaY-0001757-2.0        NaN\n",
       "3  aczYWgtYNYw-0002651-2.0        NaN\n",
       "4  J7Cu0sIACKo-0000566-2.0        NaN"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# read ground truth\n",
    "gt_df = pd.read_csv('manual_label_result/csvs/labelled_subtitle.csv')\n",
    "gt_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 227,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>prediction</th>\n",
       "      <th>confidence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7-eVc37Q_w8-0000004-2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>swKJqvSc3ek-0000007-2.0</td>\n",
       "      <td>你找不到人你打電話給我幹什麼</td>\n",
       "      <td>0.99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Dj8o7JJSFf0-0000013-2.0</td>\n",
       "      <td>我不是在怪她吧</td>\n",
       "      <td>0.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NCnvTl5jPlQ-0000013-2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-1.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>vDpOeD7URIg-0000018-2.0</td>\n",
       "      <td>真希望得最後決賽後</td>\n",
       "      <td>0.97</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        id      prediction  confidence\n",
       "0  7-eVc37Q_w8-0000004-2.0             NaN        0.00\n",
       "1  swKJqvSc3ek-0000007-2.0  你找不到人你打電話給我幹什麼        0.99\n",
       "2  Dj8o7JJSFf0-0000013-2.0         我不是在怪她吧        0.98\n",
       "3  NCnvTl5jPlQ-0000013-2.0             NaN       -1.00\n",
       "4  vDpOeD7URIg-0000018-2.0       真希望得最後決賽後        0.97"
      ]
     },
     "execution_count": 227,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# read prediction\n",
    "\n",
    "#2018-8-1-15-54 resnet101 weights run on resnet50\n",
    "#2018-8-1-16-19 resnet101 weights run on resnet101\n",
    "prediction_df = pd.read_csv('manual_label_result/csvs/2018-8-6-9-34.csv')\n",
    "prediction_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 228,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>subtitle</th>\n",
       "      <th>prediction</th>\n",
       "      <th>confidence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>DTfHpK5W4M4-0001929-2.0</td>\n",
       "      <td>但不也是 挺好的嗎</td>\n",
       "      <td>但不也是 , 挺好的嗎</td>\n",
       "      <td>0.69</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>QKYCPuoxj6U-0001075-2.0</td>\n",
       "      <td>白虎</td>\n",
       "      <td>白虎</td>\n",
       "      <td>0.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9cIr5pINQaY-0001757-2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>aczYWgtYNYw-0002651-2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-1.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>J7Cu0sIACKo-0000566-2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-1.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        id   subtitle   prediction  confidence\n",
       "0  DTfHpK5W4M4-0001929-2.0  但不也是 挺好的嗎  但不也是 , 挺好的嗎        0.69\n",
       "1  QKYCPuoxj6U-0001075-2.0         白虎           白虎        0.98\n",
       "2  9cIr5pINQaY-0001757-2.0        NaN          NaN        0.00\n",
       "3  aczYWgtYNYw-0002651-2.0        NaN          NaN       -1.00\n",
       "4  J7Cu0sIACKo-0000566-2.0        NaN          NaN       -1.00"
      ]
     },
     "execution_count": 228,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# merge ground truth with prediction\n",
    "merge_df = gt_df.merge(prediction_df, on='id')\n",
    "merge_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 229,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>subtitle</th>\n",
       "      <th>prediction</th>\n",
       "      <th>confidence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>DTfHpK5W4M4-0001929-2.0</td>\n",
       "      <td>但不也是挺好的嗎</td>\n",
       "      <td>但不也是,挺好的嗎</td>\n",
       "      <td>0.69</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>QKYCPuoxj6U-0001075-2.0</td>\n",
       "      <td>白虎</td>\n",
       "      <td>白虎</td>\n",
       "      <td>0.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9cIr5pINQaY-0001757-2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>aczYWgtYNYw-0002651-2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-1.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>J7Cu0sIACKo-0000566-2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-1.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        id  subtitle prediction  confidence\n",
       "0  DTfHpK5W4M4-0001929-2.0  但不也是挺好的嗎  但不也是,挺好的嗎        0.69\n",
       "1  QKYCPuoxj6U-0001075-2.0        白虎         白虎        0.98\n",
       "2  9cIr5pINQaY-0001757-2.0       NaN        NaN        0.00\n",
       "3  aczYWgtYNYw-0002651-2.0       NaN        NaN       -1.00\n",
       "4  J7Cu0sIACKo-0000566-2.0       NaN        NaN       -1.00"
      ]
     },
     "execution_count": 229,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "merge_df['subtitle'] = merge_df.subtitle.str.replace(' ', '')\n",
    "merge_df['prediction'] = merge_df.prediction.str.replace(' ', '')\n",
    "merge_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## No threshold\n",
    "\n",
    "Let's see what happen if we don't set any threshold."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 230,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total images: 996\n",
      "Total ground truth images containing subtitles: 511\n",
      "Total correctly predicted images that contain subtitles: 324\n",
      "Percentage of matched images containing subtitles 0.6340508806262231\n",
      "Total ground truth images not containing subtitles: 485\n",
      "Total predictions matched with groud truth that doesn't contain subtitle: 461\n",
      "Total correctly predicted images: 785\n"
     ]
    }
   ],
   "source": [
    "same_predictoin = merge_df.subtitle==merge_df.prediction\n",
    "both_nan = merge_df.subtitle.isna() & merge_df.prediction.isna()\n",
    "merge_df.loc[merge_df.subtitle.isna(),]\n",
    "print(f\"Total images: {merge_df.shape[0]}\")\n",
    "print(f\"Total ground truth images containing subtitles: {merge_df.loc[~merge_df.subtitle.isna()].shape[0]}\")\n",
    "print(f\"Total correctly predicted images that contain subtitles: {merge_df.loc[same_predictoin].shape[0]}\")\n",
    "print(f\"Percentage of matched images containing subtitles {merge_df.loc[same_predictoin].shape[0] / merge_df.loc[~merge_df.subtitle.isna()].shape[0]}\")\n",
    "print(f\"Total ground truth images not containing subtitles: {merge_df.loc[merge_df.subtitle.isna()].shape[0]}\")\n",
    "print(f\"Total predictions matched with groud truth that doesn't contain subtitle: {merge_df.loc[both_nan].shape[0]}\")\n",
    "print(f\"Total correctly predicted images: {merge_df.loc[both_nan | same_predictoin].shape[0]}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 231,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/A60174/anaconda3/envs/py36/lib/python3.6/site-packages/pandas/core/indexing.py:362: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  self.obj[key] = _infer_fill_value(value)\n",
      "/home/A60174/anaconda3/envs/py36/lib/python3.6/site-packages/pandas/core/indexing.py:543: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  self.obj[item] = s\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>threshold</th>\n",
       "      <th>precision</th>\n",
       "      <th>recall</th>\n",
       "      <th>f1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.50</td>\n",
       "      <td>0.684989</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.813049</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.60</td>\n",
       "      <td>0.708333</td>\n",
       "      <td>0.996914</td>\n",
       "      <td>0.828205</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.70</td>\n",
       "      <td>0.725843</td>\n",
       "      <td>0.996914</td>\n",
       "      <td>0.840052</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.80</td>\n",
       "      <td>0.747073</td>\n",
       "      <td>0.984568</td>\n",
       "      <td>0.849534</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.90</td>\n",
       "      <td>0.793651</td>\n",
       "      <td>0.925926</td>\n",
       "      <td>0.854701</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.95</td>\n",
       "      <td>0.846395</td>\n",
       "      <td>0.833333</td>\n",
       "      <td>0.839813</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   threshold  precision    recall        f1\n",
       "0       0.50   0.684989  1.000000  0.813049\n",
       "1       0.60   0.708333  0.996914  0.828205\n",
       "2       0.70   0.725843  0.996914  0.840052\n",
       "3       0.80   0.747073  0.984568  0.849534\n",
       "4       0.90   0.793651  0.925926  0.854701\n",
       "5       0.95   0.846395  0.833333  0.839813"
      ]
     },
     "execution_count": 231,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "merge_df_filtered = merge_df.loc[~merge_df.prediction.isna(),:]\n",
    "merge_df_filtered.loc[:,'label'] = 0\n",
    "merge_df_filtered.loc[merge_df_filtered.subtitle == merge_df_filtered.prediction, 'label'] = 1\n",
    "precisions = []\n",
    "recalls = []\n",
    "f1s = []\n",
    "thresholds= []\n",
    "record_df = pd.DataFrame(columns=['threshold', 'precision', 'recall', 'f1'])\n",
    "for threshold in [0.5,0.6,0.7,0.8,0.9, 0.95]:\n",
    "# threshold=0.7\n",
    "    matched_prediction = merge_df_filtered.subtitle == merge_df_filtered.prediction\n",
    "    higher_threshold = merge_df_filtered.confidence >= threshold\n",
    "    merge_df_filtered.loc[:,'predicted_label'] = 0\n",
    "    merge_df_filtered.loc[higher_threshold, 'predicted_label'] = 1\n",
    "    recall = recall_score(merge_df_filtered.label, merge_df_filtered.predicted_label)\n",
    "    precision = precision_score(merge_df_filtered.label.values, merge_df_filtered.predicted_label.values)\n",
    "    f1 = f1_score(merge_df_filtered.label.values, merge_df_filtered.predicted_label.values)\n",
    "    recalls.append(recall)\n",
    "    precisions.append(precision)\n",
    "    f1s.append(f1)\n",
    "    thresholds.append(threshold)\n",
    "record_df.loc[:,'threshold']= thresholds\n",
    "record_df.loc[:,'precision'] = precisions\n",
    "record_df.loc[:,'recall'] = recalls\n",
    "record_df.loc[:,'f1'] = f1s\n",
    "record_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discussion\n",
    "\n",
    "From the above table, it seems that setting confidence threshold = 0.9 is optimal since it derives the best f-measure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 232,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/A60174/anaconda3/envs/py36/lib/python3.6/site-packages/ipykernel_launcher.py:8: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>subtitle</th>\n",
       "      <th>prediction</th>\n",
       "      <th>confidence</th>\n",
       "      <th>label</th>\n",
       "      <th>predicted_label</th>\n",
       "      <th>insertion</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>DTfHpK5W4M4-0001929-2.0</td>\n",
       "      <td>但不也是挺好的嗎</td>\n",
       "      <td>但不也是,挺好的嗎</td>\n",
       "      <td>0.69</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>QKYCPuoxj6U-0001075-2.0</td>\n",
       "      <td>白虎</td>\n",
       "      <td>白虎</td>\n",
       "      <td>0.98</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>vDpOeD7URIg-0000517-2.0</td>\n",
       "      <td>妳竟然說我們的實力沒什麼了不起</td>\n",
       "      <td>竟然說我們的實力沒什麼了不起</td>\n",
       "      <td>0.93</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>YLiiOvjBqsk-0002153-2.0</td>\n",
       "      <td>信賴友情愛情感情</td>\n",
       "      <td>信賴友情愛情感情</td>\n",
       "      <td>0.83</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>T3icvVg_SZw-0004794-2.0</td>\n",
       "      <td>後來他家裡沒錢了</td>\n",
       "      <td>後來他家裡沒錢了</td>\n",
       "      <td>0.98</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         id         subtitle      prediction  confidence  \\\n",
       "0   DTfHpK5W4M4-0001929-2.0         但不也是挺好的嗎       但不也是,挺好的嗎        0.69   \n",
       "1   QKYCPuoxj6U-0001075-2.0               白虎              白虎        0.98   \n",
       "12  vDpOeD7URIg-0000517-2.0  妳竟然說我們的實力沒什麼了不起  竟然說我們的實力沒什麼了不起        0.93   \n",
       "15  YLiiOvjBqsk-0002153-2.0         信賴友情愛情感情        信賴友情愛情感情        0.83   \n",
       "17  T3icvVg_SZw-0004794-2.0         後來他家裡沒錢了        後來他家裡沒錢了        0.98   \n",
       "\n",
       "    label  predicted_label  insertion  \n",
       "0       0                0          1  \n",
       "1       1                1          0  \n",
       "12      0                0          0  \n",
       "15      1                0          0  \n",
       "17      1                1          0  "
      ]
     },
     "execution_count": 232,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def get_insertion(row):\n",
    "\n",
    "    # if it is nan\n",
    "    if pd.isna(row['subtitle']):\n",
    "        return len(row['prediction'])\n",
    "    else:\n",
    "        return len(set(row['prediction']) - set(row['subtitle']))    \n",
    "merge_df_filtered['insertion'] = merge_df_filtered.apply(get_insertion, axis=1)\n",
    "merge_df_filtered.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 233,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/A60174/anaconda3/envs/py36/lib/python3.6/site-packages/ipykernel_launcher.py:8: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>subtitle</th>\n",
       "      <th>prediction</th>\n",
       "      <th>confidence</th>\n",
       "      <th>label</th>\n",
       "      <th>predicted_label</th>\n",
       "      <th>insertion</th>\n",
       "      <th>deletion</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>DTfHpK5W4M4-0001929-2.0</td>\n",
       "      <td>但不也是挺好的嗎</td>\n",
       "      <td>但不也是,挺好的嗎</td>\n",
       "      <td>0.69</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>QKYCPuoxj6U-0001075-2.0</td>\n",
       "      <td>白虎</td>\n",
       "      <td>白虎</td>\n",
       "      <td>0.98</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>vDpOeD7URIg-0000517-2.0</td>\n",
       "      <td>妳竟然說我們的實力沒什麼了不起</td>\n",
       "      <td>竟然說我們的實力沒什麼了不起</td>\n",
       "      <td>0.93</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>YLiiOvjBqsk-0002153-2.0</td>\n",
       "      <td>信賴友情愛情感情</td>\n",
       "      <td>信賴友情愛情感情</td>\n",
       "      <td>0.83</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>T3icvVg_SZw-0004794-2.0</td>\n",
       "      <td>後來他家裡沒錢了</td>\n",
       "      <td>後來他家裡沒錢了</td>\n",
       "      <td>0.98</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         id         subtitle      prediction  confidence  \\\n",
       "0   DTfHpK5W4M4-0001929-2.0         但不也是挺好的嗎       但不也是,挺好的嗎        0.69   \n",
       "1   QKYCPuoxj6U-0001075-2.0               白虎              白虎        0.98   \n",
       "12  vDpOeD7URIg-0000517-2.0  妳竟然說我們的實力沒什麼了不起  竟然說我們的實力沒什麼了不起        0.93   \n",
       "15  YLiiOvjBqsk-0002153-2.0         信賴友情愛情感情        信賴友情愛情感情        0.83   \n",
       "17  T3icvVg_SZw-0004794-2.0         後來他家裡沒錢了        後來他家裡沒錢了        0.98   \n",
       "\n",
       "    label  predicted_label  insertion  deletion  \n",
       "0       0                0          1         0  \n",
       "1       1                1          0         0  \n",
       "12      0                0          0         1  \n",
       "15      1                0          0         0  \n",
       "17      1                1          0         0  "
      ]
     },
     "execution_count": 233,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def get_deletion(row):\n",
    "\n",
    "    # if it is nan\n",
    "    if pd.isna(row['subtitle']):\n",
    "        return 0\n",
    "    else:\n",
    "        return len(set(row['subtitle']) - set(row['prediction']))    \n",
    "merge_df_filtered['deletion'] = merge_df_filtered.apply(get_deletion, axis=1)\n",
    "merge_df_filtered.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 234,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total filtered sample: 378\n",
      "Total correct sample: 300\n",
      "Correct sample rate: 0.7936507936507936\n",
      "Average insertion: 1.1454545454545455 (characters / sentence)\n",
      "Average deletion: 1.76 (characters / sentence)\n",
      "Average sentence length: 7.965608465608466 (characters / sentence)\n"
     ]
    }
   ],
   "source": [
    "higher_threshold = merge_df_filtered.confidence>0.9\n",
    "matched_predictions = merge_df_filtered.prediction == merge_df_filtered.subtitle\n",
    "filtered_samples = merge_df_filtered.loc[higher_threshold].shape[0]\n",
    "correct_samples = merge_df_filtered.loc[higher_threshold &matched_predictions].shape[0]\n",
    "incorrect_samples = filtered_samples - correct_samples\n",
    "deleted_samples = merge_df_filtered.loc[(merge_df_filtered.deletion>0) & higher_threshold].shape[0]\n",
    "inserted_samples = merge_df_filtered.loc[(merge_df_filtered.insertion>0) & higher_threshold].shape[0]\n",
    "total_insertions = merge_df_filtered.loc[higher_threshold, 'insertion'].sum()\n",
    "total_deletions = merge_df_filtered.loc[higher_threshold, 'deletion'].sum()\n",
    "total_sentence_lengths = merge_df_filtered.loc[higher_threshold, 'subtitle'].str.len().sum()\n",
    "print(f\"Total filtered sample: {filtered_samples}\")\n",
    "print(f\"Total correct sample: {correct_samples}\")\n",
    "print(f\"Correct sample rate: {correct_samples/filtered_samples }\")\n",
    "print(f\"Average insertion: {total_insertions/inserted_samples } (characters / sentence)\")\n",
    "print(f\"Average deletion: {total_deletions/deleted_samples } (characters / sentence)\")\n",
    "print(f\"Average sentence length: {total_sentence_lengths/filtered_samples } (characters / sentence)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 235,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/A60174/anaconda3/envs/py36/lib/python3.6/site-packages/ipykernel_launcher.py:28: RuntimeWarning: invalid value encountered in long_scalars\n",
      "/home/A60174/anaconda3/envs/py36/lib/python3.6/site-packages/ipykernel_launcher.py:29: RuntimeWarning: invalid value encountered in long_scalars\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>threshold</th>\n",
       "      <th>output_samples</th>\n",
       "      <th>correct_output_samples</th>\n",
       "      <th>output_correct_rate</th>\n",
       "      <th>output_no_subtitles</th>\n",
       "      <th>filter_no_subtitles</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.980</td>\n",
       "      <td>260</td>\n",
       "      <td>232</td>\n",
       "      <td>0.892308</td>\n",
       "      <td>1</td>\n",
       "      <td>286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.984</td>\n",
       "      <td>158</td>\n",
       "      <td>146</td>\n",
       "      <td>0.924051</td>\n",
       "      <td>1</td>\n",
       "      <td>286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.986</td>\n",
       "      <td>155</td>\n",
       "      <td>143</td>\n",
       "      <td>0.922581</td>\n",
       "      <td>1</td>\n",
       "      <td>286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.988</td>\n",
       "      <td>153</td>\n",
       "      <td>143</td>\n",
       "      <td>0.934641</td>\n",
       "      <td>1</td>\n",
       "      <td>286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.990</td>\n",
       "      <td>153</td>\n",
       "      <td>143</td>\n",
       "      <td>0.934641</td>\n",
       "      <td>1</td>\n",
       "      <td>286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.992</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.994</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.996</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>286</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   threshold  output_samples  correct_output_samples  output_correct_rate  \\\n",
       "0      0.980             260                     232             0.892308   \n",
       "1      0.984             158                     146             0.924051   \n",
       "2      0.986             155                     143             0.922581   \n",
       "3      0.988             153                     143             0.934641   \n",
       "4      0.990             153                     143             0.934641   \n",
       "5      0.992               1                       1             1.000000   \n",
       "6      0.994               1                       1             1.000000   \n",
       "7      0.996               1                       1             1.000000   \n",
       "\n",
       "   output_no_subtitles  filter_no_subtitles  \n",
       "0                    1                  286  \n",
       "1                    1                  286  \n",
       "2                    1                  286  \n",
       "3                    1                  286  \n",
       "4                    1                  286  \n",
       "5                    0                  286  \n",
       "6                    0                  286  \n",
       "7                    0                  286  "
      ]
     },
     "execution_count": 235,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "average_insertions = []\n",
    "average_deletions = []\n",
    "average_sentence_lengths = []\n",
    "output_counts = []\n",
    "correct_counts = []\n",
    "output_no_subtitles = []\n",
    "filter_no_subtitles = []\n",
    "thresholds = [0.98,0.984,0.986, 0.988, 0.99, 0.992,0.994,0.996]\n",
    "for threshold in thresholds :\n",
    "    higher_threshold = merge_df_filtered.confidence>threshold\n",
    "    matched_predictions = merge_df_filtered.prediction == merge_df_filtered.subtitle\n",
    "    gt_no_subtitle = merge_df.subtitle.isna()\n",
    "    predict_no_subtitle = merge_df.confidence == -1\n",
    "    \n",
    "#     print(predict_no_subtitle.head())\n",
    "#     print(merge_df_filtered.confidence.head())\n",
    "    filtered_samples = merge_df_filtered.loc[higher_threshold].shape[0]\n",
    "    correct_samples = merge_df_filtered.loc[higher_threshold &matched_predictions].shape[0]\n",
    "    incorrect_samples = filtered_samples - correct_samples\n",
    "    deleted_samples = merge_df_filtered.loc[(merge_df_filtered.deletion>0) & higher_threshold].shape[0]\n",
    "    inserted_samples = merge_df_filtered.loc[(merge_df_filtered.insertion>0) & higher_threshold].shape[0]\n",
    "    total_insertions = merge_df_filtered.loc[higher_threshold, 'insertion'].sum()\n",
    "    total_deletions = merge_df_filtered.loc[higher_threshold, 'deletion'].sum()\n",
    "    total_sentence_lengths = merge_df_filtered.loc[higher_threshold, 'subtitle'].str.len().sum()\n",
    "    output_no_subtitle = merge_df_filtered.loc[higher_threshold &   gt_no_subtitle].shape[0]\n",
    "    filter_no_subtitle = merge_df.loc[gt_no_subtitle & predict_no_subtitle].shape[0]\n",
    "\n",
    "    average_insertion = total_insertions/inserted_samples\n",
    "    average_deletion = total_deletions/deleted_samples\n",
    "    average_sentence_length = total_sentence_lengths/filtered_samples\n",
    "    output_count = filtered_samples\n",
    "    correct_count = correct_samples\n",
    "    \n",
    "    average_insertions.append(average_insertion)\n",
    "    average_deletions.append(average_deletion)\n",
    "    average_sentence_lengths.append(average_sentence_length)\n",
    "    output_counts.append(output_count)\n",
    "    correct_counts.append(correct_count)\n",
    "    output_no_subtitles.append(output_no_subtitle)\n",
    "    filter_no_subtitles.append(filter_no_subtitle)\n",
    "\n",
    "df = pd.DataFrame(columns=['threshold', 'output_samples','correct_output_samples', 'output_correct_rate', 'output_no_subtitles', 'filter_no_subtitles'])#,'average_insertions', 'average_deletions'])\n",
    "\n",
    "df['threshold'] = thresholds\n",
    "df['output_samples'] = output_counts\n",
    "df['correct_output_samples'] = correct_counts\n",
    "df['output_no_subtitles'] = output_no_subtitles\n",
    "df['filter_no_subtitles'] = filter_no_subtitles\n",
    "# df['average_insertions'] = average_insertions\n",
    "# df['average_deletions'] = average_deletions\n",
    "# df['average_sentence_lengths'] = average_sentence_lengths\n",
    "df['output_correct_rate'] = df.correct_output_samples / df.output_samples\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 236,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>subtitle</th>\n",
       "      <th>prediction</th>\n",
       "      <th>confidence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>DTfHpK5W4M4-0001929-2.0</td>\n",
       "      <td>但不也是挺好的嗎</td>\n",
       "      <td>但不也是,挺好的嗎</td>\n",
       "      <td>0.69</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>QKYCPuoxj6U-0001075-2.0</td>\n",
       "      <td>白虎</td>\n",
       "      <td>白虎</td>\n",
       "      <td>0.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9cIr5pINQaY-0001757-2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>aczYWgtYNYw-0002651-2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-1.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>J7Cu0sIACKo-0000566-2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-1.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        id  subtitle prediction  confidence\n",
       "0  DTfHpK5W4M4-0001929-2.0  但不也是挺好的嗎  但不也是,挺好的嗎        0.69\n",
       "1  QKYCPuoxj6U-0001075-2.0        白虎         白虎        0.98\n",
       "2  9cIr5pINQaY-0001757-2.0       NaN        NaN        0.00\n",
       "3  aczYWgtYNYw-0002651-2.0       NaN        NaN       -1.00\n",
       "4  J7Cu0sIACKo-0000566-2.0       NaN        NaN       -1.00"
      ]
     },
     "execution_count": 236,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "merge_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 237,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total filtered sample: 378\n",
      "Total correct sample: 300\n",
      "Correct sample rate: 0.7936507936507936\n",
      "Average insertion: 0.16666666666666666 (characters / sentence)\n",
      "Average deletion: 0.2328042328042328 (characters / sentence)\n",
      "Average sentence length: 7.965608465608466 (characters / sentence)\n"
     ]
    }
   ],
   "source": [
    "higher_threshold = merge_df_filtered.confidence>0.9\n",
    "matched_predictions = merge_df_filtered.prediction == merge_df_filtered.subtitle\n",
    "filtered_samples = merge_df_filtered.loc[higher_threshold].shape[0]\n",
    "correct_samples = merge_df_filtered.loc[higher_threshold &matched_predictions].shape[0]\n",
    "total_insertions = merge_df_filtered.loc[higher_threshold, 'insertion'].sum()\n",
    "total_deletions = merge_df_filtered.loc[higher_threshold, 'deletion'].sum()\n",
    "total_sentence_lengths = merge_df_filtered.loc[higher_threshold, 'subtitle'].str.len().sum()\n",
    "print(f\"Total filtered sample: {filtered_samples}\")\n",
    "print(f\"Total correct sample: {correct_samples}\")\n",
    "print(f\"Correct sample rate: {correct_samples/filtered_samples }\")\n",
    "print(f\"Average insertion: {total_insertions/filtered_samples } (characters / sentence)\")\n",
    "print(f\"Average deletion: {total_deletions/filtered_samples } (characters / sentence)\")\n",
    "print(f\"Average sentence length: {total_sentence_lengths/filtered_samples } (characters / sentence)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "The result is acceptable. \n",
    "\n",
    "In total, there are __510__ images containing subtitles. \n",
    "\n",
    "My algorithm generates __394__ positive predicted samples, __310__ of which are correctly predicted.\n",
    "\n",
    "The F-measure by setting __confidence threshold = 0.9__ is around 0.85.\n",
    "\n",
    "Average subtitle length = __8__ (characters / sentence)\n",
    "\n",
    "Average insertion = __0.14__ (characters / sentence)\n",
    "\n",
    "Average deletion = __0.26__ (characters / sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}