EARIA_2019.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hackaton EARIA 2019"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Groupe _KaaRIs_\n",
    "> **Agnès** (LIP6) : Chef d'équipe, algorithmes de ranking et de diversité des résultats  \n",
    "> **Léo** (LIMSI) : Extraction et nettoyage des documents web  \n",
    "> **Aymen** : Recherches sur le résumé de texte par sélection  \n",
    "> **André** (Adesoft) : Fonctions utilitaires et rédaction des commentaires  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Les fausses nouvelles sont de plus en plus fréquentes dans l'actualité, et avec elles les travaux de vérification et de démentis de ces nouvelles. Certains journaux ont à cet effet créé des rubriques dédiées à ces travaux, comme la section des Décodeurs créée par les journaliste du Monde, ou les Fact Checkers de Libération. C'est sur ces derniers que porte le sujet de ce hackathon."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "L'objectif de ce projet est d'établir des pistes pour répondre à la problématique du fact-checking des fake news. Plus formellement, il s'agira de proposer des articles de réponse, accompagnés de sources pertinentes, à des questions posées par des internautes au sujet d'informations suspectes non vérifiées. Pour cela, on dispose d'un corpus constitué d'environ 3000 questions posées aux journalistes de Libération, accompagnées pour la moitié d'entre elles des réponses apportées et des documents cités comme sources."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Il s'agit pour chaque question de rendre un document textuel comparable à la réponse des journalistes et au contenu des documents cités pour appuyer cette réponse."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On bénéficie pour cette session de hackathon d'un accès privilégié à l'API de recherche du moteur de recherche Qwant. Les réponses de cette API seront évidemment filtrées pour éliminer les articles de réponse des fact-checker de Libération, pour éviter de biaiser les résultats."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 610,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "import nltk\n",
    "from nltk.util import ngrams\n",
    "import pandas as pd\n",
    "from os import listdir\n",
    "import string\n",
    "import urllib.request as urlr\n",
    "import urllib.parse as urlp\n",
    "from urllib.request import Request\n",
    "import json\n",
    "from nltk.corpus import wordnet\n",
    "from collections import Counter\n",
    "import datetime\n",
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "from bs4.element import Comment\n",
    "import pandas as pd\n",
    "import os\n",
    "import time\n",
    "import numpy as np\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.cluster import KMeans\n",
    "from sklearn.metrics import adjusted_rand_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### API Qwant"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Basé sur du code fourni pour l'hackathon."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "BASEURL=\"http://192.168.1.1:8000/qwant\"\n",
    "USER=\"kaaRIs\"\n",
    "PASSWORD=\"***************\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 663,
   "metadata": {},
   "outputs": [],
   "source": [
    "def qwant_raw(q):\n",
    "    query=\"user={}&pass={}&q={}\".format(urlp.quote(USER),urlp.quote(PASSWORD),urlp.quote(q))\n",
    "    #print(\"{}/query?{}\".format(BASEURL,query))\n",
    "    r = urlr.urlopen(\"{}/query?{}\".format(BASEURL,query))\n",
    "    x = r.read().decode(\"utf-8\")\n",
    "    response = dict()\n",
    "    if len(x):\n",
    "        response = json.loads(x)\n",
    "    return response\n",
    "\n",
    "def qwant(q):\n",
    "    response = qwant_raw(q)\n",
    "    links = [r['url'] for r in response.get('result', dict()).get('items', [])]\n",
    "    now = datetime.datetime.now()\n",
    "    print(\"[\", now.hour, now.minute, now.second, \"]   Requête traitée :\", q[:40], \"... -\", len(links), \"liens trouvés.\")\n",
    "    return links\n",
    "\n",
    "def get_doc(url):\n",
    "    query=\"user={}&pass={}&url={}\".format(urlp.quote(USER),urlp.quote(PASSWORD),urlp.quote(url))\n",
    "    r = urlr.urlopen(\"{}/document?{}\".format(BASEURL,query))\n",
    "    \n",
    "    return r.read().decode(\"utf-8\", errors=\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Chargement des données fournies : ensemble d'entraînement (questions posées, réponses apportées, et liste des documents fournis en source), ensemble de test (questions seules), et contenu des documents utliss dans l'ensemble d'entraînement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TRAIN_DATASET = pd.read_json(\"trainQD.json\").set_index('id')\n",
    "TEST_DATASET  = pd.read_json(\"testQD.json\").set_index('id')\n",
    "FILES_DATASET = pd.DataFrame([[int(document.split(\".\")[0]), open(\"train/\" + document).read()] for document in listdir(\"train\")]).set_index(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2585</th>\n",
       "      <td>Est-ce qu'un «black face» est répréhensible pa...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>750</th>\n",
       "      <td>Peter Madsen a été condamné à la prison à vie ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>140</th>\n",
       "      <td>Est-ce que seuls les bus de banlieue ont été a...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               question\n",
       "id                                                     \n",
       "2585  Est-ce qu'un «black face» est répréhensible pa...\n",
       "750   Peter Madsen a été condamné à la prison à vie ...\n",
       "140   Est-ce que seuls les bus de banlieue ont été a..."
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TEST_DATASET.sample(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer</th>\n",
       "      <th>docs</th>\n",
       "      <th>question</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1269</th>\n",
       "      <td>Question posée par Ber33 le 01/06/2018\\nBonjou...</td>\n",
       "      <td>[7231]</td>\n",
       "      <td>Qui est le vrai voisin de l'enfant du balcon?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2495</th>\n",
       "      <td>Question posée par le 16/01/2019\\nGuten Tag,\\n...</td>\n",
       "      <td>[1924, 796, 6532, 3303, 10632, 3463, 2951, 522...</td>\n",
       "      <td>Que prévoit vraiment le traité d'Aix-la-Chapel...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>280</th>\n",
       "      <td>Question posée par le 22/11/2018\\nBonjour,\\nNo...</td>\n",
       "      <td>[9515, 7231]</td>\n",
       "      <td>Doit-on attendre janvier 2019 pour bénéficier ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 answer  \\\n",
       "id                                                        \n",
       "1269  Question posée par Ber33 le 01/06/2018\\nBonjou...   \n",
       "2495  Question posée par le 16/01/2019\\nGuten Tag,\\n...   \n",
       "280   Question posée par le 22/11/2018\\nBonjour,\\nNo...   \n",
       "\n",
       "                                                   docs  \\\n",
       "id                                                        \n",
       "1269                                             [7231]   \n",
       "2495  [1924, 796, 6532, 3303, 10632, 3463, 2951, 522...   \n",
       "280                                        [9515, 7231]   \n",
       "\n",
       "                                               question  \n",
       "id                                                       \n",
       "1269      Qui est le vrai voisin de l'enfant du balcon?  \n",
       "2495  Que prévoit vraiment le traité d'Aix-la-Chapel...  \n",
       "280   Doit-on attendre janvier 2019 pour bénéficier ...  "
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TRAIN_DATASET.sample(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9968</th>\n",
       "      <td>__\\n\\n__\\n\\n  * [Home](https://www.tnp.no/)\\n ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11849</th>\n",
       "      <td>Lettre du garde des Sceaux à un futur ministre...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10386</th>\n",
       "      <td>\\n\\n</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                       1\n",
       "0                                                       \n",
       "9968   __\\n\\n__\\n\\n  * [Home](https://www.tnp.no/)\\n ...\n",
       "11849  Lettre du garde des Sceaux à un futur ministre...\n",
       "10386                                               \\n\\n"
      ]
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "FILES_DATASET.sample(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Utils"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Word processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 686,
   "metadata": {},
   "outputs": [],
   "source": [
    "BLACKLIST = {\"aujourd'hui\", \"quelqu'un\", \"affirme\", \"bien\", \"oui\", \"site\", \"lien\", \"vidéo\", \"article\", \"contre\", \"pour\", \"non\", \"plus\", \"dit\", \"questions\", \"monde\", \"libé\", \"facebook\", \"twitter\", \"vrai\", \"merci\", \"libération\", \"combien\", \"signaler\", \"signalé\", \"vraiment\", \"est\", \"ce\", \"que\", \"journal\", \"bonjour\", \"vérifier\", \"vérifiez\"}\n",
    "STOPWORDS = BLACKLIST | {'a', 'abord', 'absolument', 'afin', 'ah', 'ai', 'aie', 'aient', 'aies', 'ailleurs', 'ainsi', 'ait', 'allaient', 'allo', 'allons', 'allô', 'alors', 'anterieur', 'anterieure', 'anterieures', 'apres', 'après', 'as', 'assez', 'attendu', 'au', 'aucun', 'aucune', 'aucuns', 'aujourd', \"aujourd'hui\", 'aupres', 'auquel', 'aura', 'aurai', 'auraient', 'aurais', 'aurait', 'auras', 'aurez', 'auriez', 'aurions', 'aurons', 'auront', 'aussi', 'autre', 'autrefois', 'autrement', 'autres', 'autrui', 'aux', 'auxquelles', 'auxquels', 'avaient', 'avais', 'avait', 'avant', 'avec', 'avez', 'aviez', 'avions', 'avoir', 'avons', 'ayant', 'ayez', 'ayons', 'b', 'bah', 'bas', 'basee', 'bat', 'beau', 'beaucoup', 'bien', 'bigre', 'bon', 'boum', 'bravo', 'brrr', 'c', 'car', 'ce', 'ceci', 'cela', 'celle', 'celle-ci', 'celle-là', 'celles', 'celles-ci', 'celles-là', 'celui', 'celui-ci', 'celui-là', 'celà', 'cent', 'cependant', 'certain', 'certaine', 'certaines', 'certains', 'certes', 'ces', 'cet', 'cette', 'ceux', 'ceux-ci', 'ceux-là', 'chacun', 'chacune', 'chaque', 'cher', 'chers', 'chez', 'chiche', 'chut', 'chère', 'chères', 'ci', 'cinq', 'cinquantaine', 'cinquante', 'cinquantième', 'cinquième', 'clac', 'clic', 'combien', 'comme', 'comment', 'comparable', 'comparables', 'compris', 'concernant', 'contre', 'couic', 'crac', 'd', 'da', 'dans', 'de', 'debout', 'dedans', 'dehors', 'deja', 'delà', 'depuis', 'dernier', 'derniere', 'derriere', 'derrière', 'des', 'desormais', 'desquelles', 'desquels', 'dessous', 'dessus', 'deux', 'deuxième', 'deuxièmement', 'devant', 'devers', 'devra', 'devrait', 'different', 'differentes', 'differents', 'différent', 'différente', 'différentes', 'différents', 'dire', 'directe', 'directement', 'dit', 'dite', 'dits', 'divers', 'diverse', 'diverses', 'dix', 'dix-huit', 'dix-neuf', 'dix-sept', 'dixième', 'doit', 'doivent', 'donc', 'dont', 'dos', 'douze', 'douzième', 'dring', 'droite', 'du', 'duquel', 'durant', 'dès', 'début', 'désormais', 'e', 'effet', 'egale', 'egalement', 'egales', 'eh', 'elle', 'elle-même', 'elles', 'elles-mêmes', 'en', 'encore', 'enfin', 'entre', 'envers', 'environ', 'es', 'essai', 'est', 'et', 'etant', 'etc', 'etre', 'eu', 'eue', 'eues', 'euh', 'eurent', 'eus', 'eusse', 'eussent', 'eusses', 'eussiez', 'eussions', 'eut', 'eux', 'eux-mêmes', 'exactement', 'excepté', 'extenso', 'exterieur', 'eûmes', 'eût', 'eûtes', 'f', 'fais', 'faisaient', 'faisant', 'fait', 'faites', 'façon', 'feront', 'fi', 'flac', 'floc', 'fois', 'font', 'force', 'furent', 'fus', 'fusse', 'fussent', 'fusses', 'fussiez', 'fussions', 'fut', 'fûmes', 'fût', 'fûtes', 'g', 'gens', 'h', 'ha', 'haut', 'hein', 'hem', 'hep', 'hi', 'ho', 'holà', 'hop', 'hormis', 'hors', 'hou', 'houp', 'hue', 'hui', 'huit', 'huitième', 'hum', 'hurrah', 'hé', 'hélas', 'i', 'ici', 'il', 'ils', 'importe', 'j', 'je', 'jusqu', 'jusque', 'juste', 'k', 'l', 'la', 'laisser', 'laquelle', 'las', 'le', 'lequel', 'les', 'lesquelles', 'lesquels', 'leur', 'leurs', 'longtemps', 'lors', 'lorsque', 'lui', 'lui-meme', 'lui-même', 'là', 'lès', 'm', 'ma', 'maint', 'maintenant', 'mais', 'malgre', 'malgré', 'maximale', 'me', 'meme', 'memes', 'merci', 'mes', 'mien', 'mienne', 'miennes', 'miens', 'mille', 'mince', 'mine', 'minimale', 'moi', 'moi-meme', 'moi-même', 'moindres', 'moins', 'mon', 'mot', 'moyennant', 'multiple', 'multiples', 'même', 'mêmes', 'n', 'na', 'naturel', 'naturelle', 'naturelles', 'ne', 'neanmoins', 'necessaire', 'necessairement', 'neuf', 'neuvième', 'ni', 'nombreuses', 'nombreux', 'nommés', 'non', 'nos', 'notamment', 'notre', 'nous', 'nous-mêmes', 'nouveau', 'nouveaux', 'nul', 'néanmoins', 'nôtre', 'nôtres', 'o', 'oh', 'ohé', 'ollé', 'olé', 'on', 'ont', 'onze', 'onzième', 'ore', 'ou', 'ouf', 'ouias', 'oust', 'ouste', 'outre', 'ouvert', 'ouverte', 'ouverts', 'o|', 'où', 'p', 'paf', 'pan', 'par', 'parce', 'parfois', 'parle', 'parlent', 'parler', 'parmi', 'parole', 'parseme', 'partant', 'particulier', 'particulière', 'particulièrement', 'pas', 'passé', 'pendant', 'pense', 'permet', 'personne', 'personnes', 'peu', 'peut', 'peuvent', 'peux', 'pff', 'pfft', 'pfut', 'pif', 'pire', 'pièce', 'plein', 'plouf', 'plupart', 'plus', 'plusieurs', 'plutôt', 'possessif', 'possessifs', 'possible', 'possibles', 'pouah', 'pour', 'pourquoi', 'pourrais', 'pourrait', 'pouvait', 'prealable', 'precisement', 'premier', 'première', 'premièrement', 'pres', 'probable', 'probante', 'procedant', 'proche', 'près', 'psitt', 'pu', 'puis', 'puisque', 'pur', 'pure', 'q', 'qu', 'quand', 'quant', 'quant-à-soi', 'quanta', 'quarante', 'quatorze', 'quatre', 'quatre-vingt', 'quatrième', 'quatrièmement', 'que', 'quel', 'quelconque', 'quelle', 'quelles', \"quelqu'un\", 'quelque', 'quelques', 'quels', 'qui', 'quiconque', 'quinze', 'quoi', 'quoique', 'r', 'rare', 'rarement', 'rares', 'relative', 'relativement', 'remarquable', 'rend', 'rendre', 'restant', 'reste', 'restent', 'restrictif', 'retour', 'revoici', 'revoilà', 'rien', 's', 'sa', 'sacrebleu', 'sait', 'sans', 'sapristi', 'sauf', 'se', 'sein', 'seize', 'selon', 'semblable', 'semblaient', 'semble', 'semblent', 'sent', 'sept', 'septième', 'sera', 'serai', 'seraient', 'serais', 'serait', 'seras', 'serez', 'seriez', 'serions', 'serons', 'seront', 'ses', 'seul', 'seule', 'seulement', 'si', 'sien', 'sienne', 'siennes', 'siens', 'sinon', 'six', 'sixième', 'soi', 'soi-même', 'soient', 'sois', 'soit', 'soixante', 'sommes', 'son', 'sont', 'sous', 'souvent', 'soyez', 'soyons', 'specifique', 'specifiques', 'speculatif', 'stop', 'strictement', 'subtiles', 'suffisant', 'suffisante', 'suffit', 'suis', 'suit', 'suivant', 'suivante', 'suivantes', 'suivants', 'suivre', 'sujet', 'superpose', 'sur', 'surtout', 't', 'ta', 'tac', 'tandis', 'tant', 'tardive', 'te', 'tel', 'telle', 'tellement', 'telles', 'tels', 'tenant', 'tend', 'tenir', 'tente', 'tes', 'tic', 'tien', 'tienne', 'tiennes', 'tiens', 'toc', 'toi', 'toi-même', 'ton', 'touchant', 'toujours', 'tous', 'tout', 'toute', 'toutefois', 'toutes', 'treize', 'trente', 'tres', 'trois', 'troisième', 'troisièmement', 'trop', 'très', 'tsoin', 'tsouin', 'tu', 'té', 'u', 'un', 'une', 'unes', 'uniformement', 'unique', 'uniques', 'uns', 'v', 'va', 'vais', 'valeur', 'vas', 'vers', 'via', 'vif', 'vifs', 'vingt', 'vivat', 'vive', 'vives', 'vlan', 'voici', 'voie', 'voient', 'voilà', 'vont', 'vos', 'votre', 'vous', 'vous-mêmes', 'vu', 'vé', 'vôtre', 'vôtres', 'w', 'x', 'y', 'z', 'zut', 'à', 'â', 'ça', 'ès', 'étaient', 'étais', 'était', 'étant', 'état', 'étiez', 'étions', 'été', 'étée', 'étées', 'étés', 'êtes', 'être', 'ô', \"a\", \"ai\", \"aie\", \"aient\", \"aies\", \"ait\", \"alors\", \"as\", \"au\", \"aucun\", \"aura\", \"aurai\", \"auraient\", \"aurais\", \"aurait\", \"auras\", \"aurez\", \"auriez\", \"aurions\", \"aurons\", \"auront\", \"aussi\", \"autre\", \"aux\", \"avaient\", \"avais\", \"avait\", \"avant\", \"avec\", \"avez\", \"aviez\", \"avions\", \"avoir\", \"avons\", \"ayant\", \"ayez\", \"ayons\", \"bon\", \"car\", \"ce\", \"ceci\", \"cela\", \"ces\", \"cet\", \"cette\", \"ceux\", \"chaque\", \"ci\", \"comme\", \"comment\", \"d\", \"dans\", \"de\", \"dedans\", \"dehors\", \"depuis\", \"des\", \"deux\", \"devoir\", \"devrait\", \"devrez\", \"devriez\", \"devrions\", \"devrons\", \"devront\", \"dois\", \"doit\", \"donc\", \"dos\", \"droite\", \"du\", \"dès\", \"début\", \"dù\", \"elle\", \"elles\", \"en\", \"encore\", \"es\", \"est\", \"et\", \"eu\", \"eue\", \"eues\", \"eurent\", \"eus\", \"eusse\", \"eussent\", \"eusses\", \"eussiez\", \"eussions\", \"eut\", \"eux\", \"eûmes\", \"eût\", \"eûtes\", \"faire\", \"fais\", \"faisez\", \"fait\", \"faites\", \"fois\", \"font\", \"force\", \"furent\", \"fus\", \"fusse\", \"fussent\", \"fusses\", \"fussiez\", \"fussions\", \"fut\", \"fûmes\", \"fût\", \"fûtes\", \"haut\", \"hors\", \"ici\", \"il\", \"ils\", \"j\", \"je\", \"juste\", \"l\", \"la\", \"le\", \"les\", \"leur\", \"leurs\", \"lui\", \"là\", \"m\", \"ma\", \"maintenant\", \"mais\", \"me\", \"mes\", \"moi\", \"moins\", \"mon\", \"mot\", \"même\", \"n\", \"ne\", \"ni\", \"nom\", \"nommé\", \"nommée\", \"nommés\", \"nos\", \"notre\", \"nous\", \"nouveau\", \"nouveaux\", \"on\", \"ont\", \"ou\", \"où\", \"par\", \"parce\", \"parole\", \"pas\", \"personne\", \"personnes\", \"peu\", \"peut\", \"plupart\", \"pour\", \"pourquoi\", \"qu\", \"quand\", \"que\", \"quel\", \"quelle\", \"quelles\", \"quels\", \"qui\", \"sa\", \"sans\", \"se\", \"sera\", \"serai\", \"seraient\", \"serais\", \"serait\", \"seras\", \"serez\", \"seriez\", \"serions\", \"serons\", \"seront\", \"ses\", \"seulement\", \"si\", \"sien\", \"soi\", \"soient\", \"sois\", \"soit\", \"sommes\", \"son\", \"sont\", \"sous\", \"soyez\", \"soyons\", \"suis\", \"sujet\", \"sur\", \"t\", \"ta\", \"tandis\", \"te\", \"tellement\", \"tels\", \"tes\", \"toi\", \"ton\", \"tous\", \"tout\", \"trop\", \"très\", \"tu\", \"un\", \"une\", \"valeur\", \"voient\", \"vois\", \"voit\", \"vont\", \"vos\", \"votre\", \"vous\", \"vu\", \"y\", \"à\", \"ça\", \"étaient\", \"étais\", \"était\", \"étant\", \"état\", \"étiez\", \"étions\", \"été\", \"étés\", \"êtes\", \"être\"}\n",
    "\n",
    "def tokenize(text):\n",
    "    return nltk.word_tokenize(re.sub(r'\\W+', ' ', text.lower()), language='french')\n",
    "\n",
    "def clean(tokens):\n",
    "    # Stop words\n",
    "    data = [word for word in tokens if word not in STOPWORDS]\n",
    "    \n",
    "    # Symbols\n",
    "    table = str.maketrans('', '', string.punctuation + \"`’0123456789\")\n",
    "    data = [w.translate(table) for w in data]\n",
    "    \n",
    "    # Useless words\n",
    "    data = [w for w in data if len(w) < 15 and len(w) > 1]\n",
    "    \n",
    "    # Whitespaces\n",
    "    data = [w for w in data if w != '']\n",
    "    \n",
    "    return data\n",
    "\n",
    "def add_syns(tokens, syns_number):\n",
    "    # Adds up to syns_number synonyms for each word in tokens.\n",
    "    # On the hypothesis that more synonyms may help to increase qwant results precision.\n",
    "    syns = set([w for word in tokens for t in wordnet.synsets(word, lang='fra') for w in t.lemma_names('fra')[:syns_number]])\n",
    "    tokens += syns\n",
    "    return tokens\n",
    "\n",
    "def ngramize(tokens, n):\n",
    "    if n < 2:\n",
    "        return tokens\n",
    "    return ngrams(tokens, n)\n",
    "\n",
    "def count(ngrams):\n",
    "    return Counter(ngrams)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Traitement"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Une étape intermédiaire utile consiste ici à effectuer une petite analyse du corpus de test. On a déjà l'intuition que des informations tels que des mots gramaticaux, ou des symboles de poncuation, n'apporteraient rien dans une requête soumise à un moteur de recherche. Nous débarsserons donc les questions de l'ensemble de test avant de les soumettre à Qwant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 688,
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_query(text, s=1):\n",
    "    data = text\n",
    "    data = tokenize(data)\n",
    "    data = clean(data)\n",
    "    data = add_syns(data, s)\n",
    "    return ' '.join(data)\n",
    "\n",
    "def process_doc(text, n=3):\n",
    "    data = text\n",
    "    data = tokenize(data)\n",
    "    data = clean(data)\n",
    "    #data = ngramize(data, n)\n",
    "    return ' '.join(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analyse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On étudie succintement les données disponibles pour la phase de test. Pour cela, on effectue un compte de tous les mots constituant les requêtes à soumettre (questions après traitement de nettoyage). On remarque alors que certains mots reviennent très souvent, alors qu'il n'est aucunnement pertinent de les utiliser dans une recherche, comme les mots interrogatifs, des adverbes abstraits comme \"aujourd'hui\" ou \"quelqu'un\", des verbes généralistes comme \"dit\" ou \"affirme\", ou encore des noms faisant référence à des des types de ressources, comme \"article\" ou \"vidéo\".\n",
    "\n",
    "Il est compliqué d'effectuer une reconnaissance d'entitées nommées ou un traitement similaire, mais on peut gagner significativement en efficacité en éliminant ces termes inutiles.\n",
    "\n",
    "On les ajoute à la main à la liste des termes à supprimer (stopwords). On ne peut pas simplement éliminer les mots les plus courants, car le corpus est fondamentalement biaisé : des mots comme \"Macron\", \"France\", ou \"gilets\" et \"jaunes\" apparaissent un grand nombre de fois pour des raisons légitimes : il s'agit de sujet faisant l'objet de nombreuses actualités, et donc mécaniquement de nombreuses informations à vérifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 559,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('macron', 111),\n",
       " ('france', 110),\n",
       " ('va', 48),\n",
       " ('français', 43),\n",
       " ('euros', 36),\n",
       " ('gilets', 35),\n",
       " ('jaunes', 34),\n",
       " ('lors', 34),\n",
       " ('entre', 30),\n",
       " ('mélenchon', 29),\n",
       " ('paris', 27),\n",
       " ('emmanuel', 26),\n",
       " ('compte', 25),\n",
       " ('affaire', 22),\n",
       " ('après', 22),\n",
       " ('pendant', 22),\n",
       " ('trump', 22),\n",
       " ('gouvernement', 21),\n",
       " ('etat', 21),\n",
       " ('pays', 21),\n",
       " ('migrants', 20),\n",
       " ('jean', 20),\n",
       " ('homme', 19),\n",
       " ('police', 19),\n",
       " ('hausse', 18),\n",
       " ('campagne', 18),\n",
       " ('députés', 18),\n",
       " ('policiers', 18),\n",
       " ('dire', 17),\n",
       " ('concernant', 17)]"
      ]
     },
     "execution_count": 559,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Counter(process_query(' '.join(list(TEST_DATASET.question))).split(\" \")).most_common(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Requêtes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 489,
   "metadata": {},
   "outputs": [],
   "source": [
    "TEST_DATASET['request'] = None\n",
    "TEST_DATASET['request'] = [process_query(t[1].question, 0) for t in TEST_DATASET.iterrows()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 309,
   "metadata": {},
   "outputs": [],
   "source": [
    "TRAIN_DATASET['request'] = None\n",
    "TRAIN_DATASET['request'] = [process_query(t[1].question, 0) for t in TRAIN_DATASET.iterrows()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Appels API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 583,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 18 37 5 ]   Requête traitée : frais scolarité coûteront chers étudiant ... - 48 liens trouvés.\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-583-f447d3bdd894>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mindex\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrow\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mTEST_DATASET\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0miterrows\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mrow\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'links'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrow\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'links'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m         \u001b[0mrow\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'links'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mqwant\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrow\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<ipython-input-527-5f9eb4bda655>\u001b[0m in \u001b[0;36mqwant\u001b[0;34m(q)\u001b[0m\n\u001b[1;32m     14\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     15\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mqwant\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mq\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 16\u001b[0;31m     \u001b[0mresponse\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mqwant_raw\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mq\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     17\u001b[0m     \u001b[0mlinks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mr\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'url'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mr\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mresponse\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'result'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'items'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     18\u001b[0m     \u001b[0mnow\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdatetime\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdatetime\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnow\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-527-5f9eb4bda655>\u001b[0m in \u001b[0;36mqwant_raw\u001b[0;34m(q)\u001b[0m\n\u001b[1;32m      6\u001b[0m     \u001b[0mquery\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"user={}&pass={}&q={}\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0murlp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mquote\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mUSER\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0murlp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mquote\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mPASSWORD\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0murlp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mquote\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mq\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m     \u001b[0;31m#print(\"{}/query?{}\".format(BASEURL,query))\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 8\u001b[0;31m     \u001b[0mr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0murlr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0murlopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"{}/query?{}\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mBASEURL\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mquery\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      9\u001b[0m     \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdecode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"utf-8\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m     \u001b[0mresponse\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/lib/python3.7/urllib/request.py\u001b[0m in \u001b[0;36murlopen\u001b[0;34m(url, data, timeout, cafile, capath, cadefault, context)\u001b[0m\n\u001b[1;32m    220\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    221\u001b[0m         \u001b[0mopener\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_opener\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 222\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mopener\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdata\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    223\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    224\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0minstall_opener\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mopener\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/lib/python3.7/urllib/request.py\u001b[0m in \u001b[0;36mopen\u001b[0;34m(self, fullurl, data, timeout)\u001b[0m\n\u001b[1;32m    523\u001b[0m             \u001b[0mreq\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmeth\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mreq\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    524\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 525\u001b[0;31m         \u001b[0mresponse\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_open\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mreq\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdata\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    526\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    527\u001b[0m         \u001b[0;31m# post-process response\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/lib/python3.7/urllib/request.py\u001b[0m in \u001b[0;36m_open\u001b[0;34m(self, req, data)\u001b[0m\n\u001b[1;32m    541\u001b[0m         \u001b[0mprotocol\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mreq\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtype\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    542\u001b[0m         result = self._call_chain(self.handle_open, protocol, protocol +\n\u001b[0;32m--> 543\u001b[0;31m                                   '_open', req)\n\u001b[0m\u001b[1;32m    544\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    545\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/lib/python3.7/urllib/request.py\u001b[0m in \u001b[0;36m_call_chain\u001b[0;34m(self, chain, kind, meth_name, *args)\u001b[0m\n\u001b[1;32m    501\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mhandler\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mhandlers\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    502\u001b[0m             \u001b[0mfunc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgetattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhandler\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmeth_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 503\u001b[0;31m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    504\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mresult\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    505\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/lib/python3.7/urllib/request.py\u001b[0m in \u001b[0;36mhttp_open\u001b[0;34m(self, req)\u001b[0m\n\u001b[1;32m   1343\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1344\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mhttp_open\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreq\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1345\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdo_open\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhttp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mHTTPConnection\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreq\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1346\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1347\u001b[0m     \u001b[0mhttp_request\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mAbstractHTTPHandler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdo_request_\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/lib/python3.7/urllib/request.py\u001b[0m in \u001b[0;36mdo_open\u001b[0;34m(self, http_class, req, **http_conn_args)\u001b[0m\n\u001b[1;32m   1318\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mOSError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0merr\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;31m# timeout error\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1319\u001b[0m                 \u001b[0;32mraise\u001b[0m \u001b[0mURLError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0merr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1320\u001b[0;31m             \u001b[0mr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mh\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetresponse\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1321\u001b[0m         \u001b[0;32mexcept\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1322\u001b[0m             \u001b[0mh\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/lib/python3.7/http/client.py\u001b[0m in \u001b[0;36mgetresponse\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m   1319\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1320\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1321\u001b[0;31m                 \u001b[0mresponse\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbegin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1322\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mConnectionError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1323\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/lib/python3.7/http/client.py\u001b[0m in \u001b[0;36mbegin\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    294\u001b[0m         \u001b[0;31m# read until we get a non-100 response\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    295\u001b[0m         \u001b[0;32mwhile\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 296\u001b[0;31m             \u001b[0mversion\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstatus\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreason\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_read_status\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    297\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mstatus\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0mCONTINUE\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    298\u001b[0m                 \u001b[0;32mbreak\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/lib/python3.7/http/client.py\u001b[0m in \u001b[0;36m_read_status\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    255\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    256\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_read_status\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 257\u001b[0;31m         \u001b[0mline\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreadline\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_MAXLINE\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"iso-8859-1\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    258\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mline\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0m_MAXLINE\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    259\u001b[0m             \u001b[0;32mraise\u001b[0m \u001b[0mLineTooLong\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"status line\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/lib/python3.7/socket.py\u001b[0m in \u001b[0;36mreadinto\u001b[0;34m(self, b)\u001b[0m\n\u001b[1;32m    587\u001b[0m         \u001b[0;32mwhile\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    588\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 589\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_sock\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrecv_into\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    590\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    591\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_timeout_occurred\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "for index, row in TEST_DATASET.iterrows():\n",
    "    # If request has already succeded, dont do it again\n",
    "    # None indicates a request has never been treated.\n",
    "    # Empty list commonly indicates a temporary fail in qwant research process.\n",
    "    if row['links'] is None or len(row['links']) == 0:\n",
    "        row['links'] = qwant(row.request)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Traitement pages HTML"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Grâce au code fourni pour le hackathon, on peut effectuer une requête de recherche sur Qwant et récupérer le contenu des premiers résultats correspondants. Afin d'extraire le contenu intéressant de ces documents, on effectue un traitement de la page HTML."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 658,
   "metadata": {},
   "outputs": [],
   "source": [
    "def scrap(html):\n",
    "    limite = 100\n",
    " \n",
    "    def tag_visible(elementT):\n",
    "        # We delete everything that is irrevelant to the page content.\n",
    "        # FIXME: Maybe should we keep the title tag ?\n",
    "        for element in elementT.parents:\n",
    "            if element.name in ['style', 'script', 'head', 'title', 'meta', 'footer', 'header']:\n",
    "                return False\n",
    "        if isinstance(elementT, Comment):\n",
    "            return False\n",
    "        return True\n",
    " \n",
    "    def parse_article(soup):\n",
    "       \n",
    "        # Main article commonly starts after the (first) first-levle title\n",
    "        h1 = soup.find('h1')\n",
    "        fullText = \"\"\n",
    " \n",
    "        # Main content is commonly into <p> tags\n",
    "        root = soup.body\n",
    "        if not root.find('p'):\n",
    "            return \"\"\n",
    "        ps = root.find_all('p', recursive=True)\n",
    " \n",
    "        # The main content is often the most massive text bloc in the page.\n",
    "        # So we can look for where are the most <p> tags, to find the most important text\n",
    "        # Without any sentence-like texte recognition.\n",
    "        currMax = 0\n",
    "        currPmax = None\n",
    "        for p in ps:\n",
    "            #if p.has_key('class'):\n",
    "                #print(p.parent[\"class\"])\n",
    "            #print(len(p.parent.find_all('p', recursive=False)))\n",
    "            if len(p.parent.find_all('p', recursive=False)) > currMax:\n",
    "                currMax = len(p.parent.find_all('p', recursive=False))\n",
    "                currPmax = p.parent\n",
    " \n",
    "        if currMax == 0:\n",
    "            print('nope')\n",
    "            return \"\"\n",
    "        tex = currPmax.find_all(text=True)\n",
    "        visible_texts = filter(tag_visible, tex)  \n",
    "        fullText = u\" \".join(t.strip() for t in visible_texts)\n",
    " \n",
    "        return h1.text + \"\\n\" + fullText\n",
    " \n",
    "    def get_article(url):\n",
    "        res = requests.get(url)\n",
    "        return parse_article(res.content)\n",
    "   \n",
    "    soup = BeautifulSoup(html, 'html.parser')\n",
    "    res = parse_article(soup)\n",
    "    taille = len(res.split())\n",
    " \n",
    "    if taille < limite:\n",
    "        tex = soup.findAll(text=True)\n",
    " \n",
    "        visible_texts = filter(tag_visible, tex)  \n",
    "        textWrite = u\" \".join(t.strip() for t in visible_texts)\n",
    "        return textWrite\n",
    "    else:\n",
    "        return res\n",
    "\n",
    "#r = requests.get('https://fr.news.yahoo.com/brigitte-macron-pourquoi-pris-distances-143024731.html')\n",
    " \n",
    "#res = scrap(r.text)\n",
    "#print(res)\n",
    "#taille = len(res.split())\n",
    "#print(taille)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Diversité"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pour la sélection des passages les plus pertinents constitutifs de l'article de réponse à rendre (ou résumé), on se propose d'effectuer un travail de diversité sur les documents. L'idée est d'effectuer un rapide clustering sur les documents recupérés suite à la requête Qwant, et sélectionner le plus pertinent de chaque cluster. Les documents choisis serviront à constituer le résumé rendu.\n",
    "\n",
    "L'objectif est de maximiser la couverture d'information par le résumé tout en limitant sa longueur."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 614,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'documents' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-614-e5a41429453a>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     29\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     30\u001b[0m \u001b[0mvectorizer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mTfidfVectorizer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstop_words\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mSTOPWORDS\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 31\u001b[0;31m \u001b[0mX\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mvectorizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit_transform\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdocuments\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     32\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     33\u001b[0m \u001b[0mtrue_k\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m3\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'documents' is not defined"
     ]
    }
   ],
   "source": [
    "class Data(object):\n",
    "    def __init__(self, directory):\n",
    "        self.corpus, self.idx_2_name = self.create_corpus(directory)\n",
    "\n",
    "    # Haha, we used the same method name in the same time. This is oddly funny. Now process my burial please x)\n",
    "    def process_doc(self, doc_path):\n",
    "        pattern = re.compile('[\\W_]+')\n",
    "        with open(doc_path, 'r') as f:\n",
    "            d = f.readlines()\n",
    "            d = ' '.join(d)\n",
    "            d = pattern.sub(' ', d).strip().lower()#.split()\n",
    "        return d\n",
    "\n",
    "    def create_corpus(self, directory):\n",
    "        corpus = []\n",
    "        idx_2_name = {}\n",
    "\n",
    "        docs_filenames = os.listdir(directory)[:10]\n",
    "        docs_path = [os.path.join(directory, f) for f in docs_filenames]\n",
    "\n",
    "        for i, doc in enumerate(docs_filenames):\n",
    "            doc_name = doc.split('.txt')[0]\n",
    "            doc_path = os.path.join(directory, doc)\n",
    "            idx_2_name[i] = int(doc_name)\n",
    "            corpus.append(self.process_doc(doc_path))\n",
    "\n",
    "        return corpus, idx_2_name\n",
    "\n",
    "\n",
    "\n",
    "vectorizer = TfidfVectorizer(stop_words=STOPWORDS)\n",
    "X = vectorizer.fit_transform(documents)\n",
    "\n",
    "true_k = 3\n",
    "model = KMeans(n_clusters=true_k, init='k-means++', max_iter=100, n_init=1)\n",
    "model.fit(X)\n",
    "clusters = list(model.predict(X))\n",
    "\n",
    "\n",
    "pertinent_docs = []\n",
    "\n",
    "for i in range(true_k):\n",
    "    doc_idx = clusters.index(i)\n",
    "    pertinent_docs.append(documents[doc_idx])\n",
    "\n",
    "print(pertinent_docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Soumission"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 679,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Task 1 : submit for each question a list of revelant documents URL\n",
    "def submit_docs(data):\n",
    "    run = dict()\n",
    "    for index, row in TEST_DATASET.iterrows():\n",
    "        if row.links is None:\n",
    "            run[index] = []\n",
    "        else:\n",
    "            run[index] = row.links[:10]\n",
    "\n",
    "    query=\"user={}&pass={}\".format(urlp.quote(USER), urlp.quote(PASSWORD))\n",
    "    r = urlr.urlopen(\"http://192.168.1.1:8000/hackathon/submitrun?{}\".format(query), data=json.dumps(run).encode(\"utf-8\"))\n",
    "    response = r.read()\n",
    "    print(response)\n",
    "    \n",
    "# Task 2 : submit for each question one string that summurize the answer and the documents\n",
    "def submit_text(data, summarize):\n",
    "    run = dict()\n",
    "    for index, row in TEST_DATASET.iterrows():\n",
    "        run[index] = summarize(row)\n",
    "    \n",
    "\n",
    "    query=\"user={}&pass={}\".format(urlp.quote(USER), urlp.quote(PASSWORD))\n",
    "    r = urlr.urlopen(\"http://192.168.1.1:8000/hackathon/submitrun?{}\".format(query), data=json.dumps(run).encode(\"utf-8\"))\n",
    "    response = r.read()\n",
    "    print(response)\n",
    "\n",
    "def submit_docs_shiny(data):\n",
    "    run = dict()\n",
    "    for index, row in data.iterrows():\n",
    "        if row.links is None:\n",
    "            run[index] = \"\"\n",
    "        else:\n",
    "            xd = []\n",
    "            for doc in row.links[:2]:\n",
    "                try:\n",
    "                    xd.append(process_doc(scrap(get_doc(doc)))[:200])\n",
    "                except Exception:\n",
    "                    pass\n",
    "            #[process_doc(scrap(get_doc(doc))) for doc in row.links[:2]]\n",
    "            run[index] = \" \".join(xd)+row.request\n",
    "        print(\"Done\")\n",
    "\n",
    "    query=\"user={}&pass={}\".format(urlp.quote(USER), urlp.quote(PASSWORD))\n",
    "    r = urlr.urlopen(\"http://192.168.1.1:8000/hackathon/submitrun?{}\".format(query), data=json.dumps(run).encode(\"utf-8\"))\n",
    "    response = r.read()\n",
    "    print(response)\n",
    "    \n",
    "def get_start(data):\n",
    "    ' '.join(nltk.tokenize.tokenize(data)[:10])\n",
    "    \n",
    "def summarize_brutal(data):\n",
    "    return ' '.join([get_doc(doc) for doc in data[:3]])\n",
    "\n",
    "def summarize_shiny(data):\n",
    "    for doc in data:\n",
    "        text = scrap(get_doc(doc))\n",
    "        ' '.join(nltk.tokenize.tokenize(text)[:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 680,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done\n",
      "Done\n",
      "Done\n",
      "Done\n",
      "Done\n",
      "Done\n",
      "Done\n",
      "Done\n",
      "Done\n",
      "Done\n",
      "Done\n",
      "Done\n",
      "Done\n",
      "Done\n",
      "Done\n",
      "Done\n",
      "Done\n",
      "Done\n",
      "Done\n",
      "Done\n",
      "b'{\"response\": \"queued\"}'\n"
     ]
    }
   ],
   "source": [
    "submit_docs_shiny(TEST_DATASET.head(20))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Liste de documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 623,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'{\"response\": \"queued\"}'\n"
     ]
    }
   ],
   "source": [
    "submit_docs(TEST_DATASET.iloc[0:20])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Contenu des articles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 683,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'{\"response\": \"queued\"}'\n"
     ]
    }
   ],
   "source": [
    "def yolo(row):\n",
    "    return row['request']\n",
    "\n",
    "submit_text(TEST_DATASET, yolo)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 689,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "716"
      ]
     },
     "execution_count": 689,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Processed questions\n",
    "len([n for n in TEST_DATASET.links if n is not None and len(n) > 0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 690,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 690,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Failed questions\n",
    "len([n for n in TEST_DATASET.links if n is not None and len(n) == 0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 691,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "638"
      ]
     },
     "execution_count": 691,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Unrocessed questions\n",
    "len([n for n in TEST_DATASET.links if n is None])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 692,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1358"
      ]
     },
     "execution_count": 692,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# All questions\n",
    "len(TEST_DATASET.links)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 392,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'mamoudou gassama naturalisé régularisé'"
      ]
     },
     "execution_count": 392,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TEST_DATASET.request[89]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}