Bibliography/PubPeer/notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Find articles with PubPeer comments into your Mendeley database \n",
    "\n",
    "[PubPeer](https://pubpeer.com/) is a tool that allow researchers to comment published articles. It's not about denunciation ! At least I don't see it that way, it's about **post publications reviewing** and should well taken by all researchers.\n",
    "\n",
    "I think post published articles reviewing is **healthy exercise** and should be encouraged. According to [this blog post](http://blog.pubpeer.com/?p=164) from PubPeer, some article definitely needs to be reviewed after publication (see the blog post for more details, some are [really suprising](https://pubpeer.com/publications/058CFA77EAF6D5E019D9902C6B3553)).\n",
    "\n",
    "Moreover PubPeer is not always about pointing out some unintentional mistakes or intentional bad behaviours but can also be a starting point for a **fruitful discussion** between authors and readers.\n",
    "\n",
    "Let's go to the fun part now : **how can I find in my Mendeley database articles with comments on PubPeer ?**\n",
    "\n",
    "Note 1: I did a Python script but it could be really easy to build a small web app and make the process a lot of easier for everyone access.\n",
    "\n",
    "Note 2: I just discovered https://peerlibrary.org/ which seems to do a similar job as PubPeer and they also have an API !\n",
    "\n",
    "Any feedback are welcome !\n",
    "\n",
    "[@hadim_](https://twitter.com/hadim_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import requests\n",
    "from lxml import html\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import tqdm\n",
    "\n",
    "from mendeley import Mendeley\n",
    "from requests_oauthlib import OAuth2Session\n",
    "from mendeley.session import MendeleySession"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get Mendeley API auth parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First you will need to generate a **client ID** and **client secret** from there : http://dev.mendeley.com/myapps.html.\n",
    "\n",
    "\n",
    "Then, put your **client ID** in **client secret** here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "client_id = \"1988\"\n",
    "client_secret = \"CXhCJQKZ8HUrtFtg\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that they are my personal credentials and at the time you read it they will be obsolete."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's start the auth process to the Mendeley API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Please go to https://api.mendeley.com/oauth/authorize?response_type=code&client_id=1988&redirect_uri=https%3A%2F%2Flocalhost&scope=all&state=3sX7ggAfEip4OxnGff9pOfYqNb1BTM&access_type=offline&approval_prompt=force and authorize access.\n"
     ]
    }
   ],
   "source": [
    "redirect_uri = 'https://localhost'\n",
    "\n",
    "authorization_base_url = \"https://api.mendeley.com/oauth/authorize\"\n",
    "token_url = \"https://api.mendeley.com/oauth/token\"\n",
    "\n",
    "oauth = OAuth2Session(client_id, redirect_uri=redirect_uri, scope=['all'])\n",
    "\n",
    "authorization_url, state = oauth.authorization_url(authorization_base_url,\n",
    "                                                   access_type=\"offline\",\n",
    "                                                   approval_prompt=\"force\")\n",
    "\n",
    "print('Please go to {} and authorize access.'.format(authorization_url))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now paste the fallback url here :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "authorization_code = \"https://localhost/?code=6fBBP91iqtnu-xPdTlsqCDVroYA&state=3sX7ggAfEip4OxnGff9pOfYqNb1BTM\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Authenticate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "token = oauth.fetch_token(token_url, authorization_response=authorization_code, client_secret=client_secret)\n",
    "\n",
    "mendeley = Mendeley(client_id, client_secret, redirect_uri=redirect_uri)\n",
    "session = MendeleySession(mendeley, token=token)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Iterate over all your articles and record them into a Pandas Dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                   "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You have 181 articles with correct identifiers (DOI or PMID)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r"
     ]
    }
   ],
   "source": [
    "articles = []\n",
    "\n",
    "all_documents = session.documents.list()\n",
    "for doc in tqdm.tqdm(session.documents.iter(), total=all_documents.count):\n",
    "    \n",
    "    if doc.identifiers:\n",
    "        d = {}\n",
    "        d['title'] = doc.title\n",
    "        d['year'] = doc.year\n",
    "        d['source'] = doc.source\n",
    "        d['doi'] = doc.identifiers['doi'] if 'doi' in doc.identifiers.keys() else None\n",
    "        d['pmid'] = doc.identifiers['pmid'] if 'pmid' in doc.identifiers.keys() else None\n",
    "\n",
    "        if doc.authors:\n",
    "            authors = [\"{}, {}\".format(author.first_name, author.last_name) for author in doc.authors]\n",
    "            d['authors'] = \" - \".join(authors)\n",
    "\n",
    "        articles.append(d)\n",
    "        \n",
    "articles = pd.DataFrame(articles)\n",
    "\n",
    "print(\"You have {} articles with correct identifiers (DOI or PMID)\".format(articles.shape[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Lets find matches with PubPeer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "2\n",
      "3\n",
      "4\n",
      "5\n",
      "6\n",
      "7\n",
      "8\n",
      "9\n",
      "10\n",
      "11\n",
      "12\n",
      "13\n",
      "14\n",
      "15\n",
      "16\n",
      "17\n",
      "18\n",
      "19\n",
      "20\n",
      "21\n",
      "22\n",
      "23\n",
      "24\n",
      "25\n",
      "26\n",
      "27\n",
      "28\n",
      "29\n",
      "30\n",
      "31\n",
      "32\n",
      "33\n",
      "34\n",
      "35\n",
      "36\n",
      "37\n",
      "38\n",
      "39\n",
      "40\n",
      "41\n",
      "42\n",
      "43\n",
      "44\n",
      "45\n",
      "46\n",
      "47\n",
      "48\n",
      "49\n",
      "50\n",
      "51\n",
      "52\n",
      "53\n",
      "54\n",
      "55\n",
      "56\n",
      "57\n",
      "58\n",
      "59\n",
      "60\n",
      "61\n",
      "62\n",
      "63\n",
      "64\n",
      "65\n",
      "66\n",
      "67\n",
      "68\n",
      "69\n",
      "70\n",
      "71\n",
      "72\n",
      "73\n",
      "74\n",
      "75\n",
      "76\n",
      "77\n",
      "78\n",
      "79\n",
      "80\n",
      "81\n",
      "82\n",
      "83\n",
      "84\n",
      "85\n",
      "86\n",
      "87\n",
      "88\n",
      "89\n",
      "90\n",
      "91\n",
      "92\n",
      "93\n",
      "94\n",
      "95\n",
      "96\n",
      "97\n",
      "98\n",
      "99\n",
      "100\n",
      "101\n",
      "102\n",
      "103\n",
      "104\n",
      "105\n",
      "106\n",
      "107\n",
      "108\n",
      "109\n",
      "110\n",
      "111\n",
      "112\n",
      "113\n",
      "114\n",
      "115\n",
      "116\n",
      "117\n",
      "118\n",
      "119\n",
      "120\n",
      "121\n",
      "122\n",
      "123\n",
      "124\n",
      "125\n",
      "126\n",
      "127\n",
      "128\n",
      "129\n",
      "130\n",
      "131\n",
      "132\n",
      "133\n",
      "134\n",
      "135\n",
      "136\n",
      "137\n",
      "138\n",
      "139\n",
      "140\n",
      "141\n",
      "142\n",
      "143\n",
      "144\n",
      "145\n",
      "146\n",
      "147\n",
      "148\n",
      "149\n",
      "150\n",
      "151\n",
      "152\n",
      "153\n",
      "154\n",
      "155\n",
      "156\n",
      "157\n",
      "158\n",
      "159\n",
      "160\n",
      "161\n",
      "162\n",
      "163\n",
      "164\n",
      "165\n",
      "166\n",
      "167\n",
      "168\n",
      "169\n",
      "170\n",
      "171\n",
      "172\n",
      "173\n",
      "174\n",
      "175\n",
      "176\n",
      "177\n",
      "178\n"
     ]
    }
   ],
   "source": [
    "pd.options.mode.chained_assignment = None\n",
    "\n",
    "#articles.loc[0, 'doi'] = \"10.5772/22496\"\n",
    "articles['comments'] = 0\n",
    "articles['comments_link'] = None\n",
    "\n",
    "old_n = -1\n",
    "for i in range(1, 179):\n",
    "    \n",
    "    print(i)\n",
    "    url = \"http://api.pubpeer.com/v1/publications/dump/{page}?devkey=9bb8f08ebef172ec518f5a4504344ceb\"\n",
    "    r = requests.get(url.format(page=i))\n",
    "    \n",
    "    all_pub = r.json()['publications']\n",
    "    if all_pub:\n",
    "        for pp in all_pub:\n",
    "            if 'doi' in pp.keys():\n",
    "                articles.loc[:, 'comments'][articles['doi'] == pp['doi']] += 1\n",
    "                articles.loc[:, 'comments_link'][articles['doi'] == pp['doi']] = pp['link']\n",
    "\n",
    "                n_comm = (articles['comments'] >= 1).sum()\n",
    "                if n_comm > 0 and n_comm > old_n:\n",
    "                    print(\"Commented articles = {}\".format(n_comm))\n",
    "                    old_n = n_comm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>authors</th>\n",
       "      <th>doi</th>\n",
       "      <th>pmid</th>\n",
       "      <th>source</th>\n",
       "      <th>title</th>\n",
       "      <th>year</th>\n",
       "      <th>comments</th>\n",
       "      <th>comments_link</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [authors, doi, pmid, source, title, year, comments, comments_link]\n",
       "Index: []"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "articles[articles['comments'] >= 1]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}