Simple python client for GROBID REST services

This Python client can be used to process in an efficient concurrent manner a set of PDF in a given directory by the GROBID service. Results are written in a given output directory and include the resulting XML TEI representation of the PDF.

Build and run

You need first to install and start the grobid service, latest stable version, see the documentation. It is assumed that the server will run on the address http://localhost:8070. You can change the server address by editing the file config.json.

Requirements

This client has been developed and tested with Python 3.5.

Install

Get the github repo (original):

git clone https://github.com/kermitt2/grobid-client-python

Get the github repo (this):

git clone https://github.com/ETretyakov/grobid-client-python

cd grobid-client-python

It is advised to setup first a virtual environment to avoid falling into one of these gloomy python dependency marshlands:

virtualenv --system-site-packages -p python3 env

source env/bin/activate

Usage and options

usage: grobid_client.py [-h] [--input INPUT] [--output OUTPUT]
                        [--config CONFIG] [--n N] [--generateIDs]
                        [--consolidate_header] [--consolidate_citations]
                        [--force]
                        service

Client for GROBID services

positional arguments:
  service               one of [processFulltextDocument,
                        processHeaderDocument, processReferences]

optional arguments:
  -h, --help            show this help message and exit
  --input INPUT         path to the directory containing PDF to process
  --output OUTPUT       path to the directory where to put the results (optional)
  --config CONFIG       path to the config file, default is ./config.json
  --n N                 concurrency for service usage
  --generateIDs         generate random xml:id to textual XML elements of the
                        result files
  --consolidate_header  call GROBID with consolidation of the metadata
                        extracted from the header
  --consolidate_citations
                        call GROBID with consolidation of the extracted
                        bibliographical references
  --force               force re-processing pdf input files when tei output
                        files already exist

Examples:

python3 grobid_client.py --input ~/tmp/in2 --output ~/tmp/out processFulltextDocument

This command will process all the PDF files present under the input directory recursively (files with extension .pdf only) with the processFulltextDocument service of GROBID, and write the resulting XML TEI files under the output directory, reusing the file name with a different file extension (.tei.xml), using the default 10 concurrent workers.

If --output is omitted, the resulting XML TEI documents will be produced alongside the PDF in the --input directory.

python3 grobid_client.py --input ~/tmp/in2 --output ~/tmp/out --n 20 processHeaderDocument

This command will process all the PDF files present in the input directory (files with extension .pdf only) with the processHeaderDocument service of GROBID, and write the resulting XML TEI files under the output directory, reusing the file name with a different file extension (.tei.xml), using 20 concurrent workers.

By default if an existing .tei.xml file is present in the output directory corresponding to a PDF in the input directory, this PDF will be skipped to avoid reprocessing several times the same PDF. To force the processing of PDF and over-write of existing TEI files, use the parameter --force.

Todo

Benchmarking with more files (e.g. million ISTEX PDF). Also implement existing GROBID services for text input (date, name, affiliation/address, raw bibliographical references, etc.). Better support for parameters (including elements where to put coordinates).

License and contact

Distributed under Apache 2.0 license.

Main author and contact: Patrice Lopez (patrice.lopez@science-miner.com)