elasticsearching

setup elastic search

docker-compose up -d

minimal example to populate es-index

run populating_es_minimal_example.py

in some browser

1. goto

    http://<some_host>:5601/app/kibana#/dev_tools/console 
   

2. have fun with kibana console (formerly known as Sense)

   

semantic scholar

get semantic scholar data

wget https://s3-us-west-2.amazonaws.com/ai2-s2-research-public/open-corpus/2019-10-01/manifest.txt
wget -B https://s3-us-west-2.amazonaws.com/ai2-s2-research-public/open-corpus/2019-10-01/ -i manifest.txt

benchmarking indexing speed

indexing rate for 2 mio. documents

indexing Semantic Scholar Open Research Corpus

• indexing 177 million documents took little more than 7 hours with a rate of aprox. 7500 docs/sec