Skip to content

Distributed web data discovery and collection framework built for serverless

License

Notifications You must be signed in to change notification settings

achannarasappa/locust

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Build Status Coverage Status

Locust

Distributed web data discovery and collection framework

Quick Start

npm install @achannarasappa/locust

Features

  • Configuration driven jobs
  • Distributed execution model to support serverless architectures
  • Handle client-side JavaScript execution
  • Data extraction using CSS selectors
  • Depth-based stop condition along with support for custom stop conditions
  • Robust dev tooling with locust-cli to build and test jobs

Use Cases

  • Web indexing (i.e. web crawling)
  • Web data extraction (i.e. web scraping)

Reference

About

Distributed web data discovery and collection framework built for serverless

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages