Skip to content

Latest commit

 

History

History

examples

Folders and files

NameName
Last commit message
Last commit date

parent directory

..

Cargo.toml

Cargo.toml

 
 

README.md

README.md

 
 

advanced_configuration.rs

advanced_configuration.rs

 
 

budget.rs

budget.rs

 
 

cache.rs

cache.rs

 
 

cache_chrome_hybrid.rs

cache_chrome_hybrid.rs

 
 

callback.rs

callback.rs

 
 

chrome.rs

chrome.rs

 
 

chrome_screenshot.rs

chrome_screenshot.rs

 
 

chrome_screenshot_with_config.rs

chrome_screenshot_with_config.rs

 
 

chrome_viewport.rs

chrome_viewport.rs

 
 

configuration.rs

configuration.rs

 
 

cron.rs

cron.rs

 
 

debug.rs

debug.rs

 
 

depth.rs

depth.rs

 
 

download.rs

download.rs

 
 

download_to_react.rs

download_to_react.rs

 
 

encoding.rs

encoding.rs

 
 

example.rs

example.rs

 
 

loop.rs

loop.rs

 
 

openai.rs

openai.rs

 
 

openai_cache.rs

openai_cache.rs

 
 

openai_extra.rs

openai_extra.rs

 
 

openai_multi.rs

openai_multi.rs

 
 

queue.rs

queue.rs

 
 

rss.rs

rss.rs

 
 

scrape.rs

scrape.rs

 
 

serde.rs

serde.rs

 
 

sitemap.rs

sitemap.rs

 
 

smart.rs

smart.rs

 
 

subscribe.rs

subscribe.rs

 
 

subscribe_download.rs

subscribe_download.rs

 
 

url_glob.rs

url_glob.rs

 
 

url_glob_subdomains.rs

url_glob_subdomains.rs

 
 

README.md

Examples

First git clone https://github.com/spider-rs/spider.git and cd spider. Use the release flag for the best performance --release when running the examples below.

Basic

Simple concurrent crawl Simple.

  • cargo run --example example

Subscribe to realtime changes Subscribe.

  • cargo run --example subscribe

Live handle index mutation example Callback.

  • cargo run --example callback

Enable log output Debug.

  • cargo run --example debug

Scrape the webpage with and gather html Scrape.

  • cargo run --example scrape

Scrape and download the html file to fs Download HTML. *Note: Enable feature flag [full_resources] to gather all files like css, jss, and etc.

  • cargo run --example download

Scrape and download html to react components and store to fs Download to React Component.

  • cargo run --example download_to_react

Crawl the page and output the links via Serde.

  • cargo run --example serde --features serde

Crawl links with a budget of amount of pages allowed Budget.

  • cargo run --example budget

Crawl links at a given cron time Cron.

  • cargo run --example cron

Crawl links with chrome headed rendering Chrome.

  • cargo run --example chrome --features chrome_headed

Crawl links with view port configuration Chrome Viewport.

  • cargo run --example chrome_viewport --features chrome

Take a screenshot of a page during crawl Chrome Screenshot.

  • cargo run --example chrome_screenshot --features="spider/sync spider/chrome spider/chrome_store_page"

Crawl links with smart mode detection. Runs HTTP by default until Chrome Rendering is needed. Smart.

  • cargo run --example smart --features smart

Use different encodings for the page. Encoding.

  • cargo run --example encoding --features encoding

Use advanced configuration re-use. Advanced Configuration.

  • cargo run --example cache_chrome_hybrid --features="spider/sync spider/chrome spider/cache_chrome_hybrid"

Use chrome hybrid caching. Chrome Cache Hybrid.

  • cargo run --example advanced_configuration

Use URL globbing for a domain. URL Globbing.

  • cargo run --example glob --features glob

Use URL globbing for a domain and subdomains. URL Globbing Subdomains.

  • cargo run --example url_glob_subdomains --features glob

Downloading files in a subscription. Subscribe Download.

  • cargo run --example subscribe_download

Add links to gather mid crawl. Queue.

  • cargo run --example queue

Use OpenAI to get custom Javascript to run in a browser. OpenAI. Make sure to set OPENAI_API_KEY=$MY_KEY as an env variable or pass it in before the script.

  • cargo run --example openai

or

  • OPENAI_API_KEY=replace_me_with_key cargo run --example openai

or setting multiple actions to drive the browser

  • OPENAI_API_KEY=replace_me_with_key cargo run --example openai_multi

or to get custom data from the GPT with JS scripts if needed.

  • OPENAI_API_KEY=replace_me_with_key cargo run --example openai_extra