Skip to content
This repository has been archived by the owner on Jul 17, 2020. It is now read-only.

A tool for scraping tweet ids from the Twitter website.

License

Notifications You must be signed in to change notification settings

justinlittman/twitter-scraper

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

9 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

twitter-scraper

A tool for scraping tweet ids from the Twitter website.

Explanation

Tweets collected from Twitter's APIs provide metadata about the tweets that is machine-readable (in JSON) and may not be available from the website. For the purposes of archiving and/or analyzing tweets this metadata is potentially significant.

Unfortunately, Twitter's APIs don't support getting a comprehensive set of a user's tweets. Twitter's statuses/user_timeline REST API method only allows collecting the last 3,200 tweets. Similarly, Twitter's Search API will only provide tweets from the last 6-9 days.

twitter-scraper attempts to support getting a comprehensive set of a user's tweets (with optional date constraints). It accomplishes this by making requests to Twitter's website search (which is different than the Search API) and extracting tweet ids. These tweet ids can then be passed to twarc to retrieve from Twitter's REST API (aka "hydrating").

Installation

  1. Install Python, Pip, Git, and Chrome.

  2. Clone this repo: git clone https://github.com/justinlittman/twitter-scraper.git

  3. Install ChromeDriver. On a Mac, this can be done with brew install chromedriver. On Ubuntu:

     wget http://chromedriver.storage.googleapis.com/2.24/chromedriver_linux64.zip
     unzip chromedriver_linux64.zip
     sudo mv chromedriver /usr/bin/
    
  4. Install Selenium: pip install selenium

  5. Install Twarc: pip install twarc

Usage

    python twitter_scraper.py -h
    usage: twitter_scraper.py [-h] [--since SINCE] [--until UNTIL]
                              [--exclude-retweets] [--delta-days DELTA_DAYS]
                              [--wait-secs WAIT_SECS] [--debug]
                              screen_name
    
    positional arguments:
      screen_name
    
    optional arguments:
      -h, --help            show this help message and exit
      --since SINCE         Tweets since this date. Default is 2011-04-05.
      --until UNTIL         Tweets until this date. Default is today.
      --exclude-retweets
      --delta-days DELTA_DAYS
                            Number of days to include in each search.
      --wait-secs WAIT_SECS
                            Number of seconds to wait between each scroll.
      --debug

Running

To collect @realDonaldTrump's tweets between January 1, 2016 and April 1, 2016:

  1. Run twitter_scraper and write the tweet ids to a file.

     python twitter_scraper.py @realDonaldTrump --since=2016-01-01 --until=2016-04-01 > tweet_ids.txt
    

    Leave your system alone while twitter_scraper is running. I received inconsistent results while I was doing other work on my system while twitter_scraper was running. Better yet, use a VM.

    Tip: You can get the date that a user joined Twitter from the user's account page.

    Tip: Timestamp the tweet id file by using > tweet_ids_$(date -d "today" +"%Y%m%d%H%M").txt

    Tip: To account for the inconsistency, you may want to run multiple scrapes. They can be merged with:

     cat tweet_ids_*.txt | sort -u > tweet_ids.txt
    
  2. Hydrate the tweet ids with Twarc and write to a file. You will need to provide Twarc with a set of Twitter API keys. For more information, see Twarc's documentation.

     twarc.py --hydrate tweet_ids.txt > tweets.json
    

Acknowledgements

This work was inspired by the Trump Twitter Archive.

And appreciative (once again) for Ed Summer's twarc.

About

A tool for scraping tweet ids from the Twitter website.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages