python-selenium

(2021) See your Python code do web browsing on your screen with GUI. I highly recommend working with Linux (including virtual machines) or MacOs.

• Important note:

• Installation

• Usage

• Using Docker

• iPython Django Shell

• API

• Admin

• Cache react app & view templates

• Mail Server

• Misc

• Contributing

• License

Important note:

Before you try to scrape any website, go through its robots.txt file. You can access it via domainname/robots.txt. There, you will see a list of pages allowed and disallowed for scraping. You should not violate any terms of service of any website you scrape.

Installation

cp .env.example .env
python3 -m venv env && \
  source env/bin/activate
pip install -r requirements.txt
python3 manage.py migrate

Add chromedriver to Path

Make sure Chromedriver is installed and added to your environment Path.

# chromedriver_mac64
# chromedriver_win32
# See https://chromedriver.storage.googleapis.com
# for drivers list.
wget https://chromedriver.storage.googleapis.com/2.37/chromedriver_linux64.zip
unzip chromedriver_linux64.zip
sudo mv chromedriver /usr/local/bin/chromedriver
chromedriver --version

Usage

XPath cheat sheet.

Update the command at crawl.py

alias py="python3"
py manage.py crawl

If you still need help installing and running the app check out the readme at https://github.com/kkamara/python-react-boilerplate which is the base system for this python-selenium app.

Using Docker?

alias compose='docker-compose -f local.yml'
compose build
compose up
# Automated runs with Docker:
# compose up --build -d && python3 manage.py crawl

iPython Django Shell

py manage.py shell -i ipython

API

py manage.py show_urls

View the api collection here.

Admin

Admin creds are set in ./compose/local/django/start

export DJANGO_SUPERUSER_PASSWORD=secret

py manage.py createsuperuser \
  --username admin_user \
  --email admin@django-app.com \
  --no-input \
  --first_name Admin \
  --last_name User

Cache react app & view templates

py manage.py collectstatic

Mail Server

Mail environment credentials are at .env.

The mailhog docker image runs at http://localhost:8025.

Misc

See Python Amazon Scraper.

See Python ReactJS Boilerplate.

See PHP Scraper.

See PHP ReactJS Boilerplate.

See Amazon Scraper.

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update tests as appropriate.

License

BSD