Skip to content

Dylan-YD/scraperproject

Repository files navigation

Django Project

This application is a Django project which is a ad scraper.

Installation

  1. Clone the repository. git clone https://github.com/Dylan-YD/scraper.git
  2. Install Python on your machine.
  3. Navigate to the project folder. cd scraper
  4. Create virtual environment. python -m venv {environment name}
  5. Activate environment. cd {enviroment name}/Scripts activate or source bin/activate
  6. Install requirements. pip install -r requirements.txt
  7. Create .env file
  8. Put your secret key, allowed host url, aws information in .env file like .env.example
  9. Change baseURL variable in \crawler\static\settings.js if you want run project on custom url
  10. Run the server. python manage.py runserver

Usage

  1. Go to the home page. http://127.0.0.1:8000/home
  2. Enter the query of the ads you want to scrape.
  3. Click the button to scrape the website.
  4. The progress will be shown on the page.
  5. After the scraping is done, the result will be shown on the page http://127.0.0.1:8000/crawler/ads.

project structure

scraper
├── crawler
│   ├── admin.py
│   ├── apps.py
│   ├── __init__.py
│   ├── migrations
│   ├── models.py
│   ├── static
│   ├── templates
│   ├── tests.py
│   ├── urls.py
│   └── views.py
├── db.sqlite3
├── manage.py
├── README.md
├── requirements.txt
└── scraper
    ├── asgi.py
    ├── __init__.py
    ├── settings.py
    ├── urls.py
    └── wsgi.py

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published