A simple Django project to illustrate the use of Celery

I created this project to accompany my blog post at https://importthis.tech/djangocelery-from-development-to-production

Table of Contents generated with DocToc

• Prerequisites
• Project Setup
• Run the project
  • without celery
  • with celery
• Tests
• Production-ready

Prerequisites

• A *nix environment.
• Python 3.6+. I use virtualenvwrapper for managing python virtual environments.
• Node.js 12+ with the following packages installed globally (depending on your Node.js setup, you might need to install as superuser):
  • gulp – npm install -g gulp-cli
  • concurrently: npm install -g concurrently
  • MailDev – npm install -g maildev
  • Sass: npm install -g sass
  • prettier: npm install prettier -g
  • Browsersync: npm install -g browser-sync
  • commitizen – npm install -g commitizen
  • DocToc: npm install -g doctoc
• yarn: See installation instructions for your platform.
• redis must be installed and configured on your development machine.

Project Setup

First, clone the project and cd into the cloned directory:

git clone https://github.com/engineervix/django-celery-sample.git
cd django-celery-sample

Then, assuming that you have created and activated your virtual environment, install the python dependencies. This project uses pip-tools to manage python dependencies.

pip install pip-tools && pip-sync
# or you could just `pip install -r requirements.txt`

Install the Node.js dependencies and copy the vendor libraries to the static directory:

yarn install && gulp cp

Create a postgres database and user for the project. If you are using tools such as PGAdmin or Postgres.app, you please feel free to use them according to their documentation. If you are using the CLI like me, you could do it as follows:

# assuming your DATABASE is my_DB
# assuming USER is my_user
# assuming your PASSWORD is my_password
psql -c "CREATE USER my_user PASSWORD 'my_password'" \
&& psql -c "CREATE DATABASE my_DB OWNER my_user" \
&& psql -c "GRANT ALL PRIVILEGES ON DATABASE my_DB TO my_user"

In order to simplify this, I wrote a simple bash script which provides a command create_db, and placed it in my $PATH. It prompts me for the database and the user names, and generates a random password. I should probably add it to this repo as a utility.

Now that your database is set up, it's time to set up your environment variables. This repo contains a direcctory .envs.sample which has sample .env files for you to build on and customize.

# first, rename the `.envs.sample` directory to `.envs` 
mv -v .envs.sample/ .envs/

# then, let's remove the .sample suffix from all the `*.env.sample` files in the renamed directory
for i in $(ls -a .envs/ | grep sample);do mv -i .envs.sample/$i .envs.sample/`basename $i .sample`; done

There are three .env files:

1. .dev.env – for the development environment
2. .test.env – for the test environment
3. .prod.env – for the production environment

Edit those files and update the environment variables accordingly. The table below shows the environment variables that need to be updated. For now, you can skip the environment variables for production, and only update them when you are ready to go into production.

Please note that, in production, this project uses

• Sendgrid for sending emails via django-anymail. You can use your preferred provider and update both the production settings and environment variables accordingly.
• Sentry for error tracking. You'll have to setup an account (if you don't have one already) and register the project.

	development	test	production
1	DJANGO_SECRET_KEY	DJANGO_SECRET_KEY	DJANGO_SECRET_KEY
2	DATABASE_URL		DATABASE_URL
3			EMAIL_RECIPIENTS
4			DEFAULT_FROM_EMAIL
5			ALLOWED_HOSTS
6			BASE_URL
7			SENDGRID_API_KEY
8			SENTRY_DSN

Okay, now that you have installed all dependencies and have set up your database and environment variables, you can now make migrations and create the superuser in readiness to run the project.

./manage.py makemigrations && ./manage.py migrate
./manage.py createsuperuser

We are now ready to run!

Run the project

without celery

yarn dev

with celery

yarn dev:celery

If all goes well, this will launch two tabs in your default browser – a maildev tab and a django tab with today's date and a quote for today, as shown in the screenshot below:

The Browsersync and gulp setup provides for automatic restarting of the dev server and autoreload of the browser, so you can work on the project and make changes to the files without having to do this manually.

Tests

This project uses pytest and the initial tests should give you about 93% test coverage.

yarn test

Production-ready

Going into production shouldn't be too complicated, as the project includes production-ready configurations right from the start:

• I have already mentioned the .prod.env file,
• There's the config/settings/production.py file,
• I created a separate wsgi_production.py file and
• There are some celery and celery beat configuration files in the .envs.example/celery directory. These will come in handy if you're using systemd. I wrote a blog post that describes how to daemonize celery and celery beat using systemd.