-
-
Notifications
You must be signed in to change notification settings - Fork 136
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implementation of the custom email backend to handle SES events #1994
Merged
mlissner
merged 19 commits into
1942_ses_sending_email_notifications
from
ses_custom_email_backend
Apr 6, 2022
Merged
Changes from 11 commits
Commits
Show all changes
19 commits
Select commit
Hold shift + click to select a range
55d89bc
feat(email): Handle SNS notifications for emails sent
albertisfu e1e1b57
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 7fa4b39
feat(email): Build dependencies for django-ses
albertisfu a5250fb
feat(email): Handle hard bounces and delivery notifications
albertisfu 67036c8
feat(email): Solves merge conflicts with main
albertisfu 77a1205
fix(email): Solves rebase conflicts with poetry deps
albertisfu 4fe1bcf
fix(email): Solves changes requested
albertisfu 612d1b5
fix(email): Fixes comments on models
albertisfu 2ba85d2
fix(email): Solves race condition on avoid duplicates
albertisfu d7958e7
fix(email): Solves rebase conflicts with main and django-ses updated
albertisfu 865d507
feat(email): Custom email backend implementation
albertisfu 2a6dbb9
Merge branch '1942_ses_sending_email_notifications' into ses_custom_e…
albertisfu 8a3ce25
fix(email): Improvements and django-ses updated
albertisfu 03419b4
fix(email): Function refactoring and improvements
albertisfu 40744d8
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] d1c99f9
fix(email): Black updated
albertisfu ccc3303
feat(typing): Add hints
mlissner 69631a3
fix(admin): Add raw_id_fields
mlissner 0289364
fix(typing): Add hints and refactor convert_list_to_str
mlissner File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
default_app_config = "cl.users.apps.UsersConfig" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
from django.apps import AppConfig | ||
|
||
|
||
class UsersConfig(AppConfig): | ||
name = "cl.users" | ||
|
||
def ready(self): | ||
# Implicitly connect a signal handlers decorated with @receiver. | ||
from cl.users import signals |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,317 @@ | ||
from threading import local | ||
|
||
from django.conf import settings | ||
from django.core.mail import ( | ||
EmailMessage, | ||
EmailMultiAlternatives, | ||
get_connection, | ||
) | ||
from django.core.mail.backends.base import BaseEmailBackend | ||
|
||
from cl.lib.command_utils import logger | ||
from cl.users.email_handlers import ( | ||
convert_list_to_str, | ||
get_email_body, | ||
store_message, | ||
) | ||
from cl.users.models import OBJECT_TYPES, BackoffEvent, EmailFlag | ||
|
||
|
||
class ConnectionHandler: | ||
"""This is an email backend connection handler, based on | ||
django-post-office/connections.py | ||
""" | ||
mlissner marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
def __init__(self): | ||
self._connections = local() | ||
|
||
def __getitem__(self, alias): | ||
try: | ||
return self._connections.connections[alias] | ||
except AttributeError: | ||
self._connections.connections = {} | ||
except KeyError: | ||
pass | ||
|
||
backend = settings.BASE_BACKEND | ||
connection = get_connection(backend) | ||
connection.open() | ||
self._connections.connections[alias] = connection | ||
return connection | ||
|
||
def all(self): | ||
return getattr(self._connections, "connections", {}).values() | ||
|
||
def close(self): | ||
for connection in self.all(): | ||
connection.close() | ||
|
||
|
||
connections = ConnectionHandler() | ||
|
||
|
||
class EmailBackend(BaseEmailBackend): | ||
"""This is a custom email backend to handle sending an email with some | ||
extra functions before the email is sent. | ||
|
||
Is neccesary to set the following settings: | ||
BASE_BACKEND: The base email backend to use to send emails, in our case: | ||
django_ses.SESBackend, for testing is used: | ||
django.core.mail.backends.locmem.EmailBackend | ||
MAX_ATTACHMENT_SIZE: The maximum file size in bytes that an attachment can | ||
have to be sent. | ||
|
||
- Verifies if the recipient's email address is not banned before sending | ||
or storing the message. | ||
- Stores a message in DB, generates a unique message_id. | ||
- Verifies if the recipient's email address is under a backoff event | ||
waiting period. | ||
- Verifies if the recipient's email address is small_email_only flagged | ||
or if attachments exceed MAX_ATTACHMENT_SIZE, if so send the small email | ||
version. | ||
- Compose messages according to available content types | ||
""" | ||
|
||
def open(self): | ||
pass | ||
|
||
def close(self): | ||
pass | ||
|
||
def send_messages(self, email_messages): | ||
|
||
if not email_messages: | ||
return | ||
|
||
for email_message in email_messages: | ||
subject = email_message.subject | ||
from_email = email_message.from_email | ||
to = email_message.to | ||
bcc = email_message.bcc | ||
cc = email_message.cc | ||
headers = email_message.extra_headers | ||
message = email_message.message() | ||
|
||
# Open a connection for the BASE_BACKEND set in settings. | ||
connection = connections["default"] | ||
mlissner marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
# Verify if recipient email address is banned. | ||
banned_email = EmailFlag.objects.filter( | ||
email_address=convert_list_to_str(to), | ||
object_type=OBJECT_TYPES.BAN, | ||
) | ||
|
||
# If the recipient's email address is not banned continue, | ||
# otherwise the message is discarded. | ||
if not banned_email.exists(): | ||
mlissner marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
# Retrieve plain normal body and html normal body | ||
[plaintext_body, html_body] = get_email_body( | ||
message, | ||
plain="text/plain", | ||
html="text/html", | ||
) | ||
|
||
# Compute attachments total size in bytes | ||
attachment_size = 0 | ||
for attachment in email_message.attachments: | ||
attachment_size = len(attachment[1]) + attachment_size | ||
mlissner marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
# Check if the message contains a small version | ||
plain_small = "" | ||
html_small = "" | ||
small_version = False | ||
for part in message.walk(): | ||
if part.get_content_type() == "text/plain_small": | ||
plain_small = part.get_payload() | ||
small_version = True | ||
if html_small: | ||
break | ||
if part.get_content_type() == "text/html_small": | ||
html_small = part.get_payload() | ||
small_version = True | ||
if plain_small: | ||
break | ||
|
||
if small_version: | ||
mlissner marked this conversation as resolved.
Show resolved
Hide resolved
|
||
# If the message has a small version | ||
small_only = EmailFlag.objects.filter( | ||
email_address=convert_list_to_str(to), | ||
object_type=OBJECT_TYPES.FLAG, | ||
flag=EmailFlag.SMALL_ONLY, | ||
) | ||
if ( | ||
small_only.exists() | ||
or attachment_size > settings.MAX_ATTACHMENT_SIZE | ||
): | ||
# If email address is small_only_email flagged or | ||
# attachments exceed the MAX_ATTACHMENT_SIZE | ||
|
||
# Retrieve plain small body and html small body | ||
[plain_small_body, html_small_body] = get_email_body( | ||
message, | ||
plain="text/plain_small", | ||
html="text/html_small", | ||
) | ||
|
||
# Store small message in DB and obtain the unique | ||
# message_id to add in headers to identify the message | ||
stored_id = store_message( | ||
from_email=from_email, | ||
to=to, | ||
cc=cc, | ||
bcc=bcc, | ||
subject=subject, | ||
message=plain_small_body, | ||
html_message=html_small_body, | ||
headers=headers, | ||
) | ||
|
||
# Compose small messages without attachments | ||
# according to available content types | ||
if html_small_body: | ||
if plain_small_body: | ||
email = EmailMultiAlternatives( | ||
subject=subject, | ||
body=plain_small_body, | ||
from_email=from_email, | ||
to=to, | ||
bcc=bcc, | ||
cc=cc, | ||
headers=headers, | ||
connection=connection, | ||
) | ||
email.attach_alternative( | ||
html_small_body, "text/html" | ||
) | ||
else: | ||
email = EmailMultiAlternatives( | ||
subject=subject, | ||
body=html_small_body, | ||
from_email=from_email, | ||
to=to, | ||
bcc=bcc, | ||
cc=cc, | ||
headers=headers, | ||
connection=connection, | ||
) | ||
email.content_subtype = "html" | ||
else: | ||
email = EmailMessage( | ||
subject=subject, | ||
body=plain_small_body, | ||
from_email=email_message.from_email, | ||
to=email_message.to, | ||
bcc=email_message.bcc, | ||
cc=email_message.cc, | ||
connection=connection, | ||
headers=headers, | ||
) | ||
|
||
else: | ||
|
||
# Store normal message in DB and obtain the unique | ||
# message_id to add in headers to identify the message | ||
stored_id = store_message( | ||
from_email=from_email, | ||
to=to, | ||
cc=cc, | ||
bcc=bcc, | ||
subject=subject, | ||
message=plaintext_body, | ||
html_message=html_body, | ||
headers=headers, | ||
) | ||
|
||
# Compose normal messages with attachments | ||
# according to available content types. | ||
# Discard text/plain_small and text/html_small | ||
# content types. | ||
if html_body: | ||
if plaintext_body: | ||
email = EmailMultiAlternatives( | ||
subject=subject, | ||
body=plaintext_body, | ||
from_email=from_email, | ||
to=to, | ||
bcc=bcc, | ||
cc=cc, | ||
headers=headers, | ||
connection=connection, | ||
) | ||
email.attach_alternative( | ||
html_body, "text/html" | ||
) | ||
else: | ||
email = EmailMultiAlternatives( | ||
subject=subject, | ||
body=html_body, | ||
from_email=from_email, | ||
to=to, | ||
bcc=bcc, | ||
cc=cc, | ||
headers=headers, | ||
connection=connection, | ||
) | ||
email.content_subtype = "html" | ||
else: | ||
email = EmailMessage( | ||
subject=subject, | ||
body=plaintext_body, | ||
from_email=email_message.from_email, | ||
to=email_message.to, | ||
bcc=email_message.bcc, | ||
cc=email_message.cc, | ||
connection=connection, | ||
headers=headers, | ||
) | ||
|
||
# Add attachments | ||
for attachment in email_message.attachments: | ||
# file name, file data, file content type | ||
email.attach( | ||
attachment[0], attachment[1], attachment[2] | ||
) | ||
|
||
else: | ||
# Store message in DB and obtain the unique message_id to | ||
# add in headers to identify the message | ||
stored_id = store_message( | ||
from_email=from_email, | ||
to=to, | ||
cc=cc, | ||
bcc=bcc, | ||
subject=subject, | ||
message=plaintext_body, | ||
html_message=html_body, | ||
headers=headers, | ||
) | ||
# If not small version, send original message. | ||
email = email_message | ||
# Use backend connection | ||
email.connection = connection | ||
# Add header to with the unique message_id | ||
email.extra_headers["X-CL-ID"] = stored_id | ||
|
||
# Verify if recipient email address is under a backoff event | ||
backoff_event = BackoffEvent.objects.filter( | ||
email_address=convert_list_to_str(to), | ||
).first() | ||
|
||
# If backoff event exists, check if it's under waiting period | ||
if ( | ||
backoff_event.under_waiting_period | ||
if backoff_event | ||
else False | ||
): | ||
# TODO QUEUE email | ||
pass | ||
else: | ||
# If not under backoff waiting period, continue sending. | ||
|
||
try: | ||
email.send() | ||
except Exception as e: | ||
# Log an error | ||
logger.error(f"Error sending email: {e}") | ||
connections.close() |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These shouldn't be in the DEVELOPMENT environment, should they? Devs keep using console.Emailbackend, and prod uses the settings above?
Probably needs a tweak in 05-private.example?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I've moved it to 05-private.example, unless Devs want to play with the functions of the custom backend email they don't need to put it under DEVELOPMENT.