Support email notification channel

[davelopez]

Add a new "Email" channel for every notification category to receive a copy of the notification via email. The users can opt out of receiving emails for each category and subscribe only to the ones they are interested in.

[davelopez]

To fully support this new notification channel we need to decide how to handle scheduled emails as the notification system supports scheduled notifications.

I need to explore a couple of options, but I wanted some feedback about what option you think is best worth exploring or if you have new ideas.

First, I will move the current "creation" of the notifications to a Celery task so we can offload the request from resolving the recipient users and dispatching emails (or any other additional channel in the future).

These are the ideas we could explore:

1. Use the eta date of the celery task to defer the execution of the task to the date of publication.

   • I've played around with this option already as it was as simple as doing:

     send_notification.apply_async(args=[request], eta=request.notification.publication_time)

     Unfortunately, this seems to keep the task in memory until it can be executed so it is only useful when the publication date is just a few minutes away... so I think is not really usable for our use case. See documentation here: https://docs.celeryq.dev/en/latest/userguide/calling.html#eta-and-countdown

2. Database migration + periodic task

   • Add a new dispatched column to the notifications table and have a periodic Celery task check those notifications that should be dispatched after the publication date is reached and have not been dispatched yet.
   • Mark the notification as dispatched and send the emails (or any other additional channels).

3. Filesytem + cron

   • If, for some reason, we want to handle this completely out of band, we could do something similar to what the short-term storage does, and have a JSON file with the details of the notification dumped in a special directory.
   • Then a periodic cron script can check if a notification is due to be sent and then dispatch the emails, etc.

Option 2 seems like a good way to go, but I wanted to see if you have other, maybe simpler ideas to handle this.

[hexylena]

Option 2 would be safest. Easiest to ensure at least once delivery.

[martenson]

What are some examples of delayed user email notifications that require emailing?

[davelopez]

Right now, the use case that triggered adding email as a new channel, is to detect suspicious activity in an account and then automatically send a notification (via email and any other additional supported channel) to the user before proceeding to block that account. In this case, the "scheduling" is not important, it should be delivered on the spot.

I can't think of a real use case for "just delaying", but the notifications API allows creating a notification and "publishing" (or scheduling) it in the future. So, if the user enables the email channel to receive notifications, the expected behavior will be to receive the email sometime around the "publication date".

[mvdbeek]

In the case of suspicious activity you're running an external script anyway, is it an option to build the email sending into that script ?

[davelopez]

I guess so. The thing is, we designed the notification system to support multiple channels in the future, and email was one of them, so it made sense to implement it now and go through the notifications API for the sake of communications consistency.

[martenson]

Once in the notification business you can only ever go deeper. ;)

[davelopez]

We decided to go with Option 2 in the backend meeting.

[davelopez]

@mvdbeek suggested to use https://novu.co/ in the wg-backend chat.

This looks like a great replacement for the whole notification system down the road. Of course, we should use the self-hosted solution at the cost of complicating a bit the deployment and local dev setup https://docs.novu.co/self-hosting-novu/introduction.

The front-end components are available for Vue3 (https://docs.novu.co/notification-center/client/vue) so, maybe, we can explore that once we make the switch.

[hexylena]

Another service to run would be a burden for especially small admins, if we have a functional notification system we should be sure to include that cost to switching.

[mvdbeek]

But only the UseGalaxy.* instances need this ... can't see a small lab requiring email notifications.

There's a cost to baking everything into Galaxy as well, giving it the impression that this is a enormous piece of software with a million things you don't need.

[hexylena]

Yep, costs to every approach, just need to balance them. If it's just for .org and no one else maybe it's fine, but I suspect then many .* instances will need it (or think they should be using it.)

I think small groups do sometimes want to send emails (especially for e.g. history cleaning reminders, password resets, etc) I configured it at one group, and would have with every other galaxy had the institution permitted it.