Implementing transactional / operational and marketing email separation

[RCheesley]

🤔 Problem outline

With the current implementation, there are problems with sending transactional/operational emails such as order confirmations, shipping notifications, password reset emails, or 1-2-1 emails through Mautic from the contact profile.

Emails which are sent on an individual basis to a single person, and not in bulk, should be treated differently to those which are sent in bulk.

• The emails still have the headers for a bulk email
• A user can unsubscribe and stop receiving these emails - there is no selectivity to exclude these important communications from the DNC setting

Other providers are already addressing this issue:

• Mailchimp Subscribe to list: no precedence, no unsubscribe
• Amazon SES verification successful: no precedence, no unsubscribe
• Apple store delivery notification: no precedence, no unsubscribe
• Github Issue notification: Precedence: list, List-Unsubscribe

Read more in the forum thread: https://forum.mautic.org/t/personal-email-should-not-contain-an-unsubscribe-header-custom-email/18525/12

✅ Key requirements

• 1-2-1 emails should not have any unsubscribe or other indication that this email comes from a list. It should be the same as if the person sends it from their personal email client.
• Transactional/operational emails should be separated from marketing emails so that users are still able to receive them if they set DNC (eg to ensure they still receive shipping emails, password resets etc)
• Marketing emails should remain the same as they are now (bulk and unsubscribe). It should not be possible to remove the unsubscribe header. Since this is just best practice and will only hurt the client’s domain reputation in the long run.
• Transactional emails should have a global setting that enables the administrator to turn bulk and unsubscribe headers on or off (with big warnings about misusing)
• A permission level so that only specific roles can create and send transactional/operational/121 emails

💡 Ideas for how this might be approached

Note that these have been suggested and are not necessarily to be considered the right way to approach the problem. They are provided for ideas/inspiration.

• Add a flag to template & segment emails for Operational (yes/no). When this is set to yes, the {unsubscribe_url} token would not be forced in to the email and the email would still be sent even if a contact has unsubscribed.
• Add an option for Operational along with Transactional & Marketing in the campaign event editor. Selecting that option would do the same as the flag on the emails.
• Set manual emails sent from a contact profile to automatically use the correct headers and not use any unsubscribe links

💰 Funding

This project is not currently funded. If you would like to contribute toward this project, please make a donation to the RFP General Fund project on Open Collective and select the 'Transactional/Operational Email Separation' tier.

ℹ️ Timelines

As this will be a new feature, it should be created against the 4.x branch for consideration in a forthcoming minor release.

👉 How to make a proposal

Please review the information in the Community Handbook regarding the workflow for proposing and working on this project.

[RCheesley]

Acquia is planning to work on this in Q1 2022.

[adiux]

I'm happy that Acquia put this on the roadmap! 🙏

If I had a wish I would love to see the manual email in there too:

Set manual emails sent from a contact profile to automatically use the correct headers and not use any unsubscribe links

I believe this brings a lot of value because. As it is now the manual email sending feature is of no use. As a Sales or Marketing person I don't want to send a personal email to one of my contacts and then have the email client tell them that it was sent from a mailing list.

[adiux]

Is this still in scope for Q1?

[YosuCadilla]

Hi, is this implemented in M4 or M5? (or not at all?)

[escopecz]

I think that these PRs solve this:
#11787
#11786
Or is something still missing?

[YosuCadilla]

Thank you @escopecz. You are correct.
Would you say these PRs can be applied to M4 too ?

[escopecz]

I think those are breaking changes and should be released in a major version..

[YosuCadilla]

Understood. What about implementing it in a way that the same "component" can be used by both M4 or M5.
In a microservices fashion, where it makes sense... maybe as an "email delivery API" or mabe gRPC?

I mean, we need to get started with distributed architecture sooner rather than later, and email is like the cornerstone of Mautic, and there's so much that marketers need for Mautic email to be on par with other MA tools out there...
I really see this as a golden opportunity to get started with it.

[escopecz]

Feel free to propose such solution in more detail so the community can comment on it.

[YosuCadilla]

The proposal is:
What about implementing it in a way that the same "component" can be used by both M4 or M5.
In a microservices fashion, where it makes sense... maybe as an "email delivery API" or mabe gRPC?

[YosuCadilla]

In more detail:

• Add extra metadata to each email to "denote" how it has to be handled.
• Send each email to a folder or a queue like rabbit for storage.
• Create an "external" email processing microservice or headless function that can consume the emails from the queue or directory and send them taking into account the values in the metadata.

Possible values to consider as metadata for the processor:

• From address
• Delivery date/time
• Delivery route (Local, SES. Sendgrid, etc)
• ETC.

So basically, most (if not all) email properties, can be specified "per email", probably in the email section of Mautic, instead of in the Config/email section.
And then, email delivery is fully configurable and controllable per "email processing microservice" Effectively uncoupling Email delivery from Symfony and giving us the freedom to implement the email processor in any way/language/paradigm is better for the purpose of sending email. (Since symfony seems to clearly NOT be optimal for that).

[escopecz]

Yosu, I was hoping you'll start a new discussion/issue/forum post on this topic. You tend to hijack other discussions and divert them elsewhere. This is not a discussion we can have in a subthread of some other topic. This needs proper proposal. Like what are the advantages of having another app handling email sending over a Mautic command. It has pros and cons. Is there more weight on the pros side? Does it make sense to build something for M4 which will not be maintained next year? Is the extra maintenance of extra app/microservice worth it? Who will make releases for it when we have a hard time finding release leaders for Mautic? What technologies do you propose to use for it? What will it share with Mautic and what will be only on the microservice side? It will need to read the configuration of email transports from Mautic. It will have to be connected to the same queuing engine. It will have to be connected to the same database to fill in the email send events. That will make it a coupled microservice that heavily depend on Mautic's resources. From my experience, such microservices are PITA to manage and eventually are merged back to the main app.

We plan to use queues for M5 with help of Symfony Messenger so you'll be able to choose which queuing engine you wish to use and you can connect to it with anything you like and consume the messages. So it will be possible to implement what you suggest. But I think that Mautic users don't want the extra hustle of installing, configuring and maintaining an extra app just to send emails. There should be a default option to do it within Mautic. You repeated several times that Mautic is for Marketers, not developers nor sys admins. A microservice would be another boulder on the road to install Mautic.

[YosuCadilla]

Yosu, I was hoping you'll start a new discussion/issue/forum post on this topic. You tend to hijack other discussions and divert them elsewhere.
True, sorry about that.

This is not a discussion we can have in a subthread of some other topic.
I understand.

This needs proper proposal.
I agree, but can you hold making/finalizing the changes for whatever amount of time it would take to propose, debate and come up with consensus?

Like what are the advantages of having another app handling email sending over a Mautic command. It has pros and cons. Is there more weight on the pros side?

• Does it make sense to build something for M4 which will not be maintained next year?
• Is the extra maintenance of extra app/microservice worth it?
• Who will make releases for it when we have a hard time finding release leaders for Mautic?
• What technologies do you propose to use for it?
• What will it share with Mautic and what will be only on the microservice side?

The following ones you didn't write as a question, so I am not certain if they belong to the same list or not:

• It will need to read the configuration of email transports from Mautic.
• It will have to be connected to the same queuing engine.
• It will have to be connected to the same database to fill in the email send events.
• That will make it a coupled microservice that heavily depend on Mautic's resources.
• From my experience, such microservices are PITA to manage and eventually are merged back to the main app.

Not only all of these are very valid points but also a very good basis to start the discussion.
Thanks for adding so much value to this conversation, I am going to start the thread Immediately.

Here it is:
https://forum.mautic.org/t/proposal-microservice-based-email-delivery-system/28182

[YosuCadilla]

@RCheesley @escopecz I took the liberty of copying your posts into the new post on the forums, hope it is alright, but if it is not please let me know so I can remove it.