Make use of regression tests

[carlosms]

In #336, #363 we have introduced regression tests.
For now they have to be run manually, to verify if a new version breaks compatibility. This way we can be aware when to add a warning in the release notes.

This issue is to keep track of what automation we want to setup.
In the last team meeting we mentioned that we may want to configure a job in our jenkins infrastructure to run it on demand.