Test Isolation

[lukesneeringer]

I started writing a CRUD test today:

def test_crud_with_request_dict(identity):
    assert len(identity.list_users().users) == 0
    try:
        user = identity.create_user({
            'display_name': 'Guido van Rossum',
            'email': 'guido@guido.fake',
        })
        assert user.display_name == 'Guido van Rossum'
        assert user.email == 'guido@guido.fake'
        assert len(identity.list_users().users) == 1
        assert identity.get_user({
            'name': user.name,
        }).display_name == 'Guido van Rossum'
    finally:
        identity.delete_user({'name': user.name})

This made me realize that we may want some way of supporting Showcase-enforced test isolation. I need it to be the case that if something goes wrong in this test, that it does not cascade to a bunch of other tests. Basically a "okay, this test is over, drop any data changes it made in the Identity or Messaging services".

That said, this warrants discussion. We may not want to do this, and there are other options:

• Making tests not care about the absolute state of the database (e.g. test that len increased by one, not that it is equal to one).
• Make tests care about the absolute state of the database, but skip the test if a beginning-state guard clause fails. This will cause situations where one test failure will lead to a host of other tests being skipped. That is okay with me actually -- it will be easy to isolate the failure.

That said, some kind of "transaction with rollback" is probably the best thing.

[landrito]

This is a good idea. I think adding a method to the the testing service would be a good way to do this.