Making Verify play nice with AutoFixture

[egil]

Hi all

---

UPDATE: after some tweaks to Verify by Simon, this post has been edited to reflect what is currently required to get AutoFixture and Verify to work together.

That also means most of the discussion below in the comments are not relevant any more.

---

I want to share how I have managed to get AutoFixture and Verify to work well together:

AutoFixture is a tool that we use quite a lot at work to generate random data for tests, where the data itself is not important, but just needs to be there to support the test. These test data change with every run of a test, which, for the most part, works well with Verify, since it scrubs things like Guid and DateTime automatically.

However, when AutoFixture generates strings for a test argument or property in a class, it does so like this. propertyName2cbc7050-919f-4fd3-867e-f69e311c9647, which Verify is not able to scrub and turn into something like propertyNameGUID_1.

This means that the following DOES NOT work out of the box:

[Theory, AutoData]
public async Task Can_create_user_with_name(string randomUserName)
{
  var sut = new UserRepository();

  var createdUser = sut.Create(randomUserName);

  await Verify(new { createdUser, randomUserName })
    .UseParameters(randomUserName);
}

[Theory]
[InlineAutoData(42)]
public async Task Can_create_user_with_age_and_random_name(int age, string randomUserName)
{
  var sut = new UserRepository();

  var createdUser = sut.Create(randomUserName, age);

  await Verify(new { createdUser, randomUserName })
    .UseParameters(age);
}

The problem is that randomUserName will be set to something like this randomUserName2cbc7050-919f-4fd3-867e-f69e311c9647, but with a new GUID on every run.

Note: the AutoData and InlineAutoData attribute is part of the AutoFixture.Xunit2 package.

AutoFixture Extensions

To solve the problem, I created an extensions for AutoFixture.

First we need to adjust AutoFixture's data attributes to generate strings that Verify's inline GUID scrubber can scrub.

These AutoData and InlineAutoData attributes (with the same names as the ones in the AutoFixture.Xuni2 package for easy drop in) does this:

namespace AutoFixture;

public sealed class AutoDataAttribute : Xunit2.AutoDataAttribute
{
    public AutoDataAttribute()
        : base(FixtureFactory)
    { }

    private static IFixture FixtureFactory()
    {
        var result = new Fixture();
        result.Customize<string>(x => x.FromSeed(seed => $"{seed}_{Guid.NewGuid()}"));
        return result;
    }
}

public sealed class InlineAutoDataAttribute : Xunit2.CompositeDataAttribute
{
    public InlineAutoDataAttribute([NotNull] params object[] data)
        : base(new InlineDataAttribute(data), new AutoDataAttribute())
    {
    }
}

The basic change is to add an underscore (_) between the "seed" and the generated GUID. In the randomUserName example, it will now generate ``randomUserName_2cbc7050-919f-4fd3-867e-f69e311c9647` which Verify can scrub.

Finally, we can enable inline scrubbing of GUIDs in our module initializer:

VerifierSettings.ScrubInlineGuids();

At this point, the tests at the top of the post will work as expected.

[SimonCropp]

side note: u need to await Verify

[egil]

Of course, forgot that.

[egil]

@SimonCropp. I get a feeling that my SetFileName and related methods are reimplementing a bunch of exiting logic, e.g. like scrubbing file names from illegal characters. Could I be doing this smarter?

[SimonCropp]

Could I be doing this smarter?

absolutely. how would you like that functionality exposed?

[SimonCropp]

sorry, poorly phrased response. I dont think you can be doing it smarter. i think Verify should help better with that code via new APIs

[egil]

Not entirely sure.

I had a couple of problems with my code.

1. File names became too long, actually had to comment out a test because the generated file paths was longer than 260 chars.
2. Parameters can contains characters not allowed in file names.

It would be lovely to have any of the code in the VerifyExtensions file above. See suggestion below for making UseParameters less strict through configuration.

[SimonCropp]

would it make it easier if i had an approach to ignore certain parameters?

eg

    [Theory]
    [InlineAutoData(42)]
    public async Task ScrubOneParameter(int age, string randomUserName)
    {
        await Verify("Value")
            .UseParameters(age, Param.Ignore);
    }

[SimonCropp]

the problem with the above is it is not possible to work out what parameter to ignore. useparameters is positional so we know what value matches to what name. so if u have 3 parameters. and u do UseParameter for 2, what 2 are u referring to?

[SimonCropp]

unless we assume the last one(s) are ignored?

[egil]

So there is some code now that I am able to suppress by providing my own filename through UseFileName that would otherwise have me provide all parameters to UseParameters. Can that not be toggled off through a setting and have UseParameters behave differently?

[SimonCropp]

that not be toggled off through a setting and have UseParameters behave differently

we can but i dont know how to make it behave

if the tests is public async Task MyTest(string value1, value2)

and u call UseParameters(value2). where the value of value2 is x what will the filename be?

• TheClasst.MyTest_value1=x.verified.txt or
• TheClasst.MyTest_value2=x.verified.txt

there is no way to work out what parameter x maps to

[egil]

Ahh i see [CallerArgumentExpression("value2")] will give you that information. That's what I do in my custom UseInlineDataParameter extension method above.

But I see the problem retrofitting the existing method, but a new method and a way to disable the current use parameters logic would be nice.

[SimonCropp]

comments fixed

[SimonCropp]

so with the relaxed param checking, do we still want/need a Verify.AutoFixture extension?

[egil]

Can you explain the logic of UseParameters now? It seems ok from my initial testing at least.

[SimonCropp]

hows this https://github.com/VerifyTests/Verify/blob/main/docs/parameterised.md#useparameters

[egil]

That's good.

However, I do still miss one feature, the ability to not call UseParameters at all, e.g.:

[Theory, AutoData]
public async Task Can_create_user_with_name(string randomUserName)
{
  var sut = new UserRepository();

  var createdUser = sut.Create(randomUserName);

  await Verify(new { createdUser, randomUserName });
}

In this scenario, I am still getting this exception:

System.Exception : Method `Can_create_user_with_name` requires parameters, but none have been defined. Add UseParameters. For example:

var settings = new VerifySettings();
settings.UseParameters(randomUserName);
await Verifier.Verify(target, settings);

or

await Verifier.Verify(target).UseParameters(randomUserName);

It would be lovely if that was not necessary.

[SimonCropp]

I do still miss one feature, the ability to not call UseParameters at all

yeah thats fair. try 16.9

[egil]

Tested with the latest release, and my solution above is now obsolete. Ill update the text here for others.

[SimonCropp]

@egil #1075