Accept external test suite files to generate test cases

[janosimas]

I don't know if this matches the plan for the crate but I though it is suitable.

The QA in the project I work on have a csv file with test cases input/output:

scenario_name,assert_http_code,assert_error_message,name
May not have linebreaks,400,error message,\n
...

I wanted to use it to generate test cases in rust, I was thinking about something like:

#[test_input] // used to generate test names
struct TestInput {
  scenario_name: String,
  assert_http_code: u16,
  assert_error_message: String,
  name: String,
}

#[generate_test_suite_from_file("path_to_file.csv")] // parsed by https://crates.io/crates/csv
fn test_suite_name(input: TestInput) {
// ... test code
}

This would generate:

mod test_suite_name {
  #[test]
  fn may_not_have_linebreaks(input: TestInput) {
    // ... test code
  }
}

[frondeus]

Hmmm....

We are talking here about loading and parsing a CSV file in proc macro. Which means it would be similar to comptime (In Rust: const fn.

That would definitely make the compilation time longer.

Also, I would not implement just CSV processing. The whole feature would have to be generic. I do not want to support N different file formats.

@luke-biel what do you think?

[janosimas]

I agree that more file formats would be ideal.

About compilation time, that is an issue but could be document and the feature be kept behind a feature flag.

[luke-biel]

#78 seems related.

There are rough plans into turning test_case into full test suite. There are also few unresolved design decisions. test_case should stay relatively simple (to ones writing/reading test code). I'd also love to keep in mind how the design will be affected by custom_test_frameworks once they hit stable - it may be an overshoot but I prefer to avoid users having to do huge rewrites of their code.

Issues like this definitely help clarify requirements.

Some thoughts:

• we can try achieve multi-format parsing if we include serde
• name test_scenario("file") or test_scenarios looks clean
• there may be problems trying to couple TestInput if the type is not specified within attribute
• there are plans to include context within test_case - this may go in tandem

Definitely need to think about this more.

[janosimas]

there may be problems trying to couple TestInput if the type is not specified within attribute

What problems do you foresee?

[frondeus]

The procedural macro has no access to the TestInput structure. It would only have an identifier.

[luke-biel]

there may be problems trying to couple TestInput if the type is not specified within attribute

What problems do you foresee?

I see this as a design limitation. First as @frondeus mentioned, this requires "Deserialize" trait implemented on TestInput - I'm not sure if we want to use serde or homebrew. Second, if I'd like to add some context to test (eg. it's name, path to file it's generated from) I would either add it as separate parameter or use Context<TestInput> and this approach should scale to normal test cases.

Overall it's not a big deal, I put it there to remember later.

[nathan-at-least]

Before I saw this ticket, I began work on a separate crate to provide this specific feature: test_vectors

It is not yet in working condition, but it may be a good proving ground for a feature in test-case while keeping the two interfaces independent. If test_vectors got to the point of being reliably usable, then it may be a candidate to merge its functionality into test-case.

[luke-biel]

I'd be happy to take a look at test_vectors in the future and if possible merge that into test case :D I appreciate the interest.
I see that you rely on TryFrom<&[u8]> in your impl. It's nicer than dependency on serde I initially envisioned :)

[nathan-at-least]

I haven't revived that project, and I do recall one or more blockers. One I saw you commenting on in rust-internals about build caching behavior for including files (if a non .rs file changes, build doesn't notice that and won't rebuild).

So I can't necessarily recommend that implementation. Maybe the API. ;-)

I was starting with TryFrom<&[u8> to get an "MVP" functionality working, but I personally think I may end up wanting serde for many cases.

I should note, one motivation for raw &[u8] input is that I envisioned generating tests from external test files which come from external sources especially for standardized test vectors, say where a (non-rust-specific) protocol publishes a set of files that should be parsable and another set that are parse errors. So the use case is "can we auto-generate tests for a rust implementation against an external specification", perhaps using git submodules or subtrees or build.rs hacks.