API tests

[gmaxwell]

We're under-covered currently for tests that simulate crazy callers. The practise of asserting on statically wrong use limits the space of things that can / need to be tested here, but there is still some freedom.

This should be less "test" and more "mechanical verification that the interface we present to callers isn't unintentionally changed".

(As an example, libopus' test_opus_api, calls into the library about 116 million times. I'd say that libopus has an API surface area of about 50x larger that libsecp256k1)

[gmaxwell]

API tests should explicitly perform tests with variously unaligned input/output buffers.

(not likely to currently be a source of bugs except through compiler screwup, but if anyone starts going around adding SIMD code it would rapidly become a potential source of extremely hard to spot bugs).

[real-or-random]

From all the existing work, what comes probably closest is #744 .

This should be less "test" and more "mechanical verification that the interface we present to callers isn't unintentionally changed".

@gmaxwell Does fuzzing match this idea, or did you have something else in mind?

[gmaxwell]

I would describe that test as the opposite of an api test: All it tests is the payloads, the usage of the API is hardcoded into the harness.

An API test does stuff like calls with a null context or a number of pubkeys = 0, it acts like callers that fail to check errors or which call functions out of order... the goal is to validate the contract implied by the header documentation and maybe a little beyond it.

[rickmark]

It seems to me one of the first steps here is to separate the library code from the test code better. (Compile the library, then compile tests that link to that library from code in a separate directory)

Broadly I see:

• Conformance tests - Like OpenSSL to ensure that the cyrpto is correct, these are very good today
• Performance tests - ensure that the operations are performant (handed by bench*.c and friends), these exist
• Client tests - Build and sanity check that a consumer can accomplish it's task, needs work
• (NEW) API tests - Do all the things that a client shouldn't do and make sure its gracefully handled. Because of the nature of this it seems like the wise idea might be to use a language like python/ruby and FFI the API surface because it's easier to create a large matrix of tests this way. It also tests that FFI works correctly (as much as I hate FFI 😆)