How to get count of hits by response.status_code

[dbire]

Hello,

I am using the CLI version and I am wondering how it is possible to print at the end of the st run the count of hits by response.status_code to check if I was able to cover the status code expected ?

For example, it could happen that st runs and find no failure just because it gets only 400 or 404, and found no errors regarding the conformity... and 20x checks would be under the radar.

[Stranger6667]

Hi, it is not implemented, but requested, see #1414. Schemathesis collects this data, but we need to display it nicely and I don’t know what would be the best way to do so

[dbire]

For example, @Stranger6667 what do you think of the following ?

ENDPOINT METHOD STATUS #HITS

---

/users post 200 50
/users post 400 10
/users/login post 200 20
....

[Stranger6667]

The main issue is that it may work for relatively small cases, but for e.g 70 endpoints with 5-6 statuses each it will not be readable and will draw attention from other parts of the output, ie 300 lines of text appearing at the end of CLI output is quite distracting. Even smaller would be problematic, e.g often we see 200 & 400 only, so for 10 endpoints it will be 10-20 lines, but I find the most APIs in the wild are much larger than that

[Stranger6667]

Drawing a table might be better though, so the data is more compact

[dbire]

I have written a small python script that uses a generated "cassette.yml" as input and give me some statistics of what have been covered If anybody is interested...