-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add updater for expected results of integration tests #542
Comments
Yes please! Looking forward to the PR 👍 |
Can we utilize syrupy for this purpose? It already supports comparing complex JSON structures and overwriting existing results. |
We have our own custom scripts to process the JSON outputs. We trim the non-deterministic parts and only check the parts that are crucial. So, I doubt Syrupy being a good solution. All we need to do really is to update the JSON outputs by copying over the files if an update flag is provided. |
If we have non-deterministic parts in the expected result files, then can't we just "eliminate those non-deterministic parts" from these expected result files? These files can store just the "deterministic subset" of the JSON result that Macaron produces, and not the whole result. This way we may be able to delegate both comparing and updating to surupy and remove the burden of maintaining the compare script. |
Removing certain values from the output and storing it as intermediate files for testing also needs maintenance and can be even more complex. I don't see the benefit here since we already have comparison scripts. |
Yep. You're right that in that case, we need to cherry-pick things from the current output, and that part requires maintenance. Something I am wondering is whether the compare script or the cherry-picking will require more maintenance long-term. |
Currently any changes that create a legitimate difference in Macaron's results and the expected results used in the integration tests must be handled manually. An automatic method for updating the expected results files would save considerable time moving forwards.
The text was updated successfully, but these errors were encountered: