New protostar testing architecture

[Arcticae]

Background

In order to prepare for coming plugin system from the cairo 1.0 package manager (cargo-like tool), we are considering migrating away from the existing flow of testing in order to align the targets with the said project.

Terms:

• I will use SNTF as "starknet testing framework instance" abbreviation for brevity
• "StarkNet" tests will replace "integration tests" in nomenclature and former "unit tests" will be "cairo tests" as of new architecture

Facts:

1. Pure cairo 1.0 code will eventually be tested on sierra-level (with a dedicated sierra runner), with a test runner plugin
2. Testing starknet code will be provided by a plugin as well (we plan to adapt protostar to be one of those)
3. Current implementation of testing is practically un-cacheable, which leads to performance losses (re-compilation, re-deployments each time we are re-running the tests) and prone to ecosystem changes

Current state

The testing flow is now basing on deploying the contracts into local in-memory SNTF as a test contract, even if no satellite external contracts are used.

Pros:

• Clear distinction of what is a unit test and what is integration test. User controls the aspect by himself.
• Integration tests can be in the same file as unit test (although not recommended)
• Deployment from any place in the code

Cons:

• overhead of deploying the contract each time to devnet
• requires re-compilation of the whole code (the test itself and implementation of the functions being tested)
• overhead of using SNTF and building whole starknet state, instead of only VM for unit tests
• hard for newcomers to differentiate between unit and integration testing (unit tests approach is not commonly used, as it's not like anything in other blockchain testing frameworks)

The division of testing

In order to achieve better performance and cacheability, we need to differentiate somehow between integration and unit tests on the level of code, configuration or infer it automatically. This is needed in case we want to save the SNTF state and load it from file (loading cache).

The drawing below shows a separate flows for Starknet and Cairo tests, in which the Cairo tests act as a bundle of code, not connected with SNTF at all, and just runs on a Cairo VM (rust or a python one - depending on available implementation).

Following approach has a few properties:

• StarkNet Tests on the other hand use the approach of using the SNTF as a fixture, which enables us to decouple the concept from protostar completely, if future integtation of rust tooling is considered.
• Support for syscalls/storage_vars will be no more in pure Cairo tests - it will apply only to StarkNet tests, because it doesn't make sense to build the whole context, making the tests logic just more confusing. Although storage_vars will no longer exists, we want to support store/load cheatcodes in that approach. We could also consider an approach where storage is mocked automatically (or with a flag passed to the test), to make testing functions with storage_var usage easier (@MaksymilianDemitraszek review needed).

Presented architecture is a draft and might change in the future.

[MaksymilianDemitraszek]

Great work in formalising ideas!

Open questions:

1. The thing I am worried about new architecture and I would like the design doc to discuss, is lack of storage vars in context of pure cairo. What if a libfunc of a contract requires access to storage var? We probably cannot use pure cairo to execute test cases in the new architecture. We probably have to use starknet lang for local tests as well wich some kind of a custom backend for storage (can be something as simple as dict).

2. I believe such isolation may have a benefit of forcing users to decouple bussines logic from contract interfaces which is also a benefit. Though user may want to keep some functions in the contract interface instead of isolating them to library. Should protostar automatically externalize every contract entrypoint or we should not to encourage users to do thing mentioned in the first sentence.

[MaksymilianDemitraszek]

It is a valid take, but what would syscalls do when there is no os to call?

[Arcticae]

2. We established that if we handle the storage in the proposed (e.g. simple Dictionary) way, the problem is gone. Although i do not understand how this is relevant since we might want to encourage users to use only exposed methods in integration testing approach, and separate function testing can be done on regular import basis (or at least we should be able to do it).

[Arcticae]

It is a valid take, but what would syscalls do when there is no os to call?

We could support a configuration of the test context in some way. But not all syscalls would be supported anyway in that flow (e.g. call, send_message_to_l2).

It kind of worries me that it would be harder for the user to differentiate between approaches if it's implicit (given that we infer the approach from usage of cheatcodes).

[MaksymilianDemitraszek]

I actually start leaning to "unit tests" in pure cairo. If user wants to test contract it has to be deployed so it would be in os context.

User still would have the same level of interaction as before thorugh store, load, (we could expose a cheatcode call_internal_entrypoint). Distinction would be clearly drawn if code is starknet code or if code is cairo code. I believe this is a huge benefit. Writing test code in starknet-lang is kinda condratictory to initial idea which would make integration with cairo package manager easier. In the separation scenario we clearly give a sign to user when test is run in os context and when is not.

Aside i am scared that allowing starknet-lang in test code would lead us to rewritting whole starknet as a dummy version.

I think we should also need to update our internal dictionary, and stop using unit test and integration test the way we do now as those are misleading. Testing a single contract deployed on the starknet is still a unit test as contract is a proper unit.

[Arcticae]

I've updated nomenclature and our conclusions from this discussion in the proposal.

[kasperski95]

The part of the code that needs that detects whether it's a Cairo test or StarkNet code needs to be in the plugin of the Cargo-like tool. Pure Cairo tests should be run by Cairo Test Runner. There are many unknowns. I am not sure the impact of Cairo 1 on this change. How do you want to inject "the StarkNet fixture" into a test? Are we going to provide cheatcodes through hints or should we avoid them because they can cause issues with a language server? Can you provide an example how StarkNet tests are going to look like? I'd create a new Protostar component to implement new StarkNet tests.

Before creating this functionality, I'd like to clearly assign responsibilities to containers.

C1 and C2 diagrams from the C4 Model:

[Arcticae]

C2 shows some state of the ecosystem for sure, but we can't say for sure when particular components (e.g. StarkNet OS) will be rewritten to rust.

As for declaration of plugins - i think they should come with some library code (in this case sierra functions with hints containing our cheatcodes). But the end-game i believe is that protostar will become a plugin.

Language server will be able to detect the library that we expose to the user, thus it should be doable to have auto-completion in this scenario.

We were thinking about those changes, because of impact that cairo-1 will have on the ecosystem, and transfering the cairo testing responsibility onto the standard tools. Cairo1 will be not only for StarkNet though - it will serve as a general tool for writing provable programs, with a decent set of concepts in the language (unlike current cairo has). If we make changes with cairo1 integration, we have a leverage of making the users used to the new flow of testing. Also - we can hide the breaking changes with the ones that are going to be apparent in cairo1 migration anyway. Although i do not yet know when the "standard" test runner will be implemented, migrating protostar to using it should be rather easy (we'll just remove it and tell users to use the standard one ¯_(ツ)_/¯ ).

Let me know if this makes things clearer.

[kasperski95]

Language server will be able to detect the library that we expose to the user, thus it should be doable to have auto-completion in this scenario.

Will the language server work in hints? If not, we should consider exposing cheatcodes as Cairo 1 functions.

How do you plan to detect test type (Cairo or StarkNet test)? Does Cairo 1 support decorators? I think detecting test type needs to take place during Test Collecting phase. The test collector should be written in Rust as part of Cairo Test Runner/Protostar Plugin projects.

[Arcticae]

we should consider exposing cheatcodes as Cairo 1 functions

It will work that way, that's the idea i believe.

I think detecting test type needs to take place during Test Collecting phase.

That's not the case when we want to have an internal differentiation between the types of test. As far as test collector goes, it should probably work similarly as pytest and allow running the tests with plugins. So that would probably include some pluggable logic to get info about the test itself, and if the given plugin from the list should run it.

[MaksymilianDemitraszek]

It will work that way, that's the idea i believe.

It wll, It is the only option possible ;)