Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

native: implement HF-based update #3402

Merged
merged 12 commits into from May 10, 2024
Merged

native: implement HF-based update #3402

merged 12 commits into from May 10, 2024

Conversation

AnnaShaleva
Copy link
Member

@AnnaShaleva AnnaShaleva commented Apr 4, 2024

Let our natives be updated! Close #3213.

TODO:

  • Implement Oracle service cache update.
  • Make lazy MD cache not so lazy and remove generic Methods/Events.
  • Use cached HF-specific MD for new HF-specific MD initialization if no changes are required.
  • Enable Cockatrice with all dependent logic.
  • Tests (unit and state compatbility).

Copy link

codecov bot commented Apr 5, 2024

Codecov Report

Attention: Patch coverage is 84.52012% with 50 lines in your changes are missing coverage. Please review.

Project coverage is 86.08%. Comparing base (83fdcc8) to head (8995f11).
Report is 18 commits behind head on master.

❗ Current head 8995f11 differs from pull request most recent head 9b8d579. Consider uploading reports for the commit 9b8d579 to get more accurate results

Files Patch % Lines
pkg/core/interop/context.go 84.61% 9 Missing and 5 partials ⚠️
pkg/core/native/oracle.go 65.21% 8 Missing ⚠️
pkg/core/native/management.go 88.33% 4 Missing and 3 partials ⚠️
pkg/config/hardfork.go 71.42% 3 Missing and 1 partial ⚠️
pkg/core/native/native_nep17.go 86.66% 2 Missing and 2 partials ⚠️
pkg/config/hardfork_string.go 33.33% 2 Missing ⚠️
pkg/core/blockchain.go 93.54% 1 Missing and 1 partial ⚠️
pkg/core/native/designate.go 71.42% 1 Missing and 1 partial ⚠️
pkg/core/native/native_gas.go 50.00% 1 Missing and 1 partial ⚠️
pkg/core/native/notary.go 50.00% 1 Missing and 1 partial ⚠️
... and 2 more
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #3402      +/-   ##
==========================================
- Coverage   86.12%   86.08%   -0.05%     
==========================================
  Files         331      332       +1     
  Lines       38114    38339     +225     
==========================================
+ Hits        32827    33005     +178     
- Misses       3776     3804      +28     
- Partials     1511     1530      +19     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@AnnaShaleva AnnaShaleva force-pushed the native-upd branch 3 times, most recently from 02adea3 to a51a62d Compare April 9, 2024 11:50
@AnnaShaleva
Copy link
Member Author

AnnaShaleva commented Apr 9, 2024

@roman-khimov, the implementation seems to be in the final stage, I'm going to test it against C# node states for mainnet/testnet, and if everything is OK, then write a couple of unit tests. Need your opinion on the implementaiton structure, it differs from C# a bit, but behaviour is compatible.

@AnnaShaleva
Copy link
Member Author

With the latest C# node fixes mainnet states match perfectly (with Cockatrace enabled at 5210001 height):

anna@kiwi:~/Documents/GitProjects/nspcc-dev/neo-go$ go run scripts/compare-dumps/compare-dumps.go ./dumps/GoDump_mainnet_5224362/ ./dumps/Storage_334f454e_mainnet_5225043_cockatrice/
Processing directory BlockStorage_0
Processing directory BlockStorage_100000
Processing directory BlockStorage_200000
Processing directory BlockStorage_300000
Processing directory BlockStorage_400000
Processing directory BlockStorage_500000
Processing directory BlockStorage_600000
Processing directory BlockStorage_700000
Processing directory BlockStorage_800000
Processing directory BlockStorage_900000
Processing directory BlockStorage_1000000
Processing directory BlockStorage_1100000
Processing directory BlockStorage_1200000
Processing directory BlockStorage_1300000
Processing directory BlockStorage_1400000
Processing directory BlockStorage_1500000
Processing directory BlockStorage_1600000
Processing directory BlockStorage_1700000
Processing directory BlockStorage_1800000
Processing directory BlockStorage_1900000
Processing directory BlockStorage_2000000
Processing directory BlockStorage_2100000
Processing directory BlockStorage_2200000
Processing directory BlockStorage_2300000
Processing directory BlockStorage_2400000
Processing directory BlockStorage_2500000
Processing directory BlockStorage_2600000
Processing directory BlockStorage_2700000
Processing directory BlockStorage_2800000
Processing directory BlockStorage_2900000
Processing directory BlockStorage_3000000
Processing directory BlockStorage_3100000
Processing directory BlockStorage_3200000
Processing directory BlockStorage_3300000
Processing directory BlockStorage_3400000
Processing directory BlockStorage_3500000
Processing directory BlockStorage_3600000
Processing directory BlockStorage_3700000
Processing directory BlockStorage_3800000
Processing directory BlockStorage_3900000
Processing directory BlockStorage_4000000
Processing directory BlockStorage_4100000
Processing directory BlockStorage_4200000
Processing directory BlockStorage_4300000
Processing directory BlockStorage_4400000
Processing directory BlockStorage_4500000
Processing directory BlockStorage_4600000
Processing directory BlockStorage_4700000
Processing directory BlockStorage_4800000
Processing directory BlockStorage_4900000
Processing directory BlockStorage_5000000
Processing directory BlockStorage_5100000
Processing directory BlockStorage_5200000
Processing directory BlockStorage_5300000
file BlockStorage_5300000/dump-block-5225000.json: dump files differ in size: 362 vs 1000
compare-dumps dumpDirA dumpDirB
exit status 1

Copy link
Member

@roman-khimov roman-khimov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be OK.

pkg/core/blockchain.go Outdated Show resolved Hide resolved
pkg/core/interop/context.go Outdated Show resolved Hide resolved
type MethodAndPrice struct {
HFSpecificMethodAndPrice
ActiveFrom *config.Hardfork
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure why are you keeping all of them this way. You can have the one and only proper contract version at the current height and then just switch it when HF happens.

Historic calls maybe? These do require some dances.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can have the one and only proper contract version at the current height

That was an early version of this PR. And yes, due to the historic calls. To manage single contract version at a time, we need to initialize it every time when new historic call is processed (which I suppose is far from the best solution) or we need some kind of cache. So if we're to implement this cache anyway, then it's better to initialize it once during the node start in order not to affect the node performance during blocks processing. And, as discussed, we don't have a lot of natives/hardforks, so the cache size is not a problem for us (and we store pointers in cache anyway, as a result the most part of CS structures are shared between hardforks).

But that's just my thoughts. We can get back to one and only proper contract version.

pkg/core/interop/context.go Outdated Show resolved Hide resolved

return s
}

// Initialize initializes Designation contract. It is called once at native Management's OnPersist
// at the genesis block, and we can't properly fill the cache at this point, as there are no roles
// data in the storage.
func (s *Designate) Initialize(ic *interop.Context) error {
func (s *Designate) Initialize(ic *interop.Context, hf *config.Hardfork) error {
if hf != s.ActiveIn() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checking here...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm against of this change because I assume (from C# implementation and neo-project/neo#3200) that in future we still might want to (e.g.) put something into the contract storage on the next hardfork. And this code will be contract-dependent.

That's the whole point of neo-project/neo#3200, otherwise if we do call Initialize only on deploy, then this issue is completely useless.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think?

pkg/core/interop/context.go Show resolved Hide resolved
docs/node-configuration.md Outdated Show resolved Hide resolved
storedJ, _ := json.Marshal(storedCS)
autogenJ, _ := json.Marshal(autogenCS)
return fmt.Errorf("native %s: version mismatch for the latest hardfork %s (stored contract state differs from autogenerated one), "+
"try to resynchronize the node from the genesis: %s vs %s", md.Name, current, string(storedJ), string(autogenJ))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure it's a friendly error message. Many users may even not notice the real message in these JSON dumps.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These dumps are not indented, so they are quite compact, I've tested it. I may add some indent. Or is it better to remove dumps at all?


// Hashes of all native contracts.
var (
Management util.Uint160
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just out of curiosity, can we have them precomputed automatically and saved as consts? //go:generate of some kind. We can combine them then with nativenames probably, consts don't cost a lot.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check out the last commit, it's an interesting experiment, but I still want to keep Uint160 variables to reuse them in other code.

We can combine them then with nativenames probably,

They have different names, is it OK? E.g. nativenames.Gas vs nativehashes.GasToken. If it's OK, then I'll merge them into single package.

pkg/core/native/management.go Show resolved Hide resolved
A part of #3213.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
Ported as a part of neo-project/neo#2942.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
External users should use HF-specific methods and events.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
Initialize all necessary HF-specific contract descriptors once during
contract construction.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
With all associated native API changes ported from
neo-project/neo#2925 and
neo-project/neo#3154.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
It will give us a clue on what's wrong with contract states if something
unexpected happen.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
Similar to nativenames, instantiate once and then reuse everywhere.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
A part of #3213.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
Ensure that Blockchain constructor is able to distinguish empty
Hardforks map (no hardforks should be enabled) from nil hardforks map
(the default value should be used in this case, i.e. all hardforks
should be active from genesis).

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
It's useful to keep the ordered set of native contract names.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
A part of #3213.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
AnnaShaleva added a commit that referenced this pull request Apr 25, 2024
Ref.
#3402 (comment).

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
@AnnaShaleva
Copy link
Member Author

@roman-khimov, do we need better unit-test coverage for this PR?

pkg/core/native/nativehashes/gen.go Outdated Show resolved Hide resolved
pkg/core/native/nativehashes/gen.go Outdated Show resolved Hide resolved
pkg/core/native/nativehashes/gen.go Outdated Show resolved Hide resolved
pkg/core/native/nativehashes/hashes.go Outdated Show resolved Hide resolved
AnnaShaleva added a commit that referenced this pull request May 8, 2024
Ref.
#3402 (comment).

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
@roman-khimov roman-khimov merged commit b21db99 into master May 10, 2024
19 of 21 checks passed
@roman-khimov roman-khimov deleted the native-upd branch May 10, 2024 08:53
AnnaShaleva added a commit that referenced this pull request May 14, 2024
Ref.
#3402 (comment).

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support HF-based native contract update
2 participants