Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Static Metadata Validation #478

Merged
merged 173 commits into from
Apr 28, 2022
Merged
Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
173 commits
Select commit Hold shift + click to select a range
12313d7
metadata: Implement MetadataHashable for deterministic hashing
lexnv Mar 14, 2022
707a5e5
metadata: Hash `scale_info::Field`
lexnv Mar 14, 2022
3353583
metadata: Hash `scale_info::Variant`
lexnv Mar 14, 2022
ebc588a
metadata: Hash `scale_info::TypeDef`
lexnv Mar 14, 2022
dc727fc
metadata: Hash pallet metadata
lexnv Mar 14, 2022
4690ac0
metadata: Avoid data representation collision via unique identifiers
lexnv Mar 14, 2022
8178351
metadata: Finalize hashing on recursive types
lexnv Mar 14, 2022
b60ebdf
metadata: Cache recursive calls
lexnv Mar 14, 2022
a9b1447
metadata: Move `MetadataHashable` to codegen to avoid cyclic dependency
lexnv Mar 16, 2022
3ece9e3
codegen: Add pallet unique hash
lexnv Mar 16, 2022
c64cc60
metadata: Wrap metadata as owned
lexnv Mar 16, 2022
a24da9a
subxt: Use MetadataHashable wrapper for clients
lexnv Mar 16, 2022
05f1bc8
subxt: Generate runtime pallet uid from metadata
lexnv Mar 16, 2022
b98a93c
Validate metadata compatibility at the pallet level
lexnv Mar 16, 2022
73befba
Update polkadot.rs
lexnv Mar 16, 2022
e576ab2
Modify examples and tests for the new API
lexnv Mar 16, 2022
76f2c9c
codegen: Implement metadata uid
lexnv Mar 17, 2022
58f743c
Update polkadot with TryFrom implementation
lexnv Mar 17, 2022
ea9e233
client: Change `to_runtime_api` to reflect TryFrom changes
lexnv Mar 17, 2022
f9d9956
client: Skip full metadata validation option
lexnv Mar 17, 2022
0ed1f43
codegen: Add option to skip pallet validation for TransactionApi
lexnv Mar 17, 2022
f5665f7
codegen: Add option to skip pallet validation for StorageApi
lexnv Mar 17, 2022
b36555f
Update polkadot.rs with ability to skip pallet validation
lexnv Mar 17, 2022
e8e8792
codegen: Change `MetadataHashable` to per function implementation
lexnv Mar 17, 2022
86b1c7e
codegen: Use metadata hashes functions
lexnv Mar 17, 2022
c0ecf32
subxt: Use metadata hashes functions
lexnv Mar 17, 2022
4431c22
codegen: Make `get_type_uid` private
lexnv Mar 17, 2022
1fa9db4
codegen, subxt: Rename metadata functions `*_uid` to `*_hash`
lexnv Mar 17, 2022
2d6e906
codegen: Update `get_field_hash` to use `codec::Encode`
lexnv Mar 17, 2022
4da2e0e
subxt: Update polkadot.rs
lexnv Mar 17, 2022
ce7a6d7
codegen, subxt: Move metadata check from client to subxt::Metadata
lexnv Mar 18, 2022
7033233
codegen, subxt: Rename metadata check functions to follow `*_hash` na…
lexnv Mar 18, 2022
ed51cd8
codegen: Update polkadot.rs to reflect naming changes
lexnv Mar 18, 2022
0d500eb
codegen: Use `encode_to` for metadata generation
lexnv Mar 18, 2022
1c2f986
codegen: Update polkadot.rs to reflect `encode_to` changes
lexnv Mar 18, 2022
02bb231
codegen: Specific name for visited set
lexnv Mar 18, 2022
13e6112
metadata: Provide cache to hashing functions
lexnv Mar 18, 2022
d388598
metadata: Compute metadata hash by sorted pallets
lexnv Mar 21, 2022
5e6a21d
metadata: Get extrinsic hash
lexnv Mar 21, 2022
45c6453
metadata: Extend metadata hash with extrinsic and metadata type
lexnv Mar 21, 2022
5eb41c9
metadata: Add cache as metadata parameter
lexnv Mar 21, 2022
e33f33a
codegen, subxt: Update metadata hash to use cache
lexnv Mar 21, 2022
c5d89a3
metadata: Implement Default trait for MetadataHasherCache
lexnv Mar 21, 2022
e7af28f
metadata: Add cache for pallets
lexnv Mar 21, 2022
e0779bd
metadata: Move functionality to metadata crate
lexnv Mar 21, 2022
953c031
codegen, subxt: Use subxt-metadata crate
lexnv Mar 21, 2022
b865f18
codegen: Remove metdata hashing functionality
lexnv Mar 21, 2022
26216fa
metadata: Add documentation
lexnv Mar 21, 2022
aad7d2d
metadata: Fix vector capacity to include extrinisc and type hash
lexnv Mar 21, 2022
5de8e83
metadata: Add empty CLI
lexnv Mar 22, 2022
3877505
metadata-cli: Fetch metadata from substrate nodes
lexnv Mar 22, 2022
1809cc7
metadata-cli: Log metadata hashes of provided nodes
lexnv Mar 22, 2022
1957e56
metadata-cli: Group compatible nodes by metadata
lexnv Mar 22, 2022
9e32361
metadata-cli: Simplify hash map insertion
lexnv Mar 22, 2022
70aa7c1
metadata-cli: Move full metadata check to function
lexnv Mar 22, 2022
bfa9c8f
metadata-cli: Group metadata validation at the pallet level
lexnv Mar 22, 2022
c7a1e27
subxt: Persist metadata cache
lexnv Mar 23, 2022
ca8c22c
metadata: Move compatibility cli from subxt-metadata to subxt-cli
lexnv Mar 23, 2022
776250e
metadata: Remove cli from subxt-metadata
lexnv Mar 23, 2022
1c600ee
cli: Fix clippy
lexnv Mar 23, 2022
95a6134
Merge remote-tracking branch 'origin/master' into 398_static_md_check
lexnv Mar 23, 2022
beee3f1
metadata: Fix compatible metadata when pallets are registered in diff…
lexnv Mar 23, 2022
84e049e
tests: Handle result of pallet hashing
lexnv Mar 24, 2022
6635423
metadata: Remove type cache for deterministic hashing
lexnv Mar 24, 2022
1fe787d
metadata: Add test assets from `substrate-node-template` tag `polkado…
lexnv Mar 24, 2022
20055e5
metadata-tests: Check cache hashing for Balances pallet
lexnv Mar 24, 2022
7a9b8e5
metadata: Fix `get_type_hash` clippy issue
lexnv Mar 24, 2022
931fcdb
metadata-tests: Compare one time cache with persistent cache
lexnv Mar 24, 2022
c9ed131
metadata-test: Check metadata hash populates cache for pallets
lexnv Mar 24, 2022
43ae4db
metadata-tests: Simplify `cache_deterministic_hash` test
lexnv Mar 24, 2022
f9ca06d
metadata-tests: Check deterministic metadata for different order pallets
lexnv Mar 24, 2022
09e74d2
metadata: Fix clippy
lexnv Mar 24, 2022
57771e8
codegen: Implement TransactionApiUnchecked for skipping pallet valida…
lexnv Mar 24, 2022
f3f4d72
codegen: Implement StorageApiUnchecked for skipping pallet validation
lexnv Mar 25, 2022
9e4c62c
codegen: Remove skip_pallet_validation boolean
lexnv Mar 25, 2022
f92383c
subxt: Implement ClientUnchecked for skipping metadata validation
lexnv Mar 25, 2022
07cf5f2
Update polkadot.rs
lexnv Mar 25, 2022
6baa8e8
Update examples of rpc_call to skip metadata
lexnv Mar 25, 2022
8378086
subxt: Remove heck dependency
lexnv Mar 28, 2022
5fa5df8
codegen: Add pallet name as an identifier for pallet hashing
lexnv Mar 28, 2022
8da1497
metadata: Implement MetadataHashDetails
lexnv Mar 28, 2022
62a7eec
metadata: Adjust testing to `MetadataHashDetails` interface
lexnv Mar 28, 2022
f026078
codegen: Remove extra `pallet_name`
lexnv Mar 28, 2022
31c016c
subxt: Update polkadot.rs
lexnv Mar 28, 2022
8fcf824
subxt: Fix clippy issue
lexnv Mar 28, 2022
816e916
codegen: Change StorageApi to support `_unchecked` methods
lexnv Mar 29, 2022
58fd3e3
codegen: Change TransactionApi to support `_unchecked` methods
lexnv Mar 29, 2022
e0ef532
subxt: Switch back from `TryFrom` to `From` for `subxt::Client`
lexnv Mar 29, 2022
b0061e4
codegen, subxt: Remove `ClientUnchecked`
lexnv Mar 29, 2022
5ecf7b7
codegen: Expose `validate_metadata` as validation of compatibility me…
lexnv Mar 29, 2022
392da6d
examples: Update to the new interface
lexnv Mar 30, 2022
f2e8dfc
subxt: Update test integration to latest interface
lexnv Mar 30, 2022
f8df12a
subxt: Update polkadot.rs
lexnv Mar 30, 2022
467d707
metadata/tests: Check different pallet index order
lexnv Mar 31, 2022
d20bcd0
metadata/tests: Check recursive type hashing
lexnv Mar 31, 2022
e042c05
metadata/tests: Check recursive types registered in different order
lexnv Mar 31, 2022
b3c4029
metadata/tests: Fix recursive types warning
lexnv Mar 31, 2022
1900091
metadata/tests: Remove test assets
lexnv Mar 31, 2022
252c4d1
metadata/tests: Extend tests to verify cached pallet values
lexnv Mar 31, 2022
964f92c
Merge remote-tracking branch 'origin/master' into 398_static_md_check
lexnv Mar 31, 2022
9278811
Merge remote-tracking branch 'origin/master' into 398_static_md_check
lexnv Apr 4, 2022
551c595
Merge remote-tracking branch 'origin/master' into 398_static_md_check
lexnv Apr 4, 2022
db8fdfb
examples: Add metadata compatiblity example
lexnv Apr 4, 2022
d25c912
examples: Revert balance_transfer to initial form
lexnv Apr 4, 2022
5fb0cdd
codegen: Add ConstantsApi metadata check
lexnv Apr 4, 2022
7293531
tests: Modify tests to accomodate ConstantsApi changes
lexnv Apr 4, 2022
a5415b8
examples: Modify verified version
lexnv Apr 4, 2022
ce05435
subxt: Generate polkadot.rs from `0.9.18-4542a603cc-aarch64-macos`
lexnv Apr 4, 2022
ba95ac8
examples: Update polkadot_metadata.scale from `0.9.18-4542a603cc-aarc…
lexnv Apr 4, 2022
1efd776
metadata: Update documentation
lexnv Apr 4, 2022
dc3b106
tests: Modify default pallet usage
lexnv Apr 4, 2022
bb0b838
metadata/tests: Remove hex dependency
lexnv Apr 4, 2022
3a2d695
metadata/tests: Add MetadataTestType to capture complex types
lexnv Apr 4, 2022
853ba6f
metadata/tests: Update tests to use complex types
lexnv Apr 4, 2022
f0a2b75
metadata/tests: Check metadata correctness via extending pallets
lexnv Apr 4, 2022
545dfd0
metadata/tests: Extend pallet hash with Events
lexnv Apr 4, 2022
63b32bc
metadata/tests: Extend pallet hash with constants
lexnv Apr 4, 2022
7d55527
metadata/tests: Extend pallet hash with error
lexnv Apr 4, 2022
4e30b52
examples: Extend metadata compatibiliy with StorageApi and ConstantsApi
lexnv Apr 4, 2022
dab2beb
Modify comments and documentation
lexnv Apr 5, 2022
a152c6b
Merge remote-tracking branch 'origin/master' into 398_static_md_check
lexnv Apr 6, 2022
4af15fe
metadata: Benchmarks for full validation and pallet validation
lexnv Apr 6, 2022
74d3bc8
Merge remote-tracking branch 'origin/master' into 398_static_md_check
lexnv Apr 6, 2022
58cbc3f
metadata/benches: Fix clippy
lexnv Apr 7, 2022
efa5df1
metadata: Hash metadata just by inspecting the provided pallets
lexnv Apr 7, 2022
414dee6
metadata: Make pallets generic over T for `AsRef<str>`
lexnv Apr 7, 2022
7396595
codegen: Expose the name of the pallets composing the metadata
lexnv Apr 7, 2022
7d1adc3
subxt/tests: Update polkadot.rs with pallets name
lexnv Apr 7, 2022
1f5c5f4
codegen: Obtain metadata hash only by inspecting pallets
lexnv Apr 7, 2022
cbc3727
codegen,subxt: Extend the metadata hash to utilize just pallets
lexnv Apr 7, 2022
a82fad2
subxt/tests: Update polkadot.rs with client metadata has per pallet
lexnv Apr 7, 2022
c96606a
metadata/tests: Test `get_metadata_per_pallet_hash` correctness
lexnv Apr 7, 2022
c558df4
metadata: Fix clippy
lexnv Apr 7, 2022
a6f5650
metadata/benches: Fix decode of metadata
lexnv Apr 7, 2022
8f57e20
metadata: Fix clippy
lexnv Apr 7, 2022
23b382f
[static metadata] validate storage, calls and constants per call (#507)
jsdw Apr 8, 2022
9a08a84
Fix clippy
lexnv Apr 8, 2022
6cbebc9
bench the per-call metadata functions
jsdw Apr 8, 2022
fd4d425
metadata: Add test for `node_template_runtime_variant`
lexnv Apr 8, 2022
640b052
ensure criteron cli opts work
jsdw Apr 8, 2022
1323696
group benchmarks and avoid unwrap issues
jsdw Apr 8, 2022
9a7bbbc
metadata: Check template runtime for handling the pallet swap order case
lexnv Apr 8, 2022
5548b5a
Merge remote-tracking branch 'origin/398_static_md_check' into 398_st…
lexnv Apr 8, 2022
fd00319
metadata: Remove debug logs
lexnv Apr 8, 2022
a0e3b6c
metadata: Optimise by removing field's name and type_name and type's …
lexnv Apr 8, 2022
0ce19bb
metadata: Refactor `get_type_hash` to break recursion earlier
lexnv Apr 8, 2022
eb051a9
subxt: Add tests for `hash_cache`
lexnv Apr 8, 2022
cb2de4a
subxt: Add tests for checking Metadata Inner cache
lexnv Apr 8, 2022
e4e2866
metadata: Check semantic changes inside enum and struct fields
lexnv Apr 11, 2022
f65dabc
metadata: Add enums named differently with compatible semantic meaning
lexnv Apr 11, 2022
6c776eb
metadata: Guard testing of release versions for `node_template_runtime`
lexnv Apr 11, 2022
9c46526
Improve documentation
lexnv Apr 11, 2022
b713993
Update polkadot.rs
lexnv Apr 11, 2022
2d55752
metadata/tests: Manually construct type of `node_template_runtimeL::C…
lexnv Apr 11, 2022
79ea3fe
no more special Call handling, avoid a little cloning, and actually s…
jsdw Apr 13, 2022
a1f9038
remove unused deps and fmt
jsdw Apr 13, 2022
9691f51
RuntimeMetadataLastVersion => RuntimeMetadataV14
jsdw Apr 13, 2022
fd7aa62
remove a bunch of allocations in the metadata hashing, speed up from …
jsdw Apr 13, 2022
a658d6f
update release docs to release metadata crate too
jsdw Apr 13, 2022
06241f4
Merge remote-tracking branch 'origin/master' into 398_static_md_check
lexnv Apr 26, 2022
8c4aca6
subxt: Remove codegen dependency
lexnv Apr 26, 2022
0f7a770
subxt: Replace std RwLock with parking_lot
lexnv Apr 26, 2022
e355ec6
subxt/tests: Add ws address to `TestNodeProcess`
lexnv Apr 26, 2022
73be46c
subxt/tests: Add metadata validation integration test
lexnv Apr 26, 2022
b4c0185
subxt: Allow setting metadata on the ClientBuilder
lexnv Apr 26, 2022
2a625a9
subxt/tests: Check incompatible metadatas
lexnv Apr 26, 2022
633c3f0
metadata: Fix constant hashing for deterministic output
lexnv Apr 26, 2022
542dec6
subxt/tests: Check metadata validation for constants
lexnv Apr 26, 2022
74be005
subxt/tests: Test validation for calls
lexnv Apr 27, 2022
83f0ba4
subxt/tests: Test validation for storage
lexnv Apr 27, 2022
1abd3ca
Merge remote-tracking branch 'origin/master' into 398_static_md_check
lexnv Apr 27, 2022
b0e8b4c
subxt: Expose `set_metadata` for testing only
lexnv Apr 27, 2022
460c18c
subxt: Guard metadata tests under integration-tests
lexnv Apr 28, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
8 changes: 6 additions & 2 deletions codegen/src/api/calls.rs
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ pub fn generate_calls(
"Call",
);
let (call_structs, call_fns): (Vec<_>, Vec<_>) = struct_defs
.iter()
.iter_mut()
.map(|(variant_name, struct_def)| {
let (call_fn_args, call_args): (Vec<_>, Vec<_>) =
match struct_def.fields {
Expand All @@ -76,12 +76,16 @@ pub fn generate_calls(
};

let pallet_name = &pallet.name;
let call_name = variant_name;
let call_name = &variant_name;
let struct_name = &struct_def.name;
let call_hash = subxt_metadata::get_call_hash(metadata, pallet_name, call_name)
.unwrap_or_else(|_| abort_call_site!("Metadata information for the call {}_{} could not be found", pallet_name, call_name));

let fn_name = format_ident!("{}", variant_name.to_snake_case());
// Propagate the documentation just to `TransactionApi` methods, while
// draining the documentation of inner call structures.
let docs = struct_def.docs.take();
// The call structure's documentation was stripped above.
let call_struct = quote! {
#struct_def

Expand Down
1 change: 1 addition & 0 deletions codegen/src/api/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -453,6 +453,7 @@ where
fields,
Some(parse_quote!(pub)),
type_gen,
var.docs(),
);
(var.name().to_string(), struct_def)
})
Expand Down
7 changes: 5 additions & 2 deletions metadata/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -302,8 +302,11 @@ pub fn get_constant_hash(
.find(|c| c.name == constant_name)
.ok_or(NotFound::Item)?;

let hash = get_type_hash(&metadata.types, constant.ty.id(), &mut HashSet::new());
Ok(hash)
let mut bytes = get_type_hash(&metadata.types, constant.ty.id(), &mut HashSet::new());
bytes = xor(bytes, hash(constant.name.as_bytes()));
bytes = xor(bytes, hash(&constant.value));

Ok(bytes)
}

/// Obtain the hash for a specific call, or an error if it's not found.
Expand Down
2 changes: 1 addition & 1 deletion subxt/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@ log = "0.4.14"
serde = { version = "1.0.124", features = ["derive"] }
serde_json = "1.0.64"
thiserror = "1.0.24"
parking_lot = "0.12.0"

subxt-codegen = { version = "0.20.0", path = "../codegen" }
subxt-macro = { version = "0.20.0", path = "../macro" }
subxt-metadata = { version = "0.20.0", path = "../metadata" }

Expand Down
21 changes: 18 additions & 3 deletions subxt/src/client.rs
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ use derivative::Derivative;
pub struct ClientBuilder {
url: Option<String>,
client: Option<RpcClient>,
metadata: Option<Metadata>,
page_size: Option<u32>,
}

Expand All @@ -61,6 +62,7 @@ impl ClientBuilder {
Self {
url: None,
client: None,
metadata: None,
page_size: None,
}
}
Expand All @@ -83,6 +85,14 @@ impl ClientBuilder {
self
}

/// Set the metadata.
///
/// *Note:* Metadata will no longer be downloaded from the runtime node.
pub fn set_metadata(mut self, metadata: Metadata) -> Self {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
pub fn set_metadata(mut self, metadata: Metadata) -> Self {
#[cfg(test)]
pub fn set_metadata(mut self, metadata: Metadata) -> Self {

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, cfg(test) is not active for integration tests. I would place this under integration_tests feature and follow up with #515 😄 .

self.metadata = Some(metadata);
self
}

/// Creates a new Client.
pub async fn build<T: Config>(self) -> Result<Client<T>, BasicError> {
let client = if let Some(client) = self.client {
Expand All @@ -92,14 +102,19 @@ impl ClientBuilder {
crate::rpc::ws_client(url).await?
};
let rpc = Rpc::new(client);
let (metadata, genesis_hash, runtime_version, properties) = future::join4(
rpc.metadata(),
let (genesis_hash, runtime_version, properties) = future::join3(
rpc.genesis_hash(),
rpc.runtime_version(None),
rpc.system_properties(),
)
.await;
let metadata = metadata?;

let metadata = if let Some(metadata) = self.metadata {
metadata
} else {
let metadata = rpc.metadata().await;
metadata?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
let metadata = rpc.metadata().await;
metadata?
rpc.metadata().await?

};

Ok(Client {
rpc,
Expand Down
6 changes: 2 additions & 4 deletions subxt/src/metadata/hash_cache.rs
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,10 @@
// You should have received a copy of the GNU General Public License
// along with subxt. If not, see <http://www.gnu.org/licenses/>.

use parking_lot::RwLock;
use std::{
borrow::Cow,
collections::HashMap,
sync::RwLock,
};

/// A cache with the simple goal of storing 32 byte hashes against pallet+item keys
Expand All @@ -41,7 +41,6 @@ impl HashCache {
let maybe_hash = self
.inner
.read()
.unwrap()
.get(&PalletItemKey::new(pallet, item))
.copied();

Expand All @@ -50,7 +49,7 @@ impl HashCache {
}

let hash = f()?;
self.inner.write().unwrap().insert(
self.inner.write().insert(
PalletItemKey::new(pallet.to_string(), item.to_string()),
hash,
);
Expand Down Expand Up @@ -96,7 +95,6 @@ mod tests {
cache
.inner
.read()
.unwrap()
.get(&PalletItemKey::new(pallet, item))
.unwrap(),
&value.unwrap()
Expand Down
2 changes: 1 addition & 1 deletion subxt/src/metadata/metadata_type.rs
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ use std::{
};

/// Metadata error.
#[derive(Debug, thiserror::Error)]
#[derive(Debug, thiserror::Error, PartialEq)]
pub enum MetadataError {
/// Module is not in metadata.
#[error("Pallet not found")]
Expand Down