-
Notifications
You must be signed in to change notification settings - Fork 35.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bundle 5/n] Prune g_chainman usage in RPC modules #21391
[Bundle 5/n] Prune g_chainman usage in RPC modules #21391
Conversation
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code review ACK last 6 commits only a041676. Many chainmans.
- 384c56d rpc/*,rest: Add review-only assertion to EnsureChainman (14/19)
- b6d3847 rpc/blockchain: Use existing NodeContext (15/19)
- 2b0fb0e rpc/mining: Use existing NodeContext (16/19)
- 8e3d455 rpc/rawtx: Use existing NodeContext (17/19)
- cbe4f7b rest: Pass in NodeContext to rest_block (18/19)
- a041676 rest: Use existing NodeContext (19/19)
a041676
to
6b89f53
Compare
The scripted-diff in the previous commit should have removed all calls to functions like: Ensure(?!Any)\(const std::any& (context|ctx)\), so we can remove them now.
Organize local variables/references such that: 1. There is always a `ChainstateManager` reference before any `LOCK(cs_main)`. 2. NodeContext references are used with Ensure*() functions introduced in previous commit where appropriate to avoid duplicate assertions.
In all rest/rpc-related modules, if there are multiple calls to ActiveChain{,State}(), and the calls fall under the same ::cs_main lock, we can simply take a local reference and use/reuse it instead of calling ActiveChain{,State}() again and again.
6d92313
to
586190f
Compare
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code review ACK 586190f. Since last review, no changes to existing commits, just some simple new commits added: three new commits renaming std::any Ensure functions (scripted diff commit and manual pre/post commits), and one new commit factoring out a repeated ActiveChain()
call made in a loop. Thanks for the updates!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
utACK 586190f
It's not a blocker, but I don't like the proliferation of code outside validation holding m_chain
. In general, I don't think there's any reason for code outside validation to take a reference to a CChain
- it should be able to call a higher abstraction function on chainman to read whatever data it needs.
{ | ||
ChainstateManager& chainman = EnsureChainman(node); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it'd be better to move this EnsureChainman()
to the top of the function. If chainman doesn't exist, then it's better to immediately exit from the function.
pcursor = std::unique_ptr<CCoinsViewCursor>(::ChainstateActive().CoinsDB().Cursor()); | ||
CChainState& active_chainstate = chainman.ActiveChainstate(); | ||
active_chainstate.ForceFlushStateToDisk(); | ||
pcursor = std::unique_ptr<CCoinsViewCursor>(active_chainstate.CoinsDB().Cursor()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
While you're touching this, perhaps change it to use std::make_unique
. That's the standard way to set unique pointers now that we're on c++17.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't Cursor return a unique_ptr?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, you're right. Cursor()
is passing ownership to the caller.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm going to mark this comment as resolved since this PR shouldn't make any changes here. I think Cursor()
's return type should be changed in a separate PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(unresolved comment to make it easier to see all that needs to be addressed in follow-ups)
@@ -2342,8 +2391,9 @@ static RPCHelpMan getblockfilter() | |||
const CBlockIndex* block_index; | |||
bool block_was_connected; | |||
{ | |||
ChainstateManager& chainman = EnsureAnyChainman(request.context); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Again, chainman is a requirement for this function. I think it makes sense to place the assumption at the top of the function.
@@ -44,11 +44,12 @@ | |||
* or from the last difficulty change if 'lookup' is nonpositive. | |||
* If 'height' is nonnegative, compute the estimate at the time when a given block was found. | |||
*/ | |||
static UniValue GetNetworkHashPS(int lookup, int height) { | |||
CBlockIndex *pb = ::ChainActive().Tip(); | |||
static UniValue GetNetworkHashPS(int lookup, int height, const CChain& active_chain) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Even though it's not annotated as such, I think that holding a reference to m_chain
and calling methods on it requires holding cs_main
. Perhaps it would be a good idea to either:
- annotate this function and add an assertion that cs_main is held; or
- pass in a
&Chainman
, then inside the function lock cs_main, get theCChain&
, and call the methods on thatCChain&
.
I think (2) is nicer - best to limit scope of locks as much as possible.
LOCK(cs_main); | ||
return GetNetworkHashPS(!request.params[0].isNull() ? request.params[0].get_int() : 120, !request.params[1].isNull() ? request.params[1].get_int() : -1); | ||
return GetNetworkHashPS(!request.params[0].isNull() ? request.params[0].get_int() : 120, !request.params[1].isNull() ? request.params[1].get_int() : -1, chainman.ActiveChain()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This line is long and difficult to read. Consider something like this:
diff --git a/src/rpc/mining.cpp b/src/rpc/mining.cpp
index 0cef310c50..b60783761d 100644
--- a/src/rpc/mining.cpp
+++ b/src/rpc/mining.cpp
@@ -102,7 +102,9 @@ static RPCHelpMan getnetworkhashps()
{
ChainstateManager& chainman = EnsureAnyChainman(request.context);
LOCK(cs_main);
- return GetNetworkHashPS(!request.params[0].isNull() ? request.params[0].get_int() : 120, !request.params[1].isNull() ? request.params[1].get_int() : -1, chainman.ActiveChain());
+ const int blocks{!request.params[0].isNull() ? request.params[0].get_int() : 120};
+ const int height{!request.params[1].isNull() ? request.params[1].get_int() : -1};
+ return GetNetworkHashPS(blocks, height, chainman.ActiveChain());
},
};
}
It almost certainly compiles to the same thing, is much easier to read and reduces the risk of precedence bugs.
@@ -111,7 +113,8 @@ static bool GenerateBlock(ChainstateManager& chainman, CBlock& block, uint64_t& | |||
|
|||
{ | |||
LOCK(cs_main); | |||
IncrementExtraNonce(&block, ::ChainActive().Tip(), extra_nonce); | |||
CHECK_NONFATAL(std::addressof(::ChainActive()) == std::addressof(chainman.ActiveChain())); | |||
IncrementExtraNonce(&block, chainman.ActiveChain().Tip(), extra_nonce); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For a follow-up: IncrementExtraNonce()
doesn't require cs_main. In fact, we could just get a height using chainman.ActiveHeight()
and pass that to IncrementExtraNonce()
.
const CTxMemPool& mempool = EnsureMemPool(request.context); | ||
NodeContext& node = EnsureAnyNodeContext(request.context); | ||
const CTxMemPool& mempool = EnsureMemPool(node); | ||
ChainstateManager& chainman = EnsureChainman(node); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I haven't commented on all of these examples, but I really do think readability is helped by placing the Ensure*()
assumptions at the top of the rpc methods.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I looked at d7824ac 🐃
Show signature and timestamp
Signature:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
I looked at d7824acdb9 🐃
-----BEGIN PGP SIGNATURE-----
iQGzBAEBCgAdFiEE+rVPoUahrI9sLGYTzit1aX5ppUgFAlwqrYAACgkQzit1aX5p
pUjV+AwAt/NhM4OI3plsZuJBMc1+PcsEelcHiTdtzMCG+qIWOyxE7zU11aZPYkJ2
xhT+gSEwHQxSWENkLb7hf5eU4l23JXvA9JD7WwO3yAptxMQwpmTPzOzyefAONC/z
2Up7r09SMieAwKKTLuHiAXTSuoA9vqx3dw361XsFhcnM1Pu06zqkYzYumduL9yP6
fSj22cFffihAaZEIg3leD2LE1CXWcaq80so3kWAv741hXZRx+vPssKUIdNDW2oNr
50pbZTaiKc7T1wU16KwxeY9xBj1x12pg4WrMhYRDAXIUzwP4gk18gmhhth9EvD6q
v0bfig+bk4bXt07HvEftTT7yWpMu381mNGZGrbljgGS4NomCRbBKfuvDNqsX17LB
z8dn54TrW5Pr3//Q46xWZLxhH9f04qH5D6nuhxoghdRXP52Zh1h7A/tRrm2tFJ1t
TNw7dlDWAUZs8NUlY1Jq6nZlm9DpueU+5/0tubX9fmqYO1k8IFI3T77btrsYC/0L
0gSquAFL
=bVUN
-----END PGP SIGNATURE-----
Timestamp of file with hash d97e47afcf29e46d1bcd8c3ce2c34513261879f42b7516ce21deecae7598dee6 -
{ | ||
CMutableTransaction tx{*block.vtx.at(0)}; | ||
tx.vout.erase(tx.vout.begin() + GetWitnessCommitmentIndex(block)); | ||
block.vtx.at(0) = MakeTransactionRef(tx); | ||
|
||
GenerateCoinbaseCommitment(block, WITH_LOCK(::cs_main, assert(std::addressof(g_chainman.m_blockman) == std::addressof(blockman)); return blockman.LookupBlockIndex(block.hashPrevBlock)), Params().GetConsensus()); | ||
WITH_LOCK(::cs_main, assert(g_chainman.m_blockman.LookupBlockIndex(block.hashPrevBlock) == prev_block)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't understand this commit. Is the goal to remove this assert? If yes, how are new callers protected from accidentally passing the wrong pointer? If no, you'll need to pass blockman again, making this commit useless.
Also, this seems to put more code burden on the caller, duplicating the LookupBlockIndex at each call site.
It might help to specify a goal of the commit here. Is the goal to avoid locking cs_main twice instead of only once? If yes, a simpler solution would be to just pass chainman (from the rpc context) instead of the blockman reference (which needs cs_main to be taken from chainman).
src/rest.cpp
Outdated
@@ -250,8 +251,9 @@ static bool rest_block(const std::any& context, | |||
CBlockIndex* tip = nullptr; | |||
{ | |||
LOCK(cs_main); | |||
tip = ::ChainActive().Tip(); | |||
pblockindex = g_chainman.m_blockman.LookupBlockIndex(hash); | |||
ChainstateManager& chainman = EnsureChainman(context); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This may crash the node if the chainman is not available.
terminate called after throwing an instance of 'UniValue'
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
review ACK 586190f 🍯
Show signature and timestamp
Signature:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
review ACK 586190f0b4740457cb86cba632e3d64e6dfe9b0c 🍯
-----BEGIN PGP SIGNATURE-----
iQGzBAEBCgAdFiEE+rVPoUahrI9sLGYTzit1aX5ppUgFAlwqrYAACgkQzit1aX5p
pUjIYgv/YeuLpuAboBxi/xa/PWLK/q/57E44wyfEF9cZHpOCgFzkR+gfJI1ouI1C
bBUBnk0CkOqLfFwi8a1Vzel2zXYvTOtYJ6OPpeau+bxJVhlaI8ysFRuI8oomn4Tx
ypQdNYi0bNMIIiuwezoYX+dPMkBRA8Ox8JuMZ/SUuO1ZH8F66s2U9EstshDhFiS5
cN9L9kujU0ZovHAnutbEF5zSFDqR/brrmuhCg0ACE5rCla87AAQqe/29SIcKvbls
TiV9EEHWj5V8eRy/CLg2ch+bGOFp4vISWJBITYdho7hnwMLRMTHC/K5RvsfR703R
t3reCFwd4whXkcBGN6z43f16GM3fvRNJVKKT7rQ0FB6n0oVjkFE3Gqxln5Ke209N
wDAbn99WerhV8tGhOhgYTcG1N9awQBTH4aZEuOt2MrtGxYkUEXw7+mchbrQ4iPPs
75AoZ51CaEWESzIYWRV82IBtNCtD0UUvGUOuo3E3fXCqrVKBmU7JnSuMDQ3UaPR5
QVKk2RIm
=kNPA
-----END PGP SIGNATURE-----
Timestamp of file with hash 3e1b6d60dc76018210f94a83c678a1e0f9c63606e06fb1b3d34881ee98abc0d8 -
@@ -181,7 +181,7 @@ static bool rest_headers(const std::any& context, | |||
headers.reserve(count); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
6fb65b4: the scripted diff will modify files that are not tracked by git. Could use ... -- $(git grep -l Ensure)
{ | ||
LOCK(::cs_main); | ||
coins_view = &active_chainstate.CoinsDB(); | ||
blockman = &active_chainstate.m_blockman; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This doesn't require cs_main
.
CCoinsView* coins_view; | ||
BlockManager* blockman; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's slightly confusing to take the references from CoinsDB()
and m_blockman
and take their addresses to make pointers, and then just dereference those pointers. Better just to use references as the local variables.
… modules 7a799c9 index: refactor-only: Reuse CChain ref (Carl Dong) db33cde index: Add chainstate member to BaseIndex (Carl Dong) f4a47a1 bench: Use existing chainman in AssembleBlock (Carl Dong) 91226eb bench: Use existing NodeContext in DuplicateInputs (Carl Dong) e6b4aa6 miner: Pass in chainman to RegenerateCommitments (Carl Dong) 9ecade1 rest: Add GetChainman function and use it (Carl Dong) fc1c282 rpc/blockchain: Use existing blockman in gettxoutsetinfo (Carl Dong) Pull request description: Overall PR: bitcoin#20158 (tree-wide: De-globalize ChainstateManager) The first 2 commits are fixups addressing review for the last bundle: bitcoin#21391 NEW note: 1. I have opened bitcoin#21766 which keeps track of potential improvements where the flaws already existed before the de-globalization work, please post on that issue about these improvements, thanks! Note to reviewers: 1. This bundle may _apparently_ introduce usage of `g_chainman` or `::Chain(state|)Active()` globals, but these are resolved later on in the overall PR. [Commits of overall PR](https://github.com/bitcoin/bitcoin/pull/20158/commits) 2. There may be seemingly obvious local references to `ChainstateManager` or other validation objects which are not being used in callers of the current function in question, this is done intentionally to **_keep each commit centered around one function/method_** to ease review and to make the overall change systematic. We don't assume anything about our callers. Rest assured that once we are considering that particular caller in later commits, we will use the obvious local references. [Commits of overall PR](https://github.com/bitcoin/bitcoin/pull/20158/commits) 3. When changing a function/method that has many callers (e.g. `LookupBlockIndex` with 55 callers), it is sometimes easier (and less error-prone) to use a scripted-diff. When doing so, there will be 3 commits in sequence so that every commit compiles like so: 1. Add `new_function`, make `old_function` a wrapper of `new_function`, divert all calls to `old_function` to `new_function` **in the local module only** 2. Scripted-diff to divert all calls to `old_function` to `new_function` **in the rest of the codebase** 3. Remove `old_function` ACKs for top commit: jarolrod: ACK 7a799c9 ariard: Code Review ACK 7a799c9 fjahr: re-ACK 7a799c9 MarcoFalke: review ACK 7a799c9 🌠 ryanofsky: Code review ACK 7a799c9. Basically no change since last review except fixed rebase conflicts and a new comment about REST Ensure() jamesob: conditional ACK 7a799c9 ([`jamesob/ackr/21767.1.dongcarl.bundle_6_n_prune_g_chai`](https://github.com/jamesob/bitcoin/tree/ackr/21767.1.dongcarl.bundle_6_n_prune_g_chai)) Tree-SHA512: 531c00ddcb318817457db2812d9a9d930bc664e58e6f7f1c746350732b031dd624270bfa6b9f49d8056aeb6321d973f0e38e4ff914acd6768edd8602c017d10e
Summary: Also pass in appropriate object to: BIP9SoftForkDescPushBack PR context: De-globalize ChainstateManager This is a backport of [[bitcoin/bitcoin#21391 | core#21391]] bitcoin/bitcoin@d485e81 Test Plan: `ninja all check-all` Reviewers: #bitcoin_abc, Fabien Reviewed By: #bitcoin_abc, Fabien Differential Revision: https://reviews.bitcoinabc.org/D11510
Summary: Also pass in appropriate object to: - GetNetworkHashPS - [gG]enerateBlock{,s} Also: - Misc style/constness changes This is a partial backport of [[bitcoin/bitcoin#21391 | core#21391]] Depends on D11507 bitcoin/bitcoin@60dc05a Test Plan: `ninja all check-all` Reviewers: #bitcoin_abc, Fabien Reviewed By: #bitcoin_abc, Fabien Subscribers: Fabien Differential Revision: https://reviews.bitcoinabc.org/D11511
Summary: Also pass in appropriate object to: - TxToJSON This is a partial backport of [[bitcoin/bitcoin#21391 | core#21391]] bitcoin/bitcoin@7be0671 Depends on D11511 Test Plan: `ninja all check-all` Reviewers: #bitcoin_abc, deadalnix Reviewed By: #bitcoin_abc, deadalnix Differential Revision: https://reviews.bitcoinabc.org/D11512
Summary: Note: The third commit fixes an issue introduced by the second, when the node context does not have a chainmanager. > rest: Pass in NodeContext to rest_block bitcoin/bitcoin@3f08934 > rest: Use existing NodeContext bitcoin/bitcoin@d7824ac > rest: Add GetChainman function and use it > > This is not the cleanest change but: > > 1. It fixes the erroneous use of RPC's Ensure*() in rest.cpp, which > cause crashes in REST contexts. > > RPC code wraps all calls in a try/except, REST code does not. > Ensure*(), being part of RPC, expects that its throw's will get > caught by a try/except. But if you use Ensure*() in REST code, since > it doesn't have a try/except wrap, a crash will happen. > > 2. It is consistent with other functions like GetMemPool. > > Someone can probably make this a bit prettier. bitcoin/bitcoin@9ecade1 This is a backport of [[bitcoin/bitcoin#21391 | core#21391]] [6 & 7/14] Test Plan: `ninja all check-all` Reviewers: #bitcoin_abc, Fabien Reviewed By: #bitcoin_abc, Fabien Subscribers: Fabien Differential Revision: https://reviews.bitcoinabc.org/D11584
Summary: Add alt `Ensure*` functions accepting `NodeContext` and rename the versions accepting a `std::any` `EnsureAny*()`. Note that the changes in rest.cpp are not applicable because of the out-of-sequence backport of commit bitcoin/bitcoin@9ecade1 in D11585 > rpc: Add alt Ensure* functions acepting NodeContext bitcoin/bitcoin@306b1cd > rpc: Add renamed EnsureAny*() functions > > The renaming avoids overloading mistakes arising out of the untyped `std::any` argument. bitcoin/bitcoin@1570c7e > scripted-diff: rest/rpc: Use renamed EnsureAny*() > > ``` > -BEGIN VERIFY SCRIPT- > sed -i -E 's@Ensure([^(]+)(\((request\.|)context\))@EnsureAny\1\2@g' \ > -- src/rest.cpp src/rpc/*.cpp > > -END VERIFY SCRIPT- > ``` bitcoin/bitcoin@6fb65b4 > rest/rpc: Remove now-unused old Ensure functions > > The scripted-diff in the previous commit should have removed all calls > to functions like: Ensure(?!Any)\(const std::any& (context|ctx)\), so we > can remove them now. bitcoin/bitcoin@038854f This is a backport of [[bitcoin/bitcoin#21391 | core#21391]] [8, 9, 10 & 11/14] Depends on D11584 Test Plan: `ninja all check-all` Reviewers: #bitcoin_abc, Fabien Reviewed By: #bitcoin_abc, Fabien Differential Revision: https://reviews.bitcoinabc.org/D11586
Summary: Organize local variables/references such that: 1. There is always a `ChainstateManager` reference before any `LOCK(cs_main)`. 2. NodeContext references are used with `Ensure*()` functions introduced in previous commit where appropriate to avoid duplicate assertions. This is a backport of [[bitcoin/bitcoin#21391 | core#21391]] [12/14] bitcoin/bitcoin@6a3d192 Depends on D11586 Test Plan: With clang and debug: `ninja all check-all` Reviewers: #bitcoin_abc, Fabien Reviewed By: #bitcoin_abc, Fabien Subscribers: Fabien Differential Revision: https://reviews.bitcoinabc.org/D11587
Summary: One of the two occurences in Bitcoin Core was already fixed in Bitcoin ABC. This is a backport of [[bitcoin/bitcoin#21391 | core#21391]] [13/14] bitcoin/bitcoin@f999139 Depends on D11587 Test Plan: `ninja all check-all` Reviewers: #bitcoin_abc, Fabien Reviewed By: #bitcoin_abc, Fabien Differential Revision: https://reviews.bitcoinabc.org/D11588
Summary: In all rest/rpc-related modules, if there are multiple calls to `ActiveChain{,State}()`, and the calls fall under the same `::cs_main` lock, we can simply take a local reference and use/reuse it instead of calling `ActiveChain{,State}()` again and again. This concludes backport of [[bitcoin/bitcoin#21391 | core#21391]] [14/14] bitcoin/bitcoin@586190f Depends on D11588 Test Plan: `ninja all check-all` Reviewers: #bitcoin_abc, Fabien Reviewed By: #bitcoin_abc, Fabien Subscribers: Fabien Differential Revision: https://reviews.bitcoinabc.org/D11589
Overall PR: #20158 (tree-wide: De-globalize ChainstateManager)
Based on:
Note to reviewers:
g_chainman
or::Chain(state|)Active()
globals, but these are resolved later on in the overall PR. Commits of overall PRChainstateManager
or other validation objects which are not being used in callers of the current function in question, this is done intentionally to keep each commit centered around one function/method to ease review and to make the overall change systematic. We don't assume anything about our callers. Rest assured that once we are considering that particular caller in later commits, we will use the obvious local references. Commits of overall PRLookupBlockIndex
with 55 callers), it is sometimes easier (and less error-prone) to use a scripted-diff. When doing so, there will be 3 commits in sequence so that every commit compiles like so:new_function
, makeold_function
a wrapper ofnew_function
, divert all calls toold_function
tonew_function
in the local module onlyold_function
tonew_function
in the rest of the codebaseold_function