Postmortem of the 1.29.3 release

[nagisa]

On 2022-11-29 we received a report of an issue in the implementation of the neard gas accounting logic that would cause a node to crash when executing a specially crafted contract. This family of issues is particularly concerning for the NEAR protocol – upon a restart this node would attempt executing the same malicious transaction again. This sort of a crash loop would ultimately lead to the entire network failing to make any more progress. Fortunately, this issue has never been exploited to that effect.

The gas accounting logic in neard runtime tracks the remaining gas and its consumption across multiple variables: used_gas, burnt_gas and promises_gas are some examples. Once the gas is found to be exhausted, these variables are then adjusted to their final values and a correct type of an error is returned (for legacy reasons there are different ways to exhaust gas.) These variables can store integer values in the range [0; 2^64). When a result of some operation (such as an addition or subtraction) does not fit into this range, the execution will terminate with a crash, unless the code explicitly specifies a different mode of handling (such as saturating, wrapping or checked).

The report we received has demonstrated that it is possible to set up the promises_gas variable such that a next crafted invocation of a host function would predictably cause a wrap-around in one of the computations part of gas exhaustion logic. The provided proof-of-concept code would create a promise to increase the promise_gas variable and then invoke the promise_batch_action_function_call. In order to execute this host function, the runtime will attempt to burn gas proportional to the value of the 2nd argument (method_name_len) passed into this function. This charge occurs before it is verified to be in bounds of the contract memory, giving an ability for contracts to attempt a charge of arbitrarily high amounts of gas.

The crash would then occur with this argument chosen such that method_name_len * cost(read_memory_byte) < 2^64 < method_name_len * cost(read_memory_byte) + promise_gas. Additionally, we have found that the results of these computations were an input to an assertion. Even if we chose to wrap-around the results in production builds, it would still have been straightforward for a malicious contract to fail this assertion (and thus crash the program!) For further details on what a malicious contract might have looked like, we encourage readers to check out the tests added in #8179.

All things considered, the issue was assigned the highest severity rating and the releases containing these fixes were assigned CODE_RED.

This vulnerability was caused by a bug, initially introduced as part of the effort to significantly improve the performance of gas tracking during the contract execution, back in November 2021. The faulty code was already identified to be wonky during said improvements (#5148) but we never found the resources to come back and improve on it. As part of handling this report we also identified a couple other process deficiencies that could have prevented this vulnerability in the first place. For example, clippy – a linter for Rust code – was flagging the faulty code with one of its lints, and had we set it up in our CI, it would potentially have drawn attention to the fault. We’re looking into setting this up and reviewing the lints sooner rather than later. Relevant issues tracking this work are #8145 and #8210.

As part of the rollout process we have also identified multiple pain points with how the security sensitive fixes make their way to a release. Our current process requires us to make public the commits fixing an issue before we can begin producing and testing the binary releases. In most cases it is going to be non-trivial to figure out how to exploit the project given a public fix, but given how straightforward the fix was for this specific vulnerability, we felt the current process was suboptimal and will be looking at adjusting that as well.

Timeline

• 2022-11-29 We received the external report of a vulnerability about neard crashing when built with debug assertions and executing specially constructed contracts.
• 2022-11-30 The report has been investigated, and although we concluded that the exact issue as reported by the external contributor is not a big deal, we discovered a second closely related issue that can occur in production builds, with much the same effects. SEV0 (CODE_RED) has been assigned to this issue.
• 2022-11-30 A fix has been drafted to mitigate the issue.
• 2022-12-01 The fix has been pushed to the 1.29.0 and 1.30.0 branches.
• 2022-12-02 Builds with the fix have been produced and rolled out for testing.
• 2022-12-05 Testing has concluded and releases containing these fixes were published.
• 2022-12-07 Majority stake runs the fixed code. A full fix (with comments and tests) has been submitted against master as gas: fix the computation of self.promises_gas #8179.