Skip to content

Commit

Permalink
fix: state listener observe writes at wrong time (#13516)
Browse files Browse the repository at this point in the history
* fix: state listener observe writes at wrong time

Closes: #13457

Currently state listener is notified when the cache store write, which happens in commit event only, which breaks the current design.
The solution (as discussed in the issue) is to listen state writes on rootmulti store only.

It also changes the file streamer to output single data file for the writes in the whole block, since we can't distinguish writes from different stage of abci events.

It adds new config items for file streamer:
- streamers.file.output-metadata
- streamers.file.stop-node-on-error
- streamers.file.fsync

* synchronous abci call, and format doc

* fix comment

* update file streamer readme and fix typos

* typo

* fix: state listener observe writes at wrong time

Closes: #13457

Currently state listener is notified when the cache store write, which happens in commit event only, which breaks the current design.
The solution (as discussed in the issue) is to listen state writes on rootmulti store only.

It also changes the file streamer to output single data file for the writes in the whole block, since we can't distinguish writes from different stage of abci events.

It adds new config items for file streamer:
- streamers.file.output-metadata
- streamers.file.stop-node-on-error
- streamers.file.fsync

synchronous abci call, and format doc

fix comment

update file streamer readme and fix typos

typo

* improve UX of file streamer, make it immediately usable after enabled

- set default value to write_dir.
- make write_dir based on home directory by default.
- auto-create the directory if not exists.

* get homePage from opts

Co-authored-by: Marko <marbar3778@yahoo.com>
(cherry picked from commit 1f91ee2)

# Conflicts:
#	CHANGELOG.md
#	baseapp/streaming.go
#	store/streaming/constructor.go
#	store/streaming/file/service.go
  • Loading branch information
yihuang authored and mergify[bot] committed Dec 2, 2022
1 parent 3a51a15 commit 9df91b3
Show file tree
Hide file tree
Showing 36 changed files with 3,042 additions and 593 deletions.
20 changes: 20 additions & 0 deletions CHANGELOG.md
Expand Up @@ -192,6 +192,21 @@ Ref: https://keepachangelog.com/en/1.0.0/
* [#13794](https://github.com/cosmos/cosmos-sdk/pull/13794) Most methods on `types/module.AppModule` have been moved to
extension interfaces. `module.Manager.Modules` is now of type `map[string]interface{}` to support in parallel the new
`cosmossdk.io/core/appmodule.AppModule` API.
<<<<<<< HEAD
=======
* (signing) [#13701](https://github.com/cosmos/cosmos-sdk/pull/) Add `context.Context` as an argument `x/auth/signing.VerifySignature`.
* (x/group) [#13876](https://github.com/cosmos/cosmos-sdk/pull/13876) Add `GetMinExecutionPeriod` method on DecisionPolicy interface.
* (x/auth)[#13780](https://github.com/cosmos/cosmos-sdk/pull/13780) Querying with `id` (type of int64) in `AccountAddressByID` grpc query now throws error, use account-id(type of uint64) instead.
* (snapshots) [14048](https://github.com/cosmos/cosmos-sdk/pull/14048) Move the Snapshot package to the store package. This is done in an effort group all storage related logic under one package.
* (baseapp) [#14050](https://github.com/cosmos/cosmos-sdk/pull/14050) refactor `ABCIListener` interface to accept go contexts
* (store) [#13516](https://github.com/cosmos/cosmos-sdk/pull/13516) Update State Streaming APIs:
* Add method `ListenCommit` to `ABCIListener`
* Move `ListeningEnabled` and `AddListener` methods to `CommitMultiStore`
* Remove `CacheWrapWithListeners` from `CacheWrap` and `CacheWrapper` interfaces
* Remove listening APIs from the caching layer (it should only listen to the `rootmulti.Store`)
* Add three new options to file streaming service constructor.
* Modify `ABCIListener` such that any error from any method will always halt the app via `panic`
>>>>>>> 1f91ee2ee (fix: state listener observe writes at wrong time (#13516))
### CLI Breaking Changes

Expand Down Expand Up @@ -220,6 +235,11 @@ extension interfaces. `module.Manager.Modules` is now of type `map[string]interf
* (server) [#13778](https://github.com/cosmos/cosmos-sdk/pull/13778) Set Cosmos SDK default endpoints to localhost to avoid unknown exposure of endpoints.
* (x/auth) [#13877](https://github.com/cosmos/cosmos-sdk/pull/13877) Handle missing account numbers during `InitGenesis`.
* (x/gov) [#13918](https://github.com/cosmos/cosmos-sdk/pull/13918) Fix propagation of message errors when executing a proposal.
<<<<<<< HEAD
=======
* (baseapp)[#14049](https://github.com/cosmos/cosmos-sdk/pull/14049) Fix state sync when interval is zero.
* (store) [#13516](https://github.com/cosmos/cosmos-sdk/pull/13516) Fix state listener that was observing writes at wrong time.
>>>>>>> 1f91ee2ee (fix: state listener observe writes at wrong time (#13516))
### Deprecated

Expand Down

0 comments on commit 9df91b3

Please sign in to comment.