WIP: Shwap prototype #3184
Draft
WIP: Shwap prototype #3184
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ation method for ids
- allow release of verifier without verify call
Wondertan
reviewed
Mar 27, 2024
share/shwap/row_id.go
Outdated
| "github.com/celestiaorg/celestia-node/share/store/file" | ||
| ) | ||
|
|
||
| // TODO(@walldiss): maybe move into separate subpkg? |
Wondertan
reviewed
Mar 27, 2024
share/shwap/row_id.go
Outdated
Comment on lines
83
to
84
| // enough size. Otherwise Caller might not see the changes. | ||
| data = binary.BigEndian.AppendUint64(data, rid.Height) |
Wondertan
reviewed
Mar 27, 2024
| type Getter struct { | ||
| // TODO(@walldiss): why not blockservice? |
There was a problem hiding this comment.
Because we don't need to cache on disk in most cases
Wondertan
reviewed
Mar 27, 2024
share/shwap/getter/getter.go
Outdated
| if ctx.Err() != nil { | ||
| return nil, ctx.Err() | ||
| } | ||
| return nil, fmt.Errorf("not all shares were found") |
There was a problem hiding this comment.
blocks, because now its generalized
Wondertan
reviewed
Mar 27, 2024
share/shwap/getter/getter.go
Outdated
| datas[idx] = d | ||
| return nil | ||
| if data.RowIndex < uint16(from) || data.RowIndex >= uint16(to) { | ||
| // should never happen, because rows should be verified against root by the time they are returned |
There was a problem hiding this comment.
Why do we need if then, if its never happens?
Wondertan
reviewed
Mar 27, 2024
share/shwap/getter/getter.go
Outdated
Comment on lines
140
to
145
| if len(blks) != sqrLn/2 { | ||
| if ctx.Err() != nil { | ||
| return nil, ctx.Err() | ||
| } | ||
| return nil, fmt.Errorf("not all rows were found") | ||
| } |
There was a problem hiding this comment.
duplicates logic in getBlocks so can be removed
Wondertan
reviewed
Mar 27, 2024
share/shwap/sample_id.go
Outdated
Comment on lines
137
to
139
| func (sid SampleID) Release() { | ||
| sampleVerifiers.Delete(sid) | ||
| } |
There was a problem hiding this comment.
little reminder to godoc this
This PR introduces `full` and `bridge` node pruning via the `--experimental-pruning` flag. Support is included for nodes that start from scratch with pruning enabled and also for `archival` (nodes retaining all historical blocks) that enable the `--experimental-pruning` flag. _Note that this PR does not support the conversion of a pruned node into an archival one explicitly (it would not support re-syncing deleted blocks)._ With pruning enabled, `full` and `bridge` nodes' block stores can be expected not to exceed ~4TB (as the upper bound). In follow-up PRs (hardening), the following features can be expected: - [x] discovery for archival nodes for archival sync - [ ] inverted_index / light node pruning - [ ] include more metrics for errors TODO: - [x] clean up some TODOs - [x] fix one flakey unit test - [x] change values back to the actual (GC cycle, sampling window, pruning window, etc). - [x] figure out whether to store error in pruner checkpoint - [x] fix issue with pruning genesis block via findPruneableHeaders - [x] metrics for failed prunes - [x] set a sane default for max pruneable / consider removing `MaxPruneablePerGC` as now context timeouts are on a per block basis - [ ] dedup findPruneableHeader test utility - [x] badger dep --------- Co-authored-by: Ryan <ryanford@poluglottos.com>
Wondertan
reviewed
Mar 28, 2024
nodebuilder/share/config.go
Outdated
| @@ -18,6 +18,7 @@ type Config struct { | |||
| EDSStoreParams *store.Parameters | |||
|
|
|||
| UseShareExchange bool | |||
| UseShareSwap bool | |||
There was a problem hiding this comment.
Better keep it Shwap and rename the above one ot shrex, so there is less confusions around namings
# Conflicts: # core/listener_test.go # go.mod # go.sum # nodebuilder/core/module.go # nodebuilder/share/module.go # share/availability/light/availability.go # share/availability/light/availability_test.go # share/availability/light/testing.go # share/eds/byzantine/byzantine.go # share/eds/inverted_index.go # share/eds/metrics.go # share/eds/retriever_test.go # share/eds/store_test.go # share/getters/getter_test.go # share/getters/shrex.go # share/getters/shrex_test.go # share/ipld/corrupted_data_test.go # share/store/store_options.go
allow id marshallers to not depend on size of provided data container fix id types to have both pointer and value receivers R
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Prototype for shwap protocol and storage. Contain all changes required for shwap implementation. This PR is not meant for merge. All changes will be extracted from prototype into smaller and nicer PRs.
Based on: #2675
Tracking issue: #2971