Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pebble datastore support for Kubo #9659

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

Pebble datastore support for Kubo #9659

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
92a5ab5
Pebble datastore support for Kubo
hsanjuan Feb 21, 2023
21a3509
Adjust BytesPerSync to 100MB
hsanjuan Feb 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
11 changes: 11 additions & 0 deletions config/init.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,17 @@ func badgerSpec() map[string]interface{} {
}
}

func pebbleSpec() map[string]interface{} {
return map[string]interface{}{
"type": "measure",
"prefix": "pebble.datastore",
"child": map[string]interface{}{
"type": "pebbleds",
"path": "pebbleds",
},
}
}

func flatfsSpec() map[string]interface{} {
return map[string]interface{}{
"type": "mount",
Expand Down
13 changes: 13 additions & 0 deletions config/profile.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,19 @@ This profile may only be applied when first initializing the node.`,
return nil
},
},
"pebbleds": {
Description: `Configures the node to use the experimental Pebble datastore.

Use this datastore for large IPFS blockstores (with multi-terabyte data).

This profile may only be applied when first initializing the node.`,

InitOnly: true,
Transform: func(c *Config) error {
c.Datastore.Spec = pebbleSpec()
return nil
},
},
"lowpower": {
Description: `Reduces daemon overhead on the system. May affect node
functionality - performance of content discovery and data
Expand Down
12 changes: 11 additions & 1 deletion docs/examples/kubo-as-a-library/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ require (
require (
bazil.org/fuse v0.0.0-20200117225306-7b5117fecadc // indirect
github.com/AndreasBriese/bbloom v0.0.0-20190825152654-46b345b51c96 // indirect
github.com/DataDog/zstd v1.4.5 // indirect
github.com/alecthomas/units v0.0.0-20210927113745-59d0afb8317a // indirect
github.com/alexbrainman/goissue34681 v0.0.0-20191006012335-3fc7a47baff5 // indirect
github.com/benbjohnson/clock v1.3.0 // indirect
Expand All @@ -27,6 +28,11 @@ require (
github.com/ceramicnetwork/go-dag-jose v0.1.0 // indirect
github.com/cespare/xxhash v1.1.0 // indirect
github.com/cespare/xxhash/v2 v2.2.0 // indirect
github.com/cockroachdb/errors v1.8.1 // indirect
github.com/cockroachdb/logtags v0.0.0-20190617123548-eb05cc24525f // indirect
github.com/cockroachdb/pebble v0.0.0-20230217215838-f01d8eff3f8b // indirect
github.com/cockroachdb/redact v1.0.8 // indirect
github.com/cockroachdb/sentry-go v0.6.1-cockroachdb.2 // indirect
github.com/containerd/cgroups v1.0.4 // indirect
github.com/coreos/go-systemd/v22 v22.5.0 // indirect
github.com/crackcomm/go-gitignore v0.0.0-20170627025303-887ab5e44cc3 // indirect
Expand Down Expand Up @@ -73,6 +79,7 @@ require (
github.com/ipfs/go-ds-flatfs v0.5.1 // indirect
github.com/ipfs/go-ds-leveldb v0.5.0 // indirect
github.com/ipfs/go-ds-measure v0.2.0 // indirect
github.com/ipfs/go-ds-pebble v0.1.0 // indirect
github.com/ipfs/go-fetcher v1.6.1 // indirect
github.com/ipfs/go-filestore v1.2.0 // indirect
github.com/ipfs/go-fs-lock v0.0.7 // indirect
Expand Down Expand Up @@ -112,9 +119,11 @@ require (
github.com/jackpal/go-nat-pmp v1.0.2 // indirect
github.com/jbenet/go-temp-err-catcher v0.1.0 // indirect
github.com/jbenet/goprocess v0.1.4 // indirect
github.com/klauspost/compress v1.15.12 // indirect
github.com/klauspost/compress v1.15.15 // indirect
github.com/klauspost/cpuid/v2 v2.2.3 // indirect
github.com/koron/go-ssdp v0.0.3 // indirect
github.com/kr/pretty v0.3.0 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/libp2p/go-buffer-pool v0.1.0 // indirect
github.com/libp2p/go-cidranger v1.1.0 // indirect
github.com/libp2p/go-doh-resolver v0.4.0 // indirect
Expand Down Expand Up @@ -170,6 +179,7 @@ require (
github.com/quic-go/quic-go v0.32.0 // indirect
github.com/quic-go/webtransport-go v0.5.1 // indirect
github.com/raulk/go-watchdog v1.3.0 // indirect
github.com/rogpeppe/go-internal v1.6.1 // indirect
github.com/samber/lo v1.36.0 // indirect
github.com/spaolacci/murmur3 v1.1.0 // indirect
github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7 // indirect
Expand Down
99 changes: 97 additions & 2 deletions docs/examples/kubo-as-a-library/go.sum
View file
Open in desktop

Large diffs are not rendered by default.

12 changes: 11 additions & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ require (
github.com/cenkalti/backoff/v4 v4.1.3
github.com/ceramicnetwork/go-dag-jose v0.1.0
github.com/cheggaaa/pb v1.0.29
github.com/cockroachdb/pebble v0.0.0-20230217215838-f01d8eff3f8b
github.com/coreos/go-systemd/v22 v22.5.0
github.com/dustin/go-humanize v1.0.0
github.com/elgris/jsondiff v0.0.0-20160530203242-765b5c24c302
Expand All @@ -26,6 +27,7 @@ require (
github.com/ipfs/go-ds-flatfs v0.5.1
github.com/ipfs/go-ds-leveldb v0.5.0
github.com/ipfs/go-ds-measure v0.2.0
github.com/ipfs/go-ds-pebble v0.1.0
github.com/ipfs/go-fetcher v1.6.1
github.com/ipfs/go-filestore v1.2.0
github.com/ipfs/go-fs-lock v0.0.7
Expand Down Expand Up @@ -114,13 +116,18 @@ require (

require (
github.com/AndreasBriese/bbloom v0.0.0-20190825152654-46b345b51c96 // indirect
github.com/DataDog/zstd v1.4.5 // indirect
github.com/Kubuxu/go-os-helper v0.0.1 // indirect
github.com/alecthomas/units v0.0.0-20210927113745-59d0afb8317a // indirect
github.com/alexbrainman/goissue34681 v0.0.0-20191006012335-3fc7a47baff5 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cenkalti/backoff v2.2.1+incompatible // indirect
github.com/cespare/xxhash v1.1.0 // indirect
github.com/cespare/xxhash/v2 v2.2.0 // indirect
github.com/cockroachdb/errors v1.8.1 // indirect
github.com/cockroachdb/logtags v0.0.0-20190617123548-eb05cc24525f // indirect
github.com/cockroachdb/redact v1.0.8 // indirect
github.com/cockroachdb/sentry-go v0.6.1-cockroachdb.2 // indirect
github.com/containerd/cgroups v1.0.4 // indirect
github.com/crackcomm/go-gitignore v0.0.0-20170627025303-887ab5e44cc3 // indirect
github.com/cskr/pubsub v1.0.2 // indirect
Expand Down Expand Up @@ -166,9 +173,11 @@ require (
github.com/ipfs/go-peertaskqueue v0.8.1 // indirect
github.com/ipld/edelweiss v0.2.0 // indirect
github.com/jackpal/go-nat-pmp v1.0.2 // indirect
github.com/klauspost/compress v1.15.12 // indirect
github.com/klauspost/compress v1.15.15 // indirect
github.com/klauspost/cpuid/v2 v2.2.3 // indirect
github.com/koron/go-ssdp v0.0.3 // indirect
github.com/kr/pretty v0.3.0 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/libp2p/go-buffer-pool v0.1.0 // indirect
github.com/libp2p/go-cidranger v1.1.0 // indirect
github.com/libp2p/go-flow-metrics v0.1.0 // indirect
Expand Down Expand Up @@ -213,6 +222,7 @@ require (
github.com/quic-go/quic-go v0.32.0 // indirect
github.com/quic-go/webtransport-go v0.5.1 // indirect
github.com/raulk/go-watchdog v1.3.0 // indirect
github.com/rogpeppe/go-internal v1.6.1 // indirect
github.com/rs/cors v1.7.0 // indirect
github.com/samber/lo v1.36.0 // indirect
github.com/spaolacci/murmur3 v1.1.0 // indirect
Expand Down