Add Browser Support #295

linden · 2024-02-18T23:49:23Z

This PR adds support for the browser using indexeddb as a storage backend for headerfs instead of the file system.

Why IndexedDB?

I'd initially chosen to go with localStorage for simplicity but due to recent rate limiting changes it made syncing past a couple 100 blocks freeze, indexddb does not have share these limitations. I'd also looked into emulating the file system but it turned out to add a lot of complexity with a performance loss over just using indexeddb directly.

Browser Testing.

Since Go only provides an executor for your native platform testing needs to be done via a custom executor targeting the browser. I've written my own called wasmexec which also provides file system access for ease of testing, you can run tests via the following:

# install wasmexec from the add-file-system branch while #1 is in progress.
go install github.com/linden/wasmexec@add-file-system

# target javascript/wasm and execute via wasmexec.
GOOS=js GOARCH=wasm go test -exec wasmexec ./...

I'm planning on provider this as part of a Github Action in the future.

`btcd` Changes.

Running this in the browser without wasmexec's file system access requires a couple of changes to btcd's addrmgr & btcutil as they require a file system. Changes made in btcsuite/btcd#2124.

bbolt has a heavy dependency on memory mapping and the file system which isn't yet supported in the browser wasm runtime, so instead tempdb keeps everything in-memory so we can support both browser and native targets while testing.

linden/wasmexec#2

Roasbeef

Nice work! This is super exciting, we have a number of other projects in the background that stand to tremendously benefit from this line of work.

Did an initial round of review, will also check out the source of the indexeddb wrapper you created so I can get a handle on what type of concurrency control we can count on.

Roasbeef · 2024-02-20T01:35:43Z

bamboozle_unit_test.go

-	db, err := walletdb.Create(
-		"bdb", tempDir+"/weks.db", true, dbOpenTimeout,
-	)
+	db, err := walletdb.Create("tempdb", "weks.db")


Perhaps we can instead modify the tests to use a build tag to specify tempdb vs bdb?

We could have a NewTestDB func defined for the build w/ and w/o the build tag defined.

Roasbeef · 2024-02-20T01:37:10Z

headerfs/index_test.go

@@ -173,12 +173,18 @@ func TestHeaderStorageFallback(t *testing.T) {
 		t.Fatalf("error setting new tip: %v", err)
 	}
 	for _, header := range newHeaderEntries {
-		if err := hIndex.truncateIndex(&header.hash, true); err != nil {
+		// Copy the header hash so we can create a iteration-specific pointer.
+		// https://github.com/golang/go/discussions/56010#discussion-4441437.


If we bump the Go version to 1.22 here, then we can bind the new behavior directly.

Roasbeef · 2024-02-20T01:40:08Z

headerfs/file.go

-// be found.
-type ErrHeaderNotFound struct {
-	error
+// headerBufPool is a pool of bytes.Buffer that will be re-used by the various


I think this would be easier to review if one commit moved the contents into this new file, then subsequent commits start to add the build tag and additional refactoring.

Roasbeef · 2024-02-20T01:40:34Z

headerfs/store.go

-// writing headers to disk.
-var headerBufPool = sync.Pool{
-	New: func() interface{} { return new(bytes.Buffer) },
+	// Close and delete the BlockHeaderStore.


godoc comment should start with the method name

Roasbeef · 2024-02-20T01:45:54Z

headerfs/js.go

+// "header" as it deals only with raw bytes, and leaves it to a higher layer to
+// interpret those raw bytes accordingly.
+//
+// TODO(roasbeef): quickcheck coverage.


stray todo/copy pasta

Roasbeef · 2024-02-20T01:46:41Z

headerfs/js.go

+type headerStore struct {
+	mtx sync.RWMutex // nolint:structcheck // false positive because used as embedded struct only
+
+	idb *indexeddb.DB


What if we had this package implement the walletdb interface? Then we could just drop it in here, as well as some other locations/projects that use that central KV-store abstraction.

Roasbeef · 2024-02-20T01:47:39Z

headerfs/js.go

+	hdrs := tx.Store(headersStore)
+
+	// Count the amount of headers.
+	count, err := hdrs.Count()


Under the hood, does this hit a fast index, or will it need to enumerate all the contents to get a final sum?

Roasbeef · 2024-02-20T01:49:07Z

headerfs/js.go

+// headerStore combines a IndexedDB store of headers within a flat file in addition
+// to a database which indexes that flat file. Together, these two abstractions
+// can be used in order to build an indexed header store for any type of
+// "header" as it deals only with raw bytes, and leaves it to a higher layer to


I think we should revise this godoc a bit to note the high level structure of this impl. IIUC, it maps height => header within indexeddb. It's the based off a similar bucket structure as boltdb, with the top level key here being headersStore.

Roasbeef · 2024-02-20T01:49:53Z

headerfs/js.go

+	}
+
+	// Delete the header.
+	return hdrs.Delete(height + 1)


not zero indexed?

Roasbeef · 2024-02-20T01:50:43Z

headerfs/js.go

+
+func (h *headerStore) singleTruncate() error {
+	// Get the current height.
+	height, genesis, err := h.height()


Need to check out the underlying implementation, but: should the height be obtained after the transaction is created? Otherwise, two writers may start the transaction with an inconsistent view.

Chinwendu20 · 2024-02-20T08:28:52Z

headerfs/file.go

@@ -1,18 +1,211 @@
+//go:build !windows && !js && !wasm


Why is windows excluded?

Chinwendu20 · 2024-02-20T08:39:29Z

headerfs/store.go

-	fhs := &FilterHeaderStore{
-		fStore,
+	bhs := &blockHeaderStore{
+		headerStore: hStore,


Do you think we can make this headerStore field an interface? since we want to interact with the methods only and there are currently two implementations for this. Each implementation smartly separated by your use of build tags.

Chinwendu20 · 2024-02-20T09:08:54Z

headerfs/js.go

+// determined by the hight value.
+func (h *headerStore) readHeader(height uint32) (wire.BlockHeader, error) {
+	// Create a new read/write transaction, scoped to headers.
+	tx, err := h.idb.NewTransaction([]string{headersStore}, indexeddb.ReadWriteMode)


Do you think this should transaction should be created in ReadMode only?

Chinwendu20

Thanks for the PR. I left some comments and suggestions. The commit title should be:
package name: Message.

lightninglabs-deploy · 2024-05-21T18:20:58Z

@linden, remember to re-request review from reviewers when ready

linden added 5 commits February 18, 2024 14:37

feat: move to tempdb for tests

8891244

bbolt has a heavy dependency on memory mapping and the file system which isn't yet supported in the browser wasm runtime, so instead tempdb keeps everything in-memory so we can support both browser and native targets while testing.

fix(headerfs): use iteration-specific pointer when truncating

9b9791b

feat(headerfs): separate file-system dependencies in headerStore

bb4bacf

feat(headerfs): add indexeddb storage backend

e6f5a70

fix: skip tests that require starting a process on js

6fba217

linden/wasmexec#2

linden mentioned this pull request Feb 18, 2024

Add Browser Support (Neutrino) btcsuite/btcd#2124

Open

Roasbeef requested changes Feb 20, 2024

View reviewed changes

Chinwendu20 reviewed Feb 20, 2024

View reviewed changes

headerfs/file.go

@@ -1,18 +1,211 @@

//go:build !windows && !js && !wasm

Copy link

Contributor

Chinwendu20 Feb 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is windows excluded?

Chinwendu20 reviewed Feb 20, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Browser Support #295

Add Browser Support #295

linden commented Feb 18, 2024 •

edited

Roasbeef left a comment

Roasbeef Feb 20, 2024

Roasbeef Feb 20, 2024

Roasbeef Feb 20, 2024

Roasbeef Feb 20, 2024

Roasbeef Feb 20, 2024

Roasbeef Feb 20, 2024

Roasbeef Feb 20, 2024

Roasbeef Feb 20, 2024

Roasbeef Feb 20, 2024

Roasbeef Feb 20, 2024

Roasbeef Feb 20, 2024

Chinwendu20 Feb 20, 2024

Chinwendu20 Feb 20, 2024

Chinwendu20 Feb 20, 2024

Chinwendu20 left a comment •

edited

lightninglabs-deploy commented May 21, 2024

Add Browser Support #295

Are you sure you want to change the base?

Add Browser Support #295

Conversation

linden commented Feb 18, 2024 • edited

Why IndexedDB?

Browser Testing.

btcd Changes.

Roasbeef left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Chinwendu20 left a comment • edited

Choose a reason for hiding this comment

lightninglabs-deploy commented May 21, 2024

linden commented Feb 18, 2024 •

edited

`btcd` Changes.

Chinwendu20 left a comment •

edited