Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SQLite migration #150

Merged
merged 42 commits into from
Jun 28, 2022
Merged

SQLite migration #150

Show file tree
Hide file tree
Changes from 41 commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
415d668
Makefile: remove sqlc
jsign Jun 27, 2022
0c8e560
sqlc: migrate to database/sql style
jsign Jun 27, 2022
560695a
sqlstore: work on json transformations
jsign Jun 27, 2022
ad3eb0e
refactor migrations
jsign Jun 27, 2022
13dccb7
Makefile: remove sqlc rule
jsign Jun 27, 2022
fe8db98
mod: add sqlite3 dependency
jsign Jun 27, 2022
330b511
internal/tableland: move from pgx to sql and add new helper for Table…
jsign Jun 27, 2022
b38a865
pkg/sqlstore: fix some models, queries and DTO transformations
jsign Jun 27, 2022
8545ced
txnprocessor: start doing sqlite3 migration
jsign Jun 27, 2022
ee7a32e
processor: receipts table semi-migration
jsign Jun 27, 2022
058db28
processor: fix queries, sql runtime error detection, and remaining ne…
jsign Jun 27, 2022
7d9bbc1
eventprocessor: fix tests for sqlite3
jsign Jun 27, 2022
615a82d
sqlstore: fix impl and tests
jsign Jun 27, 2022
d716a1a
noncetracker: sqlite3 fixes and tests
jsign Jun 27, 2022
eb00877
general signatures modifications
jsign Jun 27, 2022
84b0eaa
postgres: keep removing dependencies
jsign Jun 27, 2022
54e3eff
mod: update to latest mattn/sqlite3
jsign Jun 27, 2022
60c9b47
mesa: integration tests fixes
jsign Jun 27, 2022
e36dbf0
paralellize more tests
jsign Jun 27, 2022
d004214
processor: fix update_at valeus
jsign Jun 27, 2022
150226e
row to json marshaling work & make tests run faster
jsign Jun 27, 2022
1a66b31
processor: rename ctids to rowids
jsign Jun 27, 2022
bfcc349
eventprocessor: add IndexInBlock feature
jsign Jun 27, 2022
de57f19
db: create table indexes and unique constraints
jsign Jun 27, 2022
1616acc
privileges: improve ergonomics of bitfields
jsign Jun 27, 2022
39d667e
.
jsign Jun 27, 2022
3ab73bc
create base directory & event feed optimizations
jsign Jun 27, 2022
ad7f4a8
lint pass
jsign Jun 27, 2022
989c898
tools: remove postgres tool
jsign Jun 27, 2022
11fff11
update deps & remove deleted field in configs
jsign Jun 27, 2022
c8a08f7
processor: use mutating statement operation to distinguish inserts
jsign Jun 27, 2022
251be4d
cmd: remove unneeded configs
jsign Jun 27, 2022
66ee09d
cleanups
jsign Jun 27, 2022
a1ba600
ci: test retries
jsign Jun 27, 2022
3c20da5
Makefile: enable race flag
jsign Jun 27, 2022
e2cca3e
sql: simplify created_at default values
jsign Jun 27, 2022
3e9a90e
tests: use ElementsMatch to ignore ordering
jsign Jun 27, 2022
52204ea
Makefile: remove sqlc var definition
jsign Jun 28, 2022
36c0073
comment typo
jsign Jun 28, 2022
2ee5c71
tests: use ElementsMatch in more places
jsign Jun 28, 2022
010d3f8
rename files
jsign Jun 28, 2022
c98074b
test: avoid shadowing and use high-order func
jsign Jun 28, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,4 +35,4 @@ jobs:
restore-keys: |
${{ runner.os }}-go-
- name: Test
run: make test
run: for i in 1 2 3; do make test && break || sleep 1; done
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool tunning to repeat make test up to 3 times (if needed) if some test is flaky. On a second (or third) run the test cache will be leveraged, so only the potential flaky tests would be run again.

6 changes: 5 additions & 1 deletion .golangci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,12 +14,16 @@ linters:
- whitespace
- godot
- lll
- sqlclosecheck
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added this extra linter that sounds useful.


issues:
exclude-use-default: false

exclude:
- stutters

run:
timeout: 30m

skip-dirs:
- "pkg/sqlstore/impl/system/internal/db"
Comment on lines +28 to +29
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The .go files here are where the previous sqlc files existed, which had a top comment saying that the code was automatically generated, which by default the linters ignore. Now that we don't use sqlic, those files aren't autogenerated, so just avoid lints there as usual.

9 changes: 4 additions & 5 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ HTTP_PORT ?= 8080
GCP_PROJECT=textile-310716

GO_BINDATA=go run github.com/go-bindata/go-bindata/v3/go-bindata@v3.1.3
SQLC=go run github.com/kyleconroy/sqlc/cmd/sqlc@v1.13.0
GOVVV=go run github.com/ahmetb/govvv@v0.3.0

GOVVV_FLAGS=$(shell $(GOVVV) -flags -version $(BIN_VERSION) -pkg $(shell go list ./buildinfo))
Expand All @@ -32,11 +31,11 @@ ethereum-testerc721a:
go run github.com/ethereum/go-ethereum/cmd/abigen@v1.10.17 --abi ./pkg/tables/impl/ethereum/test/erc721aQueryable/abi.json --pkg erc721aQueryable --type Contract --out pkg/tables/impl/ethereum/test/erc721aQueryable/erc721aQueryable.go --bin pkg/tables/impl/ethereum/test/erc721aQueryable/bytecode.bin
.PHONY: ethereum-testerc721a


system-sql-assets:
cd pkg/sqlstore/impl/system && $(GO_BINDATA) -pkg migrations -prefix migrations/ -o migrations/migrations.go -ignore=migrations.go migrations && $(SQLC) generate; cd -;
cd pkg/sqlstore/impl/system && $(GO_BINDATA) -pkg migrations -prefix migrations/ -o migrations/migrations.go -ignore=migrations.go migrations
jsign marked this conversation as resolved.
Show resolved Hide resolved
.PHONY: system-sql-assets


# Building and publishing image to GCP

build-api:
Expand All @@ -62,8 +61,8 @@ publish:

# Test

test:
go test -v ./...
test:
go test ./... -race
Comment on lines +64 to +65
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We didn't have the -race flag enabled here!
I removed the -v flag since that adds a lot of noise if we want to see the output if the tests fail since it shows logs also for passed tests. This way we have more signal into what it's important.

.PHONY: test

# Lint
Expand Down
52 changes: 27 additions & 25 deletions cmd/api/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,21 +2,22 @@ package main

import (
"encoding/json"
"fmt"
"flag"
"os"
"path"
"strings"

"github.com/omeid/uconfig"
"github.com/omeid/uconfig/plugins"
"github.com/omeid/uconfig/plugins/file"
"github.com/rs/zerolog/log"
"github.com/textileio/go-tableland/internal/tableland"
)

// configFilename is the filename of the config file automatically loaded.
var configFilename = "config.json"

type config struct {
Impl string `default:"mesa"` // service implementation (mock or mesa)
HTTP struct {
Port string `default:"8080"` // HTTP port (e.g. 8080)

Expand All @@ -26,13 +27,6 @@ type config struct {
Gateway struct {
ExternalURIPrefix string `default:"https://testnet.tableland.network"`
}
DB struct {
Host string `default:"database"`
Port string `default:"5432"`
User string `default:"dev_user"`
Pass string `default:"dev_password"`
Name string `default:"dev_database"`
}
TableConstraints TableConstraints
QueryConstraints QueryConstraints
Metrics struct {
Expand All @@ -42,10 +36,6 @@ type config struct {
Human bool `default:"false"`
Debug bool `default:"false"`
}
AdminAPI struct {
Username string `default:""`
Password string `default:""`
}
Chains []ChainConfig
}

Expand Down Expand Up @@ -74,10 +64,9 @@ type ChainConfig struct {
PrivateKey string `default:""`
}
EventFeed struct {
ChainAPIBackoff string `default:"15s"`
MaxBlocksFetchSize int `default:"10000"`
MinBlockDepth int `default:"5"`
NewBlockTimeout string `default:"30s"`
ChainAPIBackoff string `default:"15s"`
MinBlockDepth int `default:"5"`
NewBlockTimeout string `default:"30s"`
}
EventProcessor struct {
BlockFailedExecutionBackoff string `default:"10s"`
Expand All @@ -90,22 +79,35 @@ type ChainConfig struct {
}
}

func setupConfig() *config {
fileBytes, err := os.ReadFile(configFilename)
fileStr := string(fileBytes)
func setupConfig() (*config, string) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some changes here to support the new --dir flag.
It loads the config.json from the directory folder, and also returns the parsed directory folder path so in main.go we can read the database file from there.

flagDirPath := flag.String("dir", "${HOME}/.tableland", "Directory where the configuration and DB exist")
flag.Parse()
if flagDirPath == nil {
log.Fatal().Msg("--dir is null")
return nil, "" // Helping the linter know the next line is safe.
}
dirPath := os.ExpandEnv(*flagDirPath)

_ = os.MkdirAll(dirPath, 0755)
Comment on lines +83 to +91
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the --dir flag via the flag package, and expand env vars.
If the folder doesn't exist, create it.


var plugins []plugins.Plugin
if err != os.ErrNotExist {
fileStr = os.ExpandEnv(fileStr)
fullPath := path.Join(dirPath, configFilename)
configFileBytes, err := os.ReadFile(fullPath)
if os.IsNotExist(err) {
log.Info().Str("configFilePath", fullPath).Msg("config file not found")
} else if err != nil {
log.Fatal().Str("configFilePath", fullPath).Err(err).Msg("opening config file")
} else {
fileStr := os.ExpandEnv(string(configFileBytes))
plugins = append(plugins, file.NewReader(strings.NewReader(fileStr), json.Unmarshal))
}

conf := &config{}
c, err := uconfig.Classic(&conf, file.Files{}, plugins...)
if err != nil {
fmt.Printf("invalid configuration: %s", err)
c.Usage()
os.Exit(1)
log.Fatal().Err(err).Msg("invalid configuration")
}

return conf
return conf, dirPath
Comment on lines +94 to +112
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There was some incorrect error handling here that was fixed.
Also, I removed the erroring style of os.Exist in favor of log.Fatal() so the output in the logs is coherent with what we do in main.go (and rest of the project).

}
20 changes: 5 additions & 15 deletions cmd/api/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import (
"context"
"fmt"
"net/http"
"path"
"sync"
"time"

Expand Down Expand Up @@ -36,7 +37,7 @@ import (
)

func main() {
config := setupConfig()
config, dirPath := setupConfig()
logging.SetupLogger(buildinfo.GitCommit, config.Log.Debug, config.Log.Human)

// Validator instrumentation configuration.
Expand All @@ -48,12 +49,8 @@ func main() {
}

databaseURL := fmt.Sprintf(
"postgres://%s:%s@%s:%s/%s?sslmode=disable&timezone=UTC",
config.DB.User,
config.DB.Pass,
config.DB.Host,
config.DB.Port,
config.DB.Name,
"file://%s?_busy_timeout=5000&_foreign_keys=on&_journal_mode=WAL",
path.Join(dirPath, "database.db"),
Comment on lines +52 to +53
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is explained in the PR description.

)

parserOpts := []parsing.Option{
Expand Down Expand Up @@ -112,12 +109,6 @@ func main() {
Handler: router.Handler(),
}

// Admin endpoint configuration.
if config.AdminAPI.Password == "" {
log.Warn().
Msg("no admin api password set")
}

go func() {
if err := server.ListenAndServe(); err != nil {
if err == http.ErrServerClosed {
Expand Down Expand Up @@ -172,7 +163,7 @@ func createChainIDStack(

systemStore, err := sqlstoreimpl.NewInstrumentedSystemStore(config.ChainID, store)
if err != nil {
return chains.ChainStack{}, fmt.Errorf("instrumenting system store pgx: %s", err)
return chains.ChainStack{}, fmt.Errorf("instrumenting system store: %s", err)
}

conn, err := ethclient.Dial(config.Registry.EthEndpoint)
Expand Down Expand Up @@ -242,7 +233,6 @@ func createChainIDStack(
}
efOpts := []eventfeed.Option{
eventfeed.WithChainAPIBackoff(chainAPIBackoff),
eventfeed.WithMaxBlocksFetchSize(config.EventFeed.MaxBlocksFetchSize),
eventfeed.WithMinBlockDepth(config.EventFeed.MinBlockDepth),
eventfeed.WithNewBlockTimeout(newBlockTimeout),
}
Expand Down
35 changes: 4 additions & 31 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,19 +9,15 @@ require (
github.com/google/uuid v1.3.0
github.com/gorilla/mux v1.8.0
github.com/hetiansu5/urlquery v1.2.7
github.com/jackc/pgconn v1.12.1
github.com/jackc/pgproto3/v2 v2.3.0
github.com/jackc/pgtype v1.11.0
github.com/jackc/pgx/v4 v4.16.1
github.com/mattn/go-sqlite3 v1.14.13
github.com/omeid/uconfig v1.2.0
github.com/ory/dockertest/v3 v3.9.1
github.com/pganalyze/pg_query_go/v2 v2.1.0
github.com/rs/zerolog v1.27.0
github.com/sethvargo/go-limiter v0.7.2
github.com/spruceid/siwe-go v0.2.0
github.com/stretchr/testify v1.7.2
github.com/stretchr/testify v1.7.4
github.com/textileio/cli v1.0.2
github.com/textileio/go-log/v2 v2.1.3-gke-2
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.32.0
go.opentelemetry.io/contrib/instrumentation/runtime v0.32.0
go.opentelemetry.io/otel v1.7.0
Expand All @@ -33,26 +29,17 @@ require (

require (
cloud.google.com/go v0.99.0 // indirect
github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 // indirect
github.com/Microsoft/go-winio v0.5.2 // indirect
github.com/Nvveen/Gotty v0.0.0-20120604004816-cd527374f1e5 // indirect
github.com/StackExchange/wmi v0.0.0-20180116203802-5d049714c4a6 // indirect
github.com/VictoriaMetrics/fastcache v1.6.0 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/btcsuite/btcd v0.20.1-beta // indirect
github.com/cenkalti/backoff/v4 v4.1.3 // indirect
github.com/census-instrumentation/opencensus-proto v0.3.0 // indirect
github.com/cespare/xxhash/v2 v2.1.2 // indirect
github.com/cncf/udpa/go v0.0.0-20210930031921-04548b0d99d4 // indirect
github.com/cncf/xds/go v0.0.0-20211130200136-a8f946100490 // indirect
github.com/containerd/continuity v0.3.0 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/dchest/uniuri v0.0.0-20200228104902-7aecb25e1fe5 // indirect
github.com/deckarep/golang-set v1.8.0 // indirect
github.com/docker/cli v20.10.14+incompatible // indirect
github.com/docker/docker v20.10.13+incompatible // indirect
github.com/docker/go-connections v0.4.0 // indirect
github.com/docker/go-units v0.4.0 // indirect
github.com/edsrzf/mmap-go v1.0.0 // indirect
github.com/envoyproxy/go-control-plane v0.10.1 // indirect
github.com/envoyproxy/protoc-gen-validate v0.6.2 // indirect
Expand All @@ -62,12 +49,10 @@ require (
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-ole/go-ole v1.2.1 // indirect
github.com/go-stack/stack v1.8.0 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/protobuf v1.5.2 // indirect
github.com/golang/snappy v0.0.4 // indirect
github.com/google/go-cmp v0.5.7 // indirect
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect
github.com/googleapis/gax-go/v2 v2.1.1 // indirect
github.com/gorilla/websocket v1.4.2 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
Expand All @@ -76,24 +61,15 @@ require (
github.com/hashicorp/hcl v1.0.0 // indirect
github.com/holiman/bloomfilter/v2 v2.0.3 // indirect
github.com/holiman/uint256 v1.2.0 // indirect
github.com/imdario/mergo v0.3.12 // indirect
github.com/jackc/chunkreader/v2 v2.0.1 // indirect
github.com/jackc/pgio v1.0.0 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20200714003250-2b9c44734f2b // indirect
github.com/jackc/puddle v1.2.1 // indirect
github.com/lib/pq v1.10.2 // indirect
github.com/jackc/pgx/v4 v4.16.1 // indirect
github.com/magiconair/properties v1.8.5 // indirect
github.com/mattn/go-colorable v0.1.12 // indirect
github.com/mattn/go-isatty v0.0.14 // indirect
github.com/mattn/go-runewidth v0.0.9 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369 // indirect
github.com/mitchellh/mapstructure v1.4.1 // indirect
github.com/moby/term v0.0.0-20210619224110-3f7ff695adc6 // indirect
github.com/olekukonko/tablewriter v0.0.5 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.0.2 // indirect
github.com/opencontainers/runc v1.1.2 // indirect
github.com/pelletier/go-toml v1.9.3 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
Expand All @@ -105,19 +81,16 @@ require (
github.com/relvacode/iso8601 v1.1.0 // indirect
github.com/rjeczalik/notify v0.9.1 // indirect
github.com/shirou/gopsutil v3.21.4-0.20210419000835-c7a38de76ee5+incompatible // indirect
github.com/sirupsen/logrus v1.8.1 // indirect
github.com/spf13/afero v1.6.0 // indirect
github.com/spf13/cast v1.3.1 // indirect
github.com/spf13/jwalterweatherman v1.1.0 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/spf13/viper v1.8.1 // indirect
github.com/subosito/gotenv v1.2.0 // indirect
github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7 // indirect
github.com/textileio/go-log/v2 v2.1.3-gke-2 // indirect
github.com/tklauser/go-sysconf v0.3.5 // indirect
github.com/tklauser/numcpus v0.2.2 // indirect
github.com/xeipuuv/gojsonpointer v0.0.0-20180127040702-4e3ac2762d5f // indirect
github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 // indirect
github.com/xeipuuv/gojsonschema v1.2.0 // indirect
go.opencensus.io v0.23.0 // indirect
go.opentelemetry.io/otel/sdk v1.7.0 // indirect
go.opentelemetry.io/otel/trace v1.7.0 // indirect
Expand Down