Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GQL API for getStorageAt and getLogs #77

Merged
merged 9 commits into from
Jul 5, 2021
6 changes: 4 additions & 2 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -12,17 +12,19 @@ require (
github.com/ipfs/go-ipld-format v0.2.0
github.com/jmoiron/sqlx v1.2.0
github.com/lib/pq v1.8.0
github.com/machinebox/graphql v0.2.2 // indirect
github.com/multiformats/go-multihash v0.0.14
github.com/onsi/ginkgo v1.15.0
github.com/onsi/gomega v1.10.1
github.com/prometheus/client_golang v1.5.1
github.com/shirou/gopsutil v3.21.5+incompatible // indirect
github.com/sirupsen/logrus v1.7.0
github.com/spf13/cobra v1.1.1
github.com/spf13/viper v1.7.0
github.com/vulcanize/ipld-eth-indexer v0.7.1-alpha
github.com/tklauser/go-sysconf v0.3.6 // indirect
github.com/vulcanize/gap-filler v0.3.1
github.com/vulcanize/ipfs-ethdb v0.0.2-alpha
golang.org/x/sys v0.0.0-20210218155724-8ebf48af031b // indirect
github.com/vulcanize/ipld-eth-indexer v0.7.1-alpha
)

replace github.com/ethereum/go-ethereum v1.9.25 => github.com/vulcanize/go-ethereum v1.9.25-statediff-0.0.15
Expand Down
10 changes: 10 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -697,6 +697,8 @@ github.com/libp2p/go-yamux v1.3.3/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZ
github.com/libp2p/go-yamux v1.3.5/go.mod h1:FGTiPvoV/3DVdgWpX+tM0OW3tsM+W5bSE3gZwqQTcow=
github.com/lucas-clemente/quic-go v0.15.7/go.mod h1:Myi1OyS0FOjL3not4BxT7KN29bRkcMUV5JVVFLKtDp8=
github.com/lunixbochs/vtclean v1.0.0/go.mod h1:pHhQNgMf3btfWnGBVipUOjRYhoOsdGqdm/+2c2E2WMI=
github.com/machinebox/graphql v0.2.2 h1:dWKpJligYKhYKO5A2gvNhkJdQMNZeChZYyBbrZkBZfo=
github.com/machinebox/graphql v0.2.2/go.mod h1:F+kbVMHuwrQ5tYgU9JXlnskM8nOaFxCAEolaQybkjWA=
github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
github.com/magiconair/properties v1.8.1 h1:ZC2Vc7/ZFkGmsVC9KvOjumD+G5lXy2RtTKyzRKO2BQ4=
github.com/magiconair/properties v1.8.1/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
Expand Down Expand Up @@ -908,6 +910,8 @@ github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAm
github.com/shirou/gopsutil v2.20.5-0.20200531151128-663af789c085+incompatible/go.mod h1:5b4v6he4MtMOwMlS0TUMTu2PcXUg8+E1lC7eC3UO/RA=
github.com/shirou/gopsutil v2.20.5+incompatible h1:tYH07UPoQt0OCQdgWWMgYHy3/a9bcxNpBIysykNIP7I=
github.com/shirou/gopsutil v2.20.5+incompatible/go.mod h1:5b4v6he4MtMOwMlS0TUMTu2PcXUg8+E1lC7eC3UO/RA=
github.com/shirou/gopsutil v3.21.5+incompatible h1:OloQyEerMi7JUrXiNzy8wQ5XN+baemxSl12QgIzt0jc=
github.com/shirou/gopsutil v3.21.5+incompatible/go.mod h1:5b4v6he4MtMOwMlS0TUMTu2PcXUg8+E1lC7eC3UO/RA=
github.com/shurcooL/component v0.0.0-20170202220835-f88ec8f54cc4/go.mod h1:XhFIlyj5a1fBNx5aJTbKoIq0mNaPvOagO+HjB3EtxrY=
github.com/shurcooL/events v0.0.0-20181021180414-410e4ca65f48/go.mod h1:5u70Mqkb5O5cxEA8nxTsgrgLehJeAw6Oc4Ab1c/P1HM=
github.com/shurcooL/github_flavored_markdown v0.0.0-20181002035957-2122de532470/go.mod h1:2dOwnU2uBioM+SGy2aZoq1f/Sd1l9OkAeAUvjSyvgU0=
Expand Down Expand Up @@ -994,6 +998,10 @@ github.com/syndtr/goleveldb v1.0.1-0.20200815110645-5c35d600f0ca h1:Ld/zXl5t4+D6
github.com/syndtr/goleveldb v1.0.1-0.20200815110645-5c35d600f0ca/go.mod h1:u2MKkTVTVJWe5D1rCvame8WqhBd88EuIwODJZ1VHCPM=
github.com/tarm/serial v0.0.0-20180830185346-98f6abe2eb07/go.mod h1:kDXzergiv9cbyO7IOYJZWg1U88JhDg3PB6klq9Hg2pA=
github.com/texttheater/golang-levenshtein v0.0.0-20180516184445-d188e65d659e/go.mod h1:XDKHRm5ThF8YJjx001LtgelzsoaEcvnA7lVWz9EeX3g=
github.com/tklauser/go-sysconf v0.3.6 h1:oc1sJWvKkmvIxhDHeKWvZS4f6AW+YcoguSfRF2/Hmo4=
github.com/tklauser/go-sysconf v0.3.6/go.mod h1:MkWzOF4RMCshBAMXuhXJs64Rte09mITnppBXY/rYEFI=
github.com/tklauser/numcpus v0.2.2 h1:oyhllyrScuYI6g+h/zUvNXNp1wy7x8qQy3t/piefldA=
github.com/tklauser/numcpus v0.2.2/go.mod h1:x3qojaO3uyYt0i56EW/VUYs7uBvdl2fkfZFu0T9wgjM=
github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
github.com/tv42/httpunix v0.0.0-20191220191345-2ba4b9c3382c/go.mod h1:hzIxponao9Kjc7aWznkXaL4U4TWaDSs8zcsY4Ka08nM=
github.com/tyler-smith/go-bip39 v1.0.1-0.20181017060643-dbb3b84ba2ef h1:wHSqTBrZW24CsNJDfeh9Ex6Pm0Rcpc7qrgKBiL44vF4=
Expand Down Expand Up @@ -1236,6 +1244,8 @@ golang.org/x/sys v0.0.0-20210218145245-beda7e5e158e h1:f5mksnk+hgXHnImpZoWj64ja9
golang.org/x/sys v0.0.0-20210218145245-beda7e5e158e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210218155724-8ebf48af031b h1:lAZ0/chPUDWwjqosYR0X4M490zQhMsiJ4K3DbA7o+3g=
golang.org/x/sys v0.0.0-20210218155724-8ebf48af031b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210316164454-77fc1eacc6aa h1:ZYxPR6aca/uhfRJyaOAtflSHjJYiktO7QnJC5ut7iY4=
golang.org/x/sys v0.0.0-20210316164454-77fc1eacc6aa/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
Expand Down
5 changes: 3 additions & 2 deletions pkg/eth/backend.go
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ import (
"github.com/ethereum/go-ethereum/rpc"
"github.com/ethereum/go-ethereum/trie"

"github.com/vulcanize/ipfs-ethdb"
ipfsethdb "github.com/vulcanize/ipfs-ethdb"
"github.com/vulcanize/ipld-eth-indexer/pkg/ipfs"
"github.com/vulcanize/ipld-eth-indexer/pkg/postgres"
shared2 "github.com/vulcanize/ipld-eth-indexer/pkg/shared"
Expand Down Expand Up @@ -509,6 +509,7 @@ func (b *Backend) GetLogs(ctx context.Context, hash common.Hash) ([][]*types.Log
if err := rlp.DecodeBytes(rctBytes, &rct); err != nil {
return nil, err
}

logs[i] = rct.Logs
}
return logs, nil
Expand Down Expand Up @@ -766,7 +767,7 @@ func (b *Backend) GetStorageByHash(ctx context.Context, address common.Address,
return nil, err
}

_, storageRlp, err := b.IPLDRetriever.RetrieveStorageAtByAddressAndStorageSlotAndBlockHash(address, key, hash)
_, _, storageRlp, err := b.IPLDRetriever.RetrieveStorageAtByAddressAndStorageSlotAndBlockHash(address, key, hash)
return storageRlp, err
}

Expand Down
18 changes: 9 additions & 9 deletions pkg/eth/ipld_retriever.go
Original file line number Diff line number Diff line change
Expand Up @@ -29,15 +29,15 @@ import (

const (
RetrieveHeadersByHashesPgStr = `SELECT cid, data
FROM eth.header_cids
FROM eth.header_cids
INNER JOIN public.blocks ON (header_cids.mh_key = blocks.key)
WHERE block_hash = ANY($1::VARCHAR(66)[])`
RetrieveHeadersByBlockNumberPgStr = `SELECT cid, data
FROM eth.header_cids
FROM eth.header_cids
INNER JOIN public.blocks ON (header_cids.mh_key = blocks.key)
WHERE block_number = $1`
RetrieveHeaderByHashPgStr = `SELECT cid, data
FROM eth.header_cids
FROM eth.header_cids
INNER JOIN public.blocks ON (header_cids.mh_key = blocks.key)
WHERE block_hash = $1`
RetrieveUnclesByHashesPgStr = `SELECT cid, data
Expand Down Expand Up @@ -429,25 +429,25 @@ func (r *IPLDRetriever) RetrieveAccountByAddressAndBlockNumber(address common.Ad
}

// RetrieveStorageAtByAddressAndStorageSlotAndBlockHash returns the cid and rlp bytes for the storage value corresponding to the provided address, storage slot, and block hash
func (r *IPLDRetriever) RetrieveStorageAtByAddressAndStorageSlotAndBlockHash(address common.Address, key, hash common.Hash) (string, []byte, error) {
func (r *IPLDRetriever) RetrieveStorageAtByAddressAndStorageSlotAndBlockHash(address common.Address, key, hash common.Hash) (string, []byte, []byte, error) {
storageResult := new(nodeInfo)
stateLeafKey := crypto.Keccak256Hash(address.Bytes())
storageHash := crypto.Keccak256Hash(key.Bytes())
if err := r.db.Get(storageResult, RetrieveStorageLeafByAddressHashAndLeafKeyAndBlockHashPgStr, stateLeafKey.Hex(), storageHash.Hex(), hash.Hex()); err != nil {
return "", nil, err
return "", nil, nil, err
}
if storageResult.Removed {
return "", []byte{}, nil
return "", []byte{}, []byte{}, nil
}
var i []interface{}
if err := rlp.DecodeBytes(storageResult.Data, &i); err != nil {
err = fmt.Errorf("error decoding storage leaf node rlp: %s", err.Error())
return "", nil, err
return "", nil, nil, err
}
if len(i) != 2 {
return "", nil, fmt.Errorf("eth IPLDRetriever expected storage leaf node rlp to decode into two elements")
return "", nil, nil, fmt.Errorf("eth IPLDRetriever expected storage leaf node rlp to decode into two elements")
}
return storageResult.CID, i[1].([]byte), nil
return storageResult.CID, storageResult.Data, i[1].([]byte), nil
}

// RetrieveStorageAtByAddressAndStorageKeyAndBlockNumber returns the cid and rlp bytes for the storage value corresponding to the provided address, storage key, and block number
Expand Down
104 changes: 104 additions & 0 deletions pkg/graphql/client.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
package graphql

import (
"context"
"encoding/json"
"fmt"

"github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/common/hexutil"
gqlclient "github.com/machinebox/graphql"
)

type StorageResponse struct {
Cid string `json:"cid"`
Value common.Hash `json:"value"`
IpldBlock hexutil.Bytes `json:"ipldBlock"`
}

type GetStorageAt struct {
Response StorageResponse `json:"getStorageAt"`
}

type LogResponse struct {
Topics []common.Hash `json:"topics"`
Data hexutil.Bytes `json:"data"`
}

type GetLogs struct {
Responses []LogResponse `json:"getLogs"`
}

type Client struct {
client *gqlclient.Client
}

func NewClient(endpoint string) *Client {
client := gqlclient.NewClient(endpoint)
return &Client{client: client}
}

func (c *Client) GetLogs(ctx context.Context, hash common.Hash, address common.Address) ([]LogResponse, error) {
getLogsQuery := fmt.Sprintf(`
query{
getLogs(blockHash: "%s", contract: "%s") {
data
topics
}
}
`, hash.String(), address.String())

req := gqlclient.NewRequest(getLogsQuery)
req.Header.Set("Cache-Control", "no-cache")

var respData map[string]interface{}
err := c.client.Run(ctx, req, &respData)
if err != nil {
return nil, err
}

jsonStr, err := json.Marshal(respData)
if err != nil {
return nil, err
}

var logs GetLogs
err = json.Unmarshal(jsonStr, &logs)
if err != nil {
return nil, err
}
return logs.Responses, nil
}

func (c *Client) GetStorageAt(ctx context.Context, hash common.Hash, address common.Address, slot string) (*StorageResponse, error) {
getLogsQuery := fmt.Sprintf(`
query{
getStorageAt(blockHash: "%s", contract: "%s",slot: "%s") {
cid
value
ipldBlock
}
}
`, hash.String(), address.String(), common.HexToHash(slot))

req := gqlclient.NewRequest(getLogsQuery)
req.Header.Set("Cache-Control", "no-cache")

var respData map[string]interface{}
err := c.client.Run(ctx, req, &respData)
if err != nil {
return nil, err
}

jsonStr, err := json.Marshal(respData)
if err != nil {
return nil, err
}

var storageAt GetStorageAt
err = json.Unmarshal(jsonStr, &storageAt)
if err != nil {
return nil, err
}
return &storageAt.Response, nil
}
93 changes: 93 additions & 0 deletions pkg/graphql/graphql.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ package graphql

import (
"context"
"database/sql"
"errors"
"time"

Expand All @@ -28,6 +29,7 @@ import (
"github.com/ethereum/go-ethereum/core/state"
"github.com/ethereum/go-ethereum/core/types"
"github.com/ethereum/go-ethereum/eth/filters"
"github.com/ethereum/go-ethereum/rlp"
"github.com/ethereum/go-ethereum/rpc"

"github.com/vulcanize/ipld-eth-server/pkg/eth"
Expand Down Expand Up @@ -91,6 +93,8 @@ type Log struct {
backend *eth.Backend
transaction *Transaction
log *types.Log
cid string
ipldBlock []byte
}

func (l *Log) Transaction(ctx context.Context) *Transaction {
Expand All @@ -117,6 +121,14 @@ func (l *Log) Data(ctx context.Context) hexutil.Bytes {
return hexutil.Bytes(l.log.Data)
}

func (l *Log) Cid(ctx context.Context) string {
return l.cid
}

func (l *Log) IpldBlock(ctx context.Context) hexutil.Bytes {
return hexutil.Bytes(l.ipldBlock)
}

// Transaction represents an Ethereum transaction.
// backend and hash are mandatory; all others will be fetched when required.
type Transaction struct {
Expand Down Expand Up @@ -944,3 +956,84 @@ func (r *Resolver) Logs(ctx context.Context, args struct{ Filter FilterCriteria
filter := filters.NewRangeFilter(filters.Backend(r.backend), begin, end, addresses, topics)
return runFilter(ctx, r.backend, filter)
}

// StorageResult represents a storage slot value. All arguments are mandatory.
type StorageResult struct {
value []byte
cid string
ipldBlock []byte
}

func (s *StorageResult) Value(ctx context.Context) common.Hash {
return common.BytesToHash(s.value)
}

func (s *StorageResult) Cid(ctx context.Context) string {
return s.cid
}

func (s *StorageResult) IpldBlock(ctx context.Context) hexutil.Bytes {
return hexutil.Bytes(s.ipldBlock)
}

func (r *Resolver) GetStorageAt(ctx context.Context, args struct {
BlockHash common.Hash
Contract common.Address
Slot common.Hash
}) (*StorageResult, error) {
cid, ipldBlock, rlpValue, err := r.backend.IPLDRetriever.RetrieveStorageAtByAddressAndStorageSlotAndBlockHash(args.Contract, args.Slot, args.BlockHash)

if err != nil {
if err == sql.ErrNoRows {
ret := StorageResult{value: []byte{}, cid: "", ipldBlock: []byte{}}

return &ret, nil
}

return nil, err
}

var value interface{}
err = rlp.DecodeBytes(rlpValue, &value)
if err != nil {
return nil, err
}

ret := StorageResult{value: value.([]byte), cid: cid, ipldBlock: ipldBlock}
return &ret, nil
}

func (r *Resolver) GetLogs(ctx context.Context, args struct {
BlockHash common.Hash
Contract common.Address
}) (*[]*Log, error) {
ret := make([]*Log, 0, 10)

receiptCIDs, receiptsBytes, err := r.backend.IPLDRetriever.RetrieveReceiptsByBlockHash(args.BlockHash)
if err != nil {
return nil, err
}

receipts := make(types.Receipts, len(receiptsBytes))
for index, receiptBytes := range receiptsBytes {
receiptCID := receiptCIDs[index]
receipt := new(types.Receipt)
if err := rlp.DecodeBytes(receiptBytes, receipt); err != nil {
Copy link
Collaborator

@i-norden i-norden Jul 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For compatibility with EIP-2718 enveloped transactions and receipts (EIP-2930 access list and EIP-1559 dynamic fee) we can no longer use RLP encoding/decoding, we instead need to use the MarshalBinary and UnmarshalBinary methods for receipts and transactions.

Related issue and more info here: #78 (comment).

return nil, err
}

receipts[index] = receipt
for _, log := range receipt.Logs {
if log.Address == args.Contract {
ret = append(ret, &Log{
backend: r.backend,
log: log,
cid: receiptCID,
ipldBlock: receiptBytes,
})
}
}
}

return &ret, nil
}