bigquery: big.Rat limited to 9 digits

[Thomasdezeeuw]

Client

BigQuery

Environment

N/A

Go Environment

go version go1.16.6 darwin/arm64

GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/thomas/Library/Caches/go-build"
GOENV="/Users/thomas/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/thomas/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/thomas/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/opt/homebrew/Cellar/go/1.16.6/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/opt/homebrew/Cellar/go/1.16.6/libexec/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.16.6"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/lv/qbz3lh7j6gdc43dd1p22bcc40000gn/T/go-build541126118=/tmp/go-build -gno-record-gcc-switches -fno-common"

Code

https://pkg.go.dev/cloud.google.com/go/bigquery#InferSchema documents that big.Rat types are limited to 9 digits after then decimal, this also seems to be the case for parameters to query. However BigQuery supports much more precision (up to 76, https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#numeric-type). It would be nice to able to query using these high precision numbers.

I guess that the parameters are send as strings so you have to make a decision, but could the limit be increased? Alternatively can we send our own string (with our own defined precision) as number?

Also related: golang/go#47911: Support precision of -1 in Rat.FloatString (that will use the required number of digits to represent the number).

Expected behavior

I expected large precision numbers to be support.

Actual behavior

Large numbers are limited to 9 digits after the decimal.

[shollyman]

Some background: The underlying issue here is the big.Rat doesn't report the value's precision or scale, so we have to pick the most appropriate type. And for compatibility sake, we continue to infer the NUMERIC type, as to do otherwise might break existing InferSchema users in surprising ways.

Much like Relax() on a schema changes all required fields to optional, one could do a similar traversal to alter all inferred NUMERIC types to your desired type. Are you using consistent destination BQ type when you infer? Would a method that converts all instances of type A to type B in a schema be sufficient?

[Thomasdezeeuw]

Some background: The underlying issue here is the big.Rat doesn't report the value's precision or scale, so we have to pick the most appropriate type.

Something like golang/go#47911 would help with this I think? Or maybe something like big.Float.MinPrec?

Also see #4703 (comment).

[shollyman]

Alvaro, mind taking a look at this one?

I suspect the resolution here (and in some related issues) is to do the work to enable explicit query parameter construction. There are multiple cases where we have ambiguous type conversions, so the goal would be to allow users to construct a query parameter explicitly, rather than the current behavior which infers everything from the Go type.

A source of potential complexity here is complex types (e.g. struct types), which are completely valid for query parameters. The interesting questions are whether we allow any type inferencing of the leaf types, or whether everything must be explicit etc.

[Thomasdezeeuw]

Thanks @alvarowolfx!