Skip to content

Releases: GoogleCloudDataproc/spark-bigquery-connector

0.39.0

21 May 20:30
Compare
Choose a tag to compare
  • PR #1221: Adding support for table snapshots. Thanks @tom-s-powell !
  • PR #1222: Add option to request lz4 compressed ReadRowsResponse
  • PR #1225: Fixing multi-release jar shading
  • PR #1227: Optimizing dynamic partition overwrite for Time partitioned tables
  • BigQuery API has been upgraded to version 2.40.1
  • BigQuery Storage API has been upgraded to version 3.5.1
  • GAX has been upgraded to version 2.48.1
  • gRPC has been upgraded to version 1.64.0

0.36.2

07 May 21:28
Compare
Choose a tag to compare
  • Issue #1215: Support predicate pushdown for DATETIME

0.38.0

02 May 03:43
Compare
Choose a tag to compare
  • PR #1205: Sending Identity token in the read API header
  • Issue #1195: Support map type with complex value
  • Issue #1215: Support predicate pushdown for DATETIME
  • BigQuery API has been upgraded to version 2.39.0
  • BigQuery Storage API has been upgraded to version 3.5.0
  • GAX has been upgraded to version 2.47.0
  • Arrow has been upgraded to version 16.0.0
  • gRPC has been upgraded to version 1.63.0
  • Netty has been upgraded to version 4.1.109.Final

0.37.0

25 Mar 22:44
Compare
Choose a tag to compare
  • ⚠️ Starting version 0.38.0 of the connector, the spark-2.4-bigquery version won't be released as Spark 2.4 is
    well-supported by the spark-bigquery-with-dependencies connectors.
  • PR #1156: Propagate stats for BigLake Managed tables
  • PR #1181: Add caching during protobuf generation
  • PR #1190: Enable connection sharing for atLeastOnce writes
  • Issue #1182: Fix query check logic
  • BigQuery API has been upgraded to version 2.38.1
  • BigQuery Storage API has been upgraded to version 3.3.1
  • GAX has been upgraded to version 2.45.0
  • Arrow has been upgraded to version 15.0.1
  • gRPC has been upgraded to version 1.62.2
  • Netty has been upgraded to version 4.1.107.Final
  • Protocol Buffers has been upgraded to version 3.25.3

0.36.1

01 Feb 05:38
Compare
Choose a tag to compare
  • PR #1176: fix timestamp filter translation issue

0.36.0

25 Jan 23:34
Compare
Choose a tag to compare
  • PR #1155: allow lazy materialization of query on load
  • PR #1163: Added config to set the BigQuery Job timeout
  • PR #1166: Fix filters by adding surrounding parenthesis. Thanks @tom-s-powell !
  • PR #1171: fix read, write issues with Timestamp
  • Issue #1116: BigQuery write fails with MessageSize is too large
  • BigQuery API has been upgraded to version 2.36.0
  • GAX has been upgraded to version 2.40.0
  • gRPC has been upgraded to version 1.61.0
  • Netty has been upgraded to version 4.1.106.Final
  • Protocol Buffers has been upgraded to version 3.25.2

0.35.1

29 Dec 10:34
Compare
Choose a tag to compare
  • PR #1153: allow writing spark string to BQ datetime

0.35.0

20 Dec 16:06
Compare
Choose a tag to compare
  • PR #1115: Added new connector, spark-3.5-bigquery aimed to be used in Spark 3.5. This connector implements new APIs and capabilities provided by the Spark Data Source V2 API.
  • PR #1117: Make read session caching duration configurable
  • PR #1118: Improve read session caching key
  • PR #1122: Set traceId on write
  • PR #1124: Added SparkListenerEvents for Query and Load jobs running on BigQuery
  • PR #1127: Fix job labeling for mixed case Dataproc job names
  • PR #1136: Consider projections for biglake stats
  • PR #1143: Enable async write for default stream
  • BigQuery API has been upgraded to version 2.35.0
  • BigQuery Storage API has been upgraded to version 2.47.0
  • GAX has been upgraded to version 2.38.0
  • gRPC has been upgraded to version 1.60.0
  • Netty has been upgraded to version 4.1.101.Final
  • Protocol Buffers has been upgraded to version 3.25.1

0.34.0

31 Oct 21:26
Compare
Choose a tag to compare
  • PR #1057: Enable async writes for greater throughput
  • PR #1094: CVE-2023-5072: Upgrading the org.json:json dependency
  • PR #1095: CVE-2023-4586: Upgrading the netty dependencies
  • PR #1104: Fixed nested field predicate pushdown
  • PR #1109: Enable read session caching by default for faster Spark planning
  • PR #1111: Enable retry of failed messages
  • Issue #103: Support for Dynamic partition overwrite for time and range partitioned table
  • Issue #1099: Fixing the usage of ExternalAccountCredentials
  • BigQuery API has been upgraded to version 2.33.2
  • BigQuery Storage API has been upgraded to version 2.44.0
  • GAX has been upgraded to version 2.35.0
  • gRPC has been upgraded to version 1.58.0
  • Protocol Buffers has been upgraded to version 3.24.4

0.33.0

17 Oct 23:24
Compare
Choose a tag to compare
  • Added new connector, spark-3.4-bigquery aimed to be used in Spark 3.4 and above. This connector implements new APIs and capabilities provided by the Spark Data Source V2 API.
  • PR #1008: Adding support to expose BigQuery metrics using Spark custom metrics API.
  • PR #1038: Logical plan now shows the BigQuery table of DirectBigQueryRelation. Thanks @idc101 !
  • PR #1058: View names will appear in query plan instead of the materialized table
  • PR #1061: Handle NPE case when reading BQ table with NUMERIC fields. Thanks @hayssams !
  • PR #1069: Support TimestampNTZ datatype in spark 3.4
  • Issue #453: fix comment handling in query
  • Issue #144: allow writing Spark String to BQ TIME type
  • Issue #867: Support writing with RangePartitioning
  • Issue #1046: Add a way to disable map type support
  • Issue #1062: Adding dataproc job ID and UUID labels to BigQuery jobs