Releases: GoogleCloudDataproc/spark-bigquery-connector
Releases · GoogleCloudDataproc/spark-bigquery-connector
0.39.0
- PR #1221: Adding support for table snapshots. Thanks @tom-s-powell !
- PR #1222: Add option to request lz4 compressed ReadRowsResponse
- PR #1225: Fixing multi-release jar shading
- PR #1227: Optimizing dynamic partition overwrite for Time partitioned tables
- BigQuery API has been upgraded to version 2.40.1
- BigQuery Storage API has been upgraded to version 3.5.1
- GAX has been upgraded to version 2.48.1
- gRPC has been upgraded to version 1.64.0
0.36.2
0.38.0
- PR #1205: Sending Identity token in the read API header
- Issue #1195: Support map type with complex value
- Issue #1215: Support predicate pushdown for DATETIME
- BigQuery API has been upgraded to version 2.39.0
- BigQuery Storage API has been upgraded to version 3.5.0
- GAX has been upgraded to version 2.47.0
- Arrow has been upgraded to version 16.0.0
- gRPC has been upgraded to version 1.63.0
- Netty has been upgraded to version 4.1.109.Final
0.37.0
⚠️ Starting version 0.38.0 of the connector, thespark-2.4-bigquery
version won't be released as Spark 2.4 is
well-supported by thespark-bigquery-with-dependencies
connectors.- PR #1156: Propagate stats for BigLake Managed tables
- PR #1181: Add caching during protobuf generation
- PR #1190: Enable connection sharing for atLeastOnce writes
- Issue #1182: Fix query check logic
- BigQuery API has been upgraded to version 2.38.1
- BigQuery Storage API has been upgraded to version 3.3.1
- GAX has been upgraded to version 2.45.0
- Arrow has been upgraded to version 15.0.1
- gRPC has been upgraded to version 1.62.2
- Netty has been upgraded to version 4.1.107.Final
- Protocol Buffers has been upgraded to version 3.25.3
0.36.1
0.36.0
- PR #1155: allow lazy materialization of query on load
- PR #1163: Added config to set the BigQuery Job timeout
- PR #1166: Fix filters by adding surrounding parenthesis. Thanks @tom-s-powell !
- PR #1171: fix read, write issues with Timestamp
- Issue #1116: BigQuery write fails with MessageSize is too large
- BigQuery API has been upgraded to version 2.36.0
- GAX has been upgraded to version 2.40.0
- gRPC has been upgraded to version 1.61.0
- Netty has been upgraded to version 4.1.106.Final
- Protocol Buffers has been upgraded to version 3.25.2
0.35.1
0.35.0
- PR #1115: Added new connector,
spark-3.5-bigquery
aimed to be used in Spark 3.5. This connector implements new APIs and capabilities provided by the Spark Data Source V2 API. - PR #1117: Make read session caching duration configurable
- PR #1118: Improve read session caching key
- PR #1122: Set traceId on write
- PR #1124: Added
SparkListenerEvent
s for Query and Load jobs running on BigQuery - PR #1127: Fix job labeling for mixed case Dataproc job names
- PR #1136: Consider projections for biglake stats
- PR #1143: Enable async write for default stream
- BigQuery API has been upgraded to version 2.35.0
- BigQuery Storage API has been upgraded to version 2.47.0
- GAX has been upgraded to version 2.38.0
- gRPC has been upgraded to version 1.60.0
- Netty has been upgraded to version 4.1.101.Final
- Protocol Buffers has been upgraded to version 3.25.1
0.34.0
- PR #1057: Enable async writes for greater throughput
- PR #1094: CVE-2023-5072: Upgrading the org.json:json dependency
- PR #1095: CVE-2023-4586: Upgrading the netty dependencies
- PR #1104: Fixed nested field predicate pushdown
- PR #1109: Enable read session caching by default for faster Spark planning
- PR #1111: Enable retry of failed messages
- Issue #103: Support for Dynamic partition overwrite for time and range partitioned table
- Issue #1099: Fixing the usage of ExternalAccountCredentials
- BigQuery API has been upgraded to version 2.33.2
- BigQuery Storage API has been upgraded to version 2.44.0
- GAX has been upgraded to version 2.35.0
- gRPC has been upgraded to version 1.58.0
- Protocol Buffers has been upgraded to version 3.24.4
0.33.0
- Added new connector,
spark-3.4-bigquery
aimed to be used in Spark 3.4 and above. This connector implements new APIs and capabilities provided by the Spark Data Source V2 API. - PR #1008: Adding support to expose BigQuery metrics using Spark custom metrics API.
- PR #1038: Logical plan now shows the BigQuery table of DirectBigQueryRelation. Thanks @idc101 !
- PR #1058: View names will appear in query plan instead of the materialized table
- PR #1061: Handle NPE case when reading BQ table with NUMERIC fields. Thanks @hayssams !
- PR #1069: Support TimestampNTZ datatype in spark 3.4
- Issue #453: fix comment handling in query
- Issue #144: allow writing Spark String to BQ TIME type
- Issue #867: Support writing with RangePartitioning
- Issue #1046: Add a way to disable map type support
- Issue #1062: Adding dataproc job ID and UUID labels to BigQuery jobs