Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

server: support compressed MySQL protocol #36780

Merged
merged 12 commits into from May 12, 2023
Merged

server: support compressed MySQL protocol #36780

merged 12 commits into from May 12, 2023

Conversation

dveeden
Copy link
Contributor

@dveeden dveeden commented Aug 1, 2022 •
edited

What problem does this PR solve?

Issue Number: close #22605

Problem Summary: Support deflate compression in the mysql protocol

This helps with:

  • Performance: Especially data intensive work over low bandwidth connections
  • Cost: This can reduce cost when bandwidth is payed for, e.g. in Cloud environments.

What is changed and how it works?

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Support for the the compressed version of the MySQL protocol was added

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Aug 1, 2022 •
edited by ti-chi-bot bot

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • hawkingrei
  • lance6716

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added release-note do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Aug 1, 2022
@dveeden
Copy link
Contributor Author

dveeden commented Aug 1, 2022

/cc @bb7133 @xhebox

@dveeden
Copy link
Contributor Author

dveeden commented Aug 1, 2022

Tested with:

  • mysql --compress
  • mysqlsh --compress
  • MySQL Connector/Python with mysql.connector.connect(compress=True, ...)

All the above with 8.x versions

@sre-bot
Copy link
Contributor

sre-bot commented Aug 1, 2022 •
edited

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/04b0a5f9bf70136a12e06daf2f35c8b61889b5fe

@ti-chi-bot ti-chi-bot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Aug 6, 2022
@dveeden
Copy link
Contributor Author

dveeden commented Aug 9, 2022

Tested with:

  • MySQL Connector/J 8.0.30 and useCompression=true
  • MySQL Client 8.0.30 and mysql --compression-algorithms=zlib,zstd --zstd-compression-level=9 ...
  • MySQL Shell 8.0.29 and mysqlsh --compression-algorithms=zstd --sql mysql://...
  • MySQL Connector/Python 8.0.21 and mysql.connector.connect(compress=True, ...)

@dveeden
Copy link
Contributor Author

dveeden commented Aug 9, 2022

/run-common-test

@xhebox xhebox marked this pull request as ready for review February 15, 2023 03:28
@ti-chi-bot ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 15, 2023
@xhebox xhebox marked this pull request as draft February 15, 2023 03:28
@ti-chi-bot ti-chi-bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 15, 2023
@lance6716
Copy link
Contributor

/cc @lance6716

@dveeden
Copy link
Contributor Author

dveeden commented Feb 17, 2023 •
edited

With my latest update it now compresses multiple mysql packets into a single compressed packet once Flush() is called.

This works somewhat:

$ mysql -h 127.0.0.1 -u root -P 4000 --ssl-mode=DISABLED --compression-algorithms=zlib 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 405
Server version: 5.7.25-TiDB-v6.7.0-alpha-66-gc67761448-dirty TiDB Server (Apache License 2.0) Community Edition, MySQL 5.7 compatible

Copyright (c) 2000, 2023, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> \s
ERROR 2013 (HY000): Lost connection to MySQL server during query
No connection. Trying to reconnect...
Connection id:    407
Current database: *** NONE ***

--------------
mysql  Ver 8.0.32 for Linux on x86_64 (MySQL Community Server - GPL)

Connection id:		407
Current database:	
Current user:		root@127.0.0.1
SSL:			Not in use
Current pager:		stdout
Using outfile:		''
Using delimiter:	;
Server version:		5.7.25-TiDB-v6.7.0-alpha-66-gc67761448-dirty TiDB Server (Apache License 2.0) Community Edition, MySQL 5.7 compatible
Protocol version:	10
Connection:		127.0.0.1 via TCP/IP
ERROR 2013 (HY000): Lost connection to MySQL server during query
No connection. Trying to reconnect...
Connection id:    409
Current database: *** NONE ***

Server characterset:	utf8mb4
Db     characterset:	utf8mb4
Client characterset:	utf8mb4
Conn.  characterset:	utf8mb4
TCP port:		4000
Protocol:		Compressed
Binary data as:		Hexadecimal
--------------

mysql>

Next up is to figure out where we violate the protocol and fix that.

@dveeden dveeden force-pushed the mysql_proto_compressed branch 2 times, most recently from 1825842 to b61b8fb Compare May 2, 2023 08:35
@dveeden
Copy link
Contributor Author

dveeden commented May 3, 2023 •
edited

This needs some tests, but otherwise it works.

Tested with:

This is with --compression-algorithms=zstd, --compression-algorithms=zlib and --compression-algorithms=zstd,zlib for MySQL Client and MySQL Shell. Other tools only seems to support zlib and not zstd.

The go-sql-driver/mysql doesn't support compression yet: go-sql-driver/mysql#24

Wireshark (master branch/4.1) supports decompressing both zlib and zstd and can be used to check the client and server flags etc.

@ti-chi-bot ti-chi-bot bot added the status/LGT1 Indicates that a PR has LGTM 1. label May 3, 2023
@ti-chi-bot ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 3, 2023
@ti-chi-bot ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels May 4, 2023
lance6716
lance6716 reviewed May 9, 2023
View reviewed changes
server/packetio.go Outdated Show resolved Hide resolved
server/packetio.go
} else if n != (4 + mysql.MaxPayloadLen) {
return errors.Trace(mysql.ErrBadConn)
if p.compressionAlgorithm != mysql.CompressionNone {
if n, err := p.compressedWriter.Write(data[:4+maxPayloadLen]); err != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems we only use Write and Flush for both compressedWriter and bufWriter, maybe we can abstract an interface so there's no if-branch

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dveeden I have no strong opinion about it. You can merge this PR or address it in following commit.

server/packetio.go Show resolved Hide resolved
server/packetio.go Show resolved Hide resolved
@ti-chi-bot ti-chi-bot bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels May 10, 2023
@dveeden
Copy link
Contributor Author

dveeden commented May 10, 2023

/test unit-test

@dveeden
Copy link
Contributor Author

dveeden commented May 12, 2023

/merge

@ti-chi-bot
Copy link

ti-chi-bot bot commented May 12, 2023

This pull request has been accepted and is ready to merge.

Commit hash: fb55f65

@ti-chi-bot ti-chi-bot bot added the status/can-merge Indicates a PR has been approved by a committer. label May 12, 2023
@ti-chi-bot ti-chi-bot bot merged commit 72cf5df into pingcap:master May 12, 2023
8 checks passed
@yahonda yahonda mentioned this pull request Jun 7, 2023
Closed
@djshow832 djshow832 mentioned this pull request Sep 7, 2023
Closed
@dveeden dveeden mentioned this pull request Oct 3, 2023
Merged
13 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lance6716 lance6716 lance6716 approved these changes

@hawkingrei hawkingrei hawkingrei approved these changes

@bb7133 bb7133 Awaiting requested review from bb7133

@xhebox xhebox Awaiting requested review from xhebox

Assignees
No one assigned
Labels
release-note size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Support mysql protocol compression
5 participants
@dveeden @ti-chi-bot @sre-bot @lance6716 @hawkingrei