Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[proposal] Support LOAD DATA LOCAL INFILE for MySQL #2179

Closed
Jille opened this issue Mar 30, 2023 · 5 comments · Fixed by #2545
Closed

[proposal] Support LOAD DATA LOCAL INFILE for MySQL #2179

Jille opened this issue Mar 30, 2023 · 5 comments · Fixed by #2545

Comments

@Jille
Copy link
Contributor

Jille commented Mar 30, 2023

https://github.com/go-sql-driver/mysql#load-data-local-infile-support supports LOAD DATA LOCAL INFILE with an io.Reader.

We could implement the :copyfrom query annotation to use LOAD DATA LOCAL INFILE when using go-sql-driver. The generated code would call https://pkg.go.dev/github.com/go-sql-driver/mysql#RegisterReaderHandler with a new reader for this call (and deregister it afterwards).

I propose to keep the same :copyfrom name and semantics, even though that's technically incorrect for MySQL.

I've looked at the three MySQL drivers listed on https://github.com/golang/go/wiki/SQLDrivers and the other two don't seem to support LOAD DATA [LOCAL] INFILE, though they might support it in the future.

We'd probably want to add a setting where people can indicate which MySQL driver they're using so we can generate the code using the relevant package.

@Jille Jille changed the title Support LOAD DATA LOCAL INFILE for MySQL [proposal] Support LOAD DATA LOCAL INFILE for MySQL Apr 4, 2023
@Jille
Copy link
Contributor Author

Jille commented Apr 4, 2023

Some more thoughts:

  • We need to encode the sqlc rows to TSV (or another CSV-like format) that is accepted by LOAD DATA INFILE. I propose to create an external library for that and let the generated code depend on it. (It felt like too much to generate inline.)

  • We need to configure sqlc which mysql driver is used, because database/sql doesn't natively support LOAD DATA INFILE. I think sql_package is the right place to put that. Passing go-sql-driver/mysql would behave mostly the same as passing database/sql, except that it allows :copyfrom support

Jille added a commit to Jille/sqlc that referenced this issue Apr 20, 2023
In preparation of sqlc-dev#2179 for LOAD DATA LOCAL INFILE
Jille added a commit to Jille/sqlc that referenced this issue Apr 20, 2023
This enables the :copyfrom query annotation for people using
go-sql-driver/mysql that transforms it into a LOAD DATA LOCAL INFILE.

issue sqlc-dev#2179
@Jille
Copy link
Contributor Author

Jille commented Apr 20, 2023

I have two PRs ready for this, and a documentation branch I'll send later.

#2220
#2219
Jille/sqlc@loaddata2...loaddata3

Jille added a commit to Jille/sqlc that referenced this issue Apr 21, 2023
In preparation of sqlc-dev#2179 for LOAD DATA LOCAL INFILE
Jille added a commit to Jille/sqlc that referenced this issue Apr 21, 2023
In preparation of sqlc-dev#2179 for LOAD DATA LOCAL INFILE
Jille added a commit to Jille/sqlc that referenced this issue Apr 21, 2023
In preparation of sqlc-dev#2179 for LOAD DATA LOCAL INFILE
Jille added a commit to Jille/sqlc that referenced this issue Apr 21, 2023
In preparation of sqlc-dev#2179 for LOAD DATA LOCAL INFILE
Jille added a commit to Jille/sqlc that referenced this issue Apr 21, 2023
This enables the :copyfrom query annotation for people using
go-sql-driver/mysql that transforms it into a LOAD DATA LOCAL INFILE.

issue sqlc-dev#2179
Jille added a commit to Jille/sqlc that referenced this issue Apr 21, 2023
In preparation of sqlc-dev#2179 for LOAD DATA LOCAL INFILE
Jille added a commit to Jille/sqlc that referenced this issue Apr 21, 2023
This enables the :copyfrom query annotation for people using
go-sql-driver/mysql that transforms it into a LOAD DATA LOCAL INFILE.

issue sqlc-dev#2179
kyleconroy pushed a commit that referenced this issue Apr 21, 2023
In preparation of #2179 for LOAD DATA LOCAL INFILE
Jille added a commit to Jille/sqlc that referenced this issue Apr 25, 2023
This enables the :copyfrom query annotation for people using
go-sql-driver/mysql that transforms it into a LOAD DATA LOCAL INFILE.

issue sqlc-dev#2179
Jille added a commit to Jille/sqlc that referenced this issue Apr 25, 2023
This enables the :copyfrom query annotation for people using
go-sql-driver/mysql that transforms it into a LOAD DATA LOCAL INFILE.

issue sqlc-dev#2179
Jille added a commit to Jille/sqlc that referenced this issue Apr 25, 2023
This enables the :copyfrom query annotation for people using
go-sql-driver/mysql that transforms it into a LOAD DATA LOCAL INFILE.

issue sqlc-dev#2179
@Jille
Copy link
Contributor Author

Jille commented Apr 25, 2023

I need go-sql-driver/mysql to expose the *time.Location to encode correctly.

@SnoozeThis go-sql-driver/mysql#1416 || go-sql-driver/mysql#1419

@SnoozeThis
Copy link

(https://snoozeth.is/NFrLngXswx4) I will either wait until go-sql-driver/mysql#1416 is closed or until go-sql-driver/mysql#1419 is merged and then add a comment.

Jille added a commit to Jille/sqlc that referenced this issue Jun 25, 2023
This enables the :copyfrom query annotation for people using
go-sql-driver/mysql that transforms it into a LOAD DATA LOCAL INFILE.

We don't have a way to get the timezone from the connection, so I've
simply blocked people from using time.Times in their copyfrom.

issue sqlc-dev#2179
@SnoozeThis
Copy link

An error occurred while snoozing: not found

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants