Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix insertion of big block #160

Closed
tdroxler opened this issue Mar 7, 2022 · 2 comments · Fixed by #162
Closed

Fix insertion of big block #160

tdroxler opened this issue Mar 7, 2022 · 2 comments · Fixed by #162
Assignees

Comments

@tdroxler
Copy link
Member

tdroxler commented Mar 7, 2022

When we insert a big block, we create too many bind variable for postgres' prepared statement.
The issue is shown in that branch: feature/insert-big-block
just run: app/testOnly org.alephium.explorer.persistence.dao.BlockDaoSpec -- -z "insert big block"

@simerplaha
Copy link
Member

simerplaha commented Mar 8, 2022

Hey @tdroxler, just like how you found, Postgres allows 32767 parameters per query.

This commit fixes this by adding the ability to split a query into 2 or more queries when parameter size grows over 32767 (configurable).

Eg: for queries with one parameter per row INSERT INTO TABLE values (?)

  • if parameter count is 32767 + 1, first query gets build with 32767 parameters and second one gets 1.
    • Maybe it's more efficient to split this in two queries of equal sizes (32767 + 1) / 2. This need benchmarking.
  • if parameter count is 32767 * 7, seven queries get build with 32767 parameters each.

Alternatively a simpler approach would be to revert back to how Slick's typed queries did batch insert i.e. single row per insert.

BEGIN;
INSERT INTO TABLE values (?, ?, ?, ...);
INSERT INTO TABLE values (?, ?, ?, ...);
COMMIT;

I can run benchmarks to see if there is a performance difference. What do you think?

@tdroxler
Copy link
Member Author

tdroxler commented Mar 8, 2022

The error was: Tried to send an out-of-range integer as a 2-byte value and I end up on that issue where we can see that actually Short.MaxValue = 32767

Let's start with your solution and as we discussed, if we feel at some point that we need some optimization we can run the various benchmarks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants