feat: read only transactions #1252

bokken · 2018-07-14T14:06:52Z

If autocommit is set to false, read only will be set on begin
transaction.
If autocommit is true, it will continue to be managed at session level.
The queries to change session have been cached to avoid re-parsing each
time readonly value changes.

#1228
#848

If autocommit is set to false, read only will be set on begin transaction. If autocommit is true, it will continue to be managed at session level. The queries to change session have been cached to avoid re-parsing each time readonly value changes. pgjdbc#1228 pgjdbc#848

vlsi · 2018-07-14T14:14:28Z

pgjdbc/src/main/java/org/postgresql/core/QueryExecutor.java

+  /**
+   * Flag indicating that when beginning a transaction, it should be read only.
+   */
+  int QUERY_BEGIN_READ_ONLY = 2048;


I would refrain from mentioning BEGIN here.

Isn't QUERY_READ_ONLY or QUERY_READ_ONLY_HINT better?

davecramer · 2018-07-14T14:20:10Z

I still fail to see the point in this. The only use case is with pgpool in transaction mode. I would much rather see us add a startup connection parameter to the backend and use that as a hint to pools to route to a secondary or read only server for the query.

checkstyle and hamcrest test import

bokken · 2018-07-14T15:17:32Z

@davecramer I agree that this is imperfect, but I think it is a move in a better direction.

If autocommit is true, the queries to adjust the session are now cached (directly addressing #1228).

If autocommit is false, read only is managed with the transaction rather than on session.

This is the simplest solution I could think of.
Trying to wrap every statement into a transaction does not seem practical (when to commit/rollback?).
Adding a new command/parameter/hint to backend would certainly work (though it seems the set session to read only could also be used as hint for pools to do the same thing), but that obviously cannot be achieved with driver alone.

I really intended this PR as a starting point for discussion. If the result is we don't really like a simple but limited approach, I am cool with that. I benefit from seeing the code written and tested to think about advantages and drawbacks more concretely.

vlsi · 2018-07-14T15:24:38Z

pgjdbc/src/main/java/org/postgresql/jdbc/PgConnection.java

+      //if autocommit is true, then we change things at the session level
+      //if autocommit is false, read only is managed with the transaction
+      if (autoCommit) {
+        execSQLUpdate(readOnly ? setSessionReadOnly : setSessionNotReadOnly);


This should be hidden by a default=true connection property.
That is by default pgjdbc should not issue SET SESSION CHARACTERISTICS AS TRANSACTION READ ONLY at all.

Just to make sure I understand. You are proposing a new connection property which would, by default, make setting readOnly only used when autoCommit is false?

This addresses issue 1228 by not doing any work when setReadOnly is called.

When autocommit is false transaction will be read only, which gives hint to pools.

Well, it might make sense to have it a tri-state option (default should be like item 2):

always ignore readonly

pass readonly for begin only (autocommit==false)

pass readonly always (like 2, but with set session in autocommit mode)

The idea is to avoid overheads, and still pass hints downstream.

Is your option 3 the behavior in 42.2.0 or the behavior currently in this PR?

Is there value in 4 options?
IGNORE
TRANSACTION
MIXED (currently this PR)
SESSION (current behavior in 42.2.0)

this PR.

I think we can abandon 42.2.0 behavior (==always fuse begin readonly)

And have default behavior be TRANSACTION?

yes. I'm not sure what the naming should be though.

bokken · 2018-07-14T15:53:58Z

Why are the travis ci jobs failing not able to find the hamcrest library? I added it to the pom. Is there something in the job config using different dependencies than what is declared in the pom?

vlsi · 2018-07-14T16:02:37Z

something in the job config using different dependencies than what is declared in the pom?

jre7 and jre6 use this repositories to build: https://github.com/pgjdbc/pgjdbc-jre7/blob/master/pom.xml and https://github.com/pgjdbc/pgjdbc-jre6/blob/master/pom.xml

CI job fetches those repositories AFAIK.

If you want add hamcrest, you'd need to to that in https://github.com/pgjdbc/pgjdbc-parent-poms/blob/master/pgjdbc-core-parent/pom.xml
Unfortunately, that would require to release a new parent-poms version and update it elsewhere.

I'm not sure if test dependencies should be added to rpm packaging or not.
@praiskup , could you please clarify? Should test scope dependencies be reflected in packaging scripts?

add connection property with 3 options to control read only behavior

fix missing property methods on BaseDataSource avoid redundant static modifier

vlsi · 2018-07-14T19:23:58Z

pgjdbc/src/main/java/org/postgresql/jdbc/PgStatement.java

@@ -404,6 +404,8 @@ private void executeInternal(CachedQuery cachedQuery, ParameterList queryParamet

    if (connection.getAutoCommit()) {
      flags |= QueryExecutor.QUERY_SUPPRESS_BEGIN;
+    } else if (connection.isTransactionReadOnly()) {


I would avoid else here as it would mean that both flags are independent.

allowing some future change on the executor side to actually disentangle them?

I hate when flags are dependent.
In my point of view, flags should either be independent or it should be a single flag with known values.

more loosely couple read only hints to backend

bokken · 2018-07-14T19:56:24Z

pgjdbc/src/main/java/org/postgresql/ds/common/BaseDataSource.java

+   * @see PGProperty#READ_ONLY_MODE
+   */
+  public String getReadOnlyMode() {
+    return PGProperty.READ_ONLY_MODE.getSetString(properties);


Should just be 'get(properties)'

codecov-io · 2018-07-14T20:14:09Z

Codecov Report

Merging #1252 into master will increase coverage by 0.01%.
The diff coverage is 70%.

@@             Coverage Diff              @@
##             master    #1252      +/-   ##
============================================
+ Coverage     68.78%   68.79%   +0.01%     
- Complexity     3908     3922      +14     
============================================
  Files           179      179              
  Lines         16420    16455      +35     
  Branches       2674     2681       +7     
============================================
+ Hits          11295    11321      +26     
- Misses         3880     3886       +6     
- Partials       1245     1248       +3

return default read only mode from data source.

avoid case conversion

praiskup · 2018-07-16T06:43:15Z

On Saturday, July 14, 2018 6:02:39 PM CEST Vladimir Sitnikov wrote: If you want add hamcrest, you'd need to to that in https://github.com/pgjdbc/pgjdbc-parent-poms/blob/master/pgjdbc-core-parent/pom.xml Unfortunately, that would require to release a new parent-poms version and update it elsewhere. I'm not sure if test dependencies should be added to [rpm packaging](https://github.com/pgjdbc/pgjdbc/blob/master/packaging/rpm/postgresql-jdbc.spec.tpl#L61-L76) or not. @praiskup , could you please clarify? Should test scope dependencies be reflected in packaging scripts?

We run the testsuite in RPM build, too. So probably yes; I fail to see what are the new dependencies except for "hamcrest"; but hamcrest is in Fedora repos, just add 'BuildRequires: mvn(org.hamcrest:hamcrest-all)'.

vlsi · 2018-07-16T11:53:36Z

@praiskup , we have junit dependency: https://github.com/pgjdbc/pgjdbc-parent-poms/blob/master/pgjdbc-core-parent/pom.xml#L27-L29
I don't find junit in BuildRequires

Do you know how that works?
Copr logs show that tests are run, however it is not clear where junit comes from.

PS. What do you think of Gradle? Is building via Gradle an option?
I think it might help us to use a single repository to manage the code (and eliminate the need for pgjdbc-parent-poms).

praiskup · 2018-07-16T13:56:03Z

On Monday, July 16, 2018 1:53:39 PM CEST Vladimir Sitnikov wrote: I don't find `junit` in `BuildRequires` Do you know how that works?

The 'junit' package is transitive dependancy through 'maven-local'.

PS. What do you think of Gradle? Is building via Gradle an option? I think it might help us to use a single repository to manage the code (and eliminate the need for `pgjdbc-parent-poms`).

I have zero experience with gradle, dunno.

bokken · 2018-07-16T19:25:48Z

I just removed the dependency on hamcrest.

# Conflicts: # pgjdbc/src/main/java/org/postgresql/core/v3/QueryExecutorImpl.java # pgjdbc/src/main/java/org/postgresql/jdbc/PgConnection.java

AppVeyorBot · 2019-04-15T22:59:18Z

✅ Build pgjdbc 1.0.227 completed (commit 13b8371f86 by @bokken)

DanilaShapkin · 2019-11-13T13:50:16Z

Hello!
Could you tell me when are you going to release this task?
There was a problem when working with the pgbouncer "pool_mode=transaction" same as #848

davecramer · 2019-11-13T14:14:46Z

@bokken can you rebase fix the conflict and I'll push this

DanilaShapkin · 2019-11-25T07:12:13Z

Guys @davecramer @bokken , I'll appreciate if you merge this is pr ASAP.

# Conflicts: # pgjdbc/src/main/java/org/postgresql/core/v3/QueryExecutorImpl.java

AppVeyorBot · 2019-11-25T13:53:19Z

✅ Build pgjdbc 1.0.533 completed (commit b4cc976b95 by @)

davecramer · 2019-11-25T14:38:07Z

@bokken seems you have a few check style issues to fix... :)

bokken · 2019-11-25T14:48:56Z

I cannot get checkstyle to run correctly on on windows :(
Let's see if I have made it happy...

AppVeyorBot · 2019-11-25T15:09:25Z

✅ Build pgjdbc 1.0.534 completed (commit 38ca671b14 by @)

AppVeyorBot · 2019-11-25T15:30:39Z

✅ Build pgjdbc 1.0.535 completed (commit a0d2fbda63 by @)

bokken · 2019-11-25T16:30:55Z

@davecramer the checkstyle gods have been sated.

vlsi reviewed Jul 14, 2018

View reviewed changes

feat: read only transactions

72c9f0e

checkstyle and hamcrest test import

vlsi reviewed Jul 14, 2018

View reviewed changes

bokken added 2 commits July 14, 2018 14:12

feat: read only transactions

741b087

add connection property with 3 options to control read only behavior

feat: read only transactions

cdcf357

fix missing property methods on BaseDataSource avoid redundant static modifier

vlsi reviewed Jul 14, 2018

View reviewed changes

feat: read only transactions

3d22199

more loosely couple read only hints to backend

bokken commented Jul 14, 2018

View reviewed changes

bokken added 2 commits July 14, 2018 15:45

feat: read only transactions

27411e7

return default read only mode from data source.

feat: read only transactions

22c48f3

avoid case conversion

Merge remote-tracking branch 'upstream/master' into perf_read_only

2b76735

# Conflicts: # pgjdbc/src/main/java/org/postgresql/core/v3/QueryExecutorImpl.java # pgjdbc/src/main/java/org/postgresql/jdbc/PgConnection.java

Merge remote-tracking branch 'upstream/master' into perf_read_only

bdf22c6

# Conflicts: # pgjdbc/src/main/java/org/postgresql/core/v3/QueryExecutorImpl.java

appease the checkstyle gods

b8be807

appease the checkstyle gods

5c8f6ae

davecramer merged commit 0507979 into pgjdbc:master Nov 25, 2019

dennishendriksen mentioned this pull request Jan 20, 2020

Upgrade spring-boot-starter-parent 2.2.2.RELEASE to 2.2.4.RELEASE molgenis/molgenis#8851

Merged

9 tasks

davecramer mentioned this pull request May 12, 2020

setReadOnly does not work with pgbouncer and transaction pooling mode #848

Open

tabishop mentioned this pull request Aug 26, 2022

update postgreSQL to 42.4.2 OpenLiberty/open-liberty#22292

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: read only transactions #1252

feat: read only transactions #1252

bokken commented Jul 14, 2018

vlsi Jul 14, 2018

davecramer commented Jul 14, 2018

bokken commented Jul 14, 2018

vlsi Jul 14, 2018

bokken Jul 14, 2018

vlsi Jul 14, 2018

bokken Jul 14, 2018

vlsi Jul 14, 2018

bokken Jul 14, 2018

vlsi Jul 14, 2018 •

edited

bokken commented Jul 14, 2018

vlsi commented Jul 14, 2018

vlsi Jul 14, 2018

bokken Jul 14, 2018

vlsi Jul 14, 2018

bokken Jul 14, 2018

codecov-io commented Jul 14, 2018 •

edited

praiskup commented Jul 16, 2018 via email •

edited

vlsi commented Jul 16, 2018

praiskup commented Jul 16, 2018 via email

bokken commented Jul 16, 2018

AppVeyorBot commented Apr 15, 2019

DanilaShapkin commented Nov 13, 2019

davecramer commented Nov 13, 2019

DanilaShapkin commented Nov 25, 2019

AppVeyorBot commented Nov 25, 2019

davecramer commented Nov 25, 2019

bokken commented Nov 25, 2019

AppVeyorBot commented Nov 25, 2019

AppVeyorBot commented Nov 25, 2019

bokken commented Nov 25, 2019

feat: read only transactions #1252

feat: read only transactions #1252

Conversation

bokken commented Jul 14, 2018

Choose a reason for hiding this comment

davecramer commented Jul 14, 2018

bokken commented Jul 14, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vlsi Jul 14, 2018 • edited

Choose a reason for hiding this comment

bokken commented Jul 14, 2018

vlsi commented Jul 14, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Jul 14, 2018 • edited

Codecov Report

praiskup commented Jul 16, 2018 via email • edited

vlsi commented Jul 16, 2018

praiskup commented Jul 16, 2018 via email

bokken commented Jul 16, 2018

AppVeyorBot commented Apr 15, 2019

DanilaShapkin commented Nov 13, 2019

davecramer commented Nov 13, 2019

DanilaShapkin commented Nov 25, 2019

AppVeyorBot commented Nov 25, 2019

davecramer commented Nov 25, 2019

bokken commented Nov 25, 2019

AppVeyorBot commented Nov 25, 2019

AppVeyorBot commented Nov 25, 2019

bokken commented Nov 25, 2019

vlsi Jul 14, 2018 •

edited

codecov-io commented Jul 14, 2018 •

edited

praiskup commented Jul 16, 2018 via email •

edited