Connection config provider #3403

Ali-Dalal · 2019-08-19T03:50:40Z

As requested in this PR #3364
@elhigu @kibertoad Can you please check? if the changes are ok then I will add unit test and update the rest of the dialects

…ion.js

lib/client.js

lib/dialects/postgres/index.js

lib/connection.js

lib/client.js

kibertoad

See comments.

Update client to leave the responsibility to determine connection settings type to connectionProvider Update postgres dialect aquireConnection function

lib/client.js

kibertoad · 2019-08-26T08:49:58Z

lib/connectionProvider.js

+class ConnectionConfigProvider {
+  constructor(client) {
+    this.client = client;
+    this.resolvedConnectionSettings = null;


Why not resolve settings at this step already? At the very least we can definitely do that if it's a static config.

kibertoad · 2019-08-26T08:50:14Z

lib/connectionProvider.js

+
+  async resolveConnectionConfig() {
+    if (this.resolvedConnectionSettings) return this.resolvedConnectionSettings;
+    if (isFunction(this.client.config.connection)) {


We can do the check once in constructor and set some flag after that.

Can you give more details about this, please?

if we need to manage this in ConnectionConfigProvider constructor, the how-to manipulate the promise as the type of the function is async?

What I mean is this:

In constructor check if it's a function. If it is - set this.isConfigSourceFunction = true and do nothing else. If it is not - set this.isConfigSourceFunction = false and this.resolvedConnectionSettings = cloneDeep <...>

in resolveConnectionConfig check if this.resolvedConnectionSettings is set, if it is, return, if it is not and isConfigSourceFunction, resolve from function, if neither, probably throw an error.

I think this line in ConnectionConfigProvider class does the same, right?

async resolveConnectionConfig() { if (this.resolvedConnectionSettings) return this.resolvedConnectionSettings;

the lines after check for the type of the connection. if it is a function resolve it, other wise, deepclone it.

Partially. What I'm saying is that everything that can be moved from resolveConnectionConfig to constructor should ideally be moved, resolveConnectionConfig should be as simple as possible.
Current implementation is different from what I've described.

kibertoad · 2019-08-26T08:51:18Z

lib/dialects/mssql/index.js

-      const settings = Object.assign({}, this.connectionSettings);
+    const client = this;
+    return new Bluebird(async (resolver, rejecter) => {
+      const settings = await client


Is there a reason why we might want to set connectionConfigProvider each time for a new connection anew and not just set it once on a client when it is created?

in ConnectionConfigProvider class I processed config.connection resolve the connection settings for only one time. So calling

const settings = await client .connectionConfigProvider() .resolveConnectionConfig();

will return the processed object direclty without processing anything anymore.

Will it?

connectionConfigProvider() { return new ConnectionConfigProvider(this); },

any recommendation?

As I've said, you can call connectionConfigProvider once after client is created and configuration is known. And probably it should be renamed to initConnectionConfigProvider to emphasise it mutates the state.

kibertoad · 2019-08-26T08:51:26Z

lib/dialects/mssql/index.js

+      const settings = await client
+        .connectionConfigProvider()
+        .resolveConnectionConfig();
+      // const settings = Object.assign({}, this.connectionSettings);


is this needed?

Yes, it is needed. if the config is passed as an async function, then this will cause a problem without resolving the config.

Also, this line

const settings = Object.assign({}, this.connectionSettings);

is not needed any more as ConnectionConfigProvider is taking the responsibility for this.

I will clarify the need for this PR again in case I miss any clarification.

The main purpose is we need to fetch the config of the DB from an async store (AWS SSM as an example) and this requires passing the config as an async function in order to wait for the config to be loaded from the promise.

Travis 1st stage passes but the other stages failed and I believe it is because of oracledb dialect which fails only in one of the test cases. Can you have a look please and help?

Can you please delete the unneeded commented out code from PR?

re: Oracle

this seems directly related:

1) Oracledb driver tests OracleDb externalAuth externalAuth and connectString should be sent to the getConnection: AssertionError: expected getConnection to have been called exactly once, but it was called 0 times at Context.<anonymous> (test/unit/dialects/oracledb.js:36:25)

I believe you'd need to update some of mocks in oracle tests, as call hierarchy has obviously changed.

re: Oracledb

I believe the issue is because of switching the bludbird callback to async in line 79

const asyncConnection = new Bluebird(async function(resolver, rejecter) { //...res of the code }

where the linter is showing an error at line 105

client.driver.fetchAsString = client.fetchAsString;

error:
possible race condition client.driver.fetchAsString might be reassinged based on an outdated value of client.driver.fetchAsString

need to debug the code but the issue I can't get the tests working on my machine due to insufficient storge issue (mac 128 GB)

any ideas?

I assume this is still relevant?

@kibertoad I've checked the tests. only Oracle dialect fails. and honestly, I have no idea why. can you please help? I will do one more round to try to find the problem

@Ali-Dalal

externalAuth and connectString should be sent to the getConnection

add await in test

- knexInstance.client + await knexInstance.client .acquireRawConnection() .then(function(resolve) {}, function(reject) {});

lib/client.js

kibertoad · 2019-10-14T11:24:54Z

@Ali-Dalal Do you need any help for finishing the PR?

Ali-Dalal · 2019-10-15T01:30:18Z

@kibertoad Yes please.

I need clarification on this point

in resolveConnectionConfig check if this.resolvedConnectionSettings is set, if it is, return, if it is not and isConfigSourceFunction, resolve from function, if neither, probably throw an error.

Do we need to resolve the config if it is a function inside the constructor? remember the type of the function is async

I am a bit confused because how to resolve connectionSettings in client.js since it is async?
in a different word, where is the best place to call
ConnectionConfigProvider.resolveConnectionConfig()
currently, I am calling this line in each dialect

  acquireRawConnection() {
    const client = this;
    return new Bluebird(async function(resolver, rejecter) {
      const connectionSettings = await client
        .connectionConfigProvider()
        .resolveConnectionConfig();

and I agree with you, the dialect is not responsible for processing or taking care of the connection config. But the question, how to place this code inside client.js?

kibertoad · 2019-10-15T07:23:14Z

@Ali-Dalal As I've said before, we can't resolve config from function since it could be async. We could do it if it's a static config, though, but disregard this part, we can refactor it later. Important part at this point is to finalize the implementation :).

kibertoad · 2019-10-15T09:01:54Z

@maximelkin Any chance you could take a look at the Oracle problem to see what is wrong here? I don't have readily available instance nor configured Docker to test easily myself, unfortunately.

kibertoad · 2019-10-15T09:02:49Z

@Ali-Dalal Can you pull changes from master, btw? I wonder if latest fixes in master would resolve the issue.

maximelkin · 2019-10-15T09:24:50Z

lib/dialects/oracledb/index.js

    // If external authentication dont have to worry about username/password and
    // if not need to set the username and password
-    const oracleDbConfig = client.connectionSettings.externalAuth
-      ? { externalAuth: client.connectionSettings.externalAuth }
+    const connectionSettings = await client


do not use async inside new promise callback
it will not catch errors, so you get unhandledRejection

@Ali-Dalal This needs to be addressed

maximelkin · 2019-10-15T11:16:40Z

eslint/eslint#11899
just mark rule require-atomic-updates as ignored, it will be removed shortly due to false-positives

Ali-Dalal · 2019-10-15T11:17:54Z

@kibertoad @maximelkin guys can you help in the last error from eslint?

check travis for more information

106:5 error Possible race condition: client.driver.fetchAsStringmight be reassigned based on an outdated value ofclient.driver.fetchAsString require-atomic-updates

kibertoad · 2019-10-15T11:21:39Z

@Ali-Dalal as @maximelkin has mentioned, "just mark rule require-atomic-updates as ignored, it will be removed shortly due to false-positives*

Ali-Dalal · 2019-10-15T11:23:46Z

@kibertoad @maximelkin done.

can you guide me where is the best place to write a unit test for testing async config provider?

Ali-Dalal · 2019-10-15T11:55:32Z

@kibertoad I will write unit tests under unit/dialects for every one

kibertoad · 2019-10-16T08:24:32Z

lib/dialects/mssql/index.js

+    const client = this;
+    return new Bluebird(async (resolver, rejecter) => {
+      const settings = await client
+        .connectionConfigProvider()


So in this line we are still always initializing a new ConnectionConfigProvider, meaning we will resolve credentials again on each connection.
Can we adjust connectionConfigProvider() method to lazy-instantiate and not reset each time?

When a new knex instance defined, a new client is attached and a new ConnectionConfoProvider as well and once we call resolve connection, for the first time it will execute the function if the connection is a function then it will mutate the state of the client and set the connection to the resolved one.

I tried it on my machine and the connection is resolved only one time and for the 2nd time it loads the config from

async resolveConnectionConfig() { if (this.resolvedConnectionSettings) return this.resolvedConnectionSettings;

https://github.com/knex/knex/pull/3403/files#diff-cbc1ae6887774cf34c8d4bc73dece12fR10

Can we adjust connectionConfigProvider() method to lazy-instantiate and not reset each time?

can you give some examples how I can solve this?

I think you are seeing connection being resolved only once because it is being reused from the connection pool and not recreated on every request - for new connections it is likely that same acquireRawConnection will be called again and you will have to resolve credentials again.

I'll take a swing at the branch sometime this week to see how to improve the situation.

Easy way to check with multiple connections is to create transactions and never commit / rollback them. In that way the old connections cannot be reused from pool.

Guys, sorry for coming in late in the process, but I noticed this PR when I was considering opening a similar issue, with the use case in mind being support of Amazon RDS IAM authentication.

For this use case though, the resolved configuration can't be cached indefinitely, because the password needs to be changed every 15 minutes.

How about adding an optional field to the returned configuration object, with a sync function determining whether or not the cached config can be reused?

So the connection config provider code can look something like this:

const { token, tokenExpiration } = await someCallToGetTheToken(); return { host : 'your_host', user : 'your_database_user', password : token, database : 'myapp_test', expired = () => { return tokenExpiration <= Date.now(); } }

expired is a synchronous function, returning a boolean determining whether the configuration has expired or or not. If it hasn't expired, it can be reused. If not specified, the previously-resolved configuration is always reused.

What do you think?

kibertoad · 2019-10-27T15:15:06Z

Superseded by #3497

Ali-Dalal added 4 commits July 20, 2019 11:52

#3340 Connection settings can be an async function only for PG client

88f69a5

connection config provider

ccbde75

change default value of resolvedConnectionSettings to null in connect…

d2cbf1a

…ion.js

Merge branch 'master' into connection-config-provider

a8aa0af

kibertoad reviewed Aug 23, 2019

View reviewed changes

lib/client.js Outdated Show resolved Hide resolved

kibertoad reviewed Aug 23, 2019

View reviewed changes

lib/dialects/postgres/index.js Outdated Show resolved Hide resolved

kibertoad reviewed Aug 23, 2019

View reviewed changes

lib/connection.js Outdated Show resolved Hide resolved

kibertoad reviewed Aug 23, 2019

View reviewed changes

lib/client.js Outdated Show resolved Hide resolved

kibertoad requested changes Aug 23, 2019

View reviewed changes

Ali-Dalal added 9 commits August 26, 2019 10:31

Rename connection.js to connectionProvider.js

79a2660

Update client to leave the responsibility to determine connection settings type to connectionProvider Update postgres dialect aquireConnection function

fix an issue in ConnectionConfigProvider

7d1b8f5

remove console.log

a2427b4

enable configProvider for sqllite

d219159

enable config provider for mssql

400b829

enable config provider for mysql

cc87967

enable config provider for oracle

4050772

enable config provider for oracledb

ddae371

Merge branch 'master' into connection-config-provider

37c805c

kibertoad reviewed Aug 26, 2019

View reviewed changes

lib/client.js Outdated Show resolved Hide resolved

kibertoad reviewed Aug 26, 2019

View reviewed changes

Ali-Dalal mentioned this pull request Aug 26, 2019

#3340 Connection settings can be an async function only for PG client #3364

Closed

kibertoad reviewed Aug 26, 2019

View reviewed changes

lib/client.js Outdated Show resolved Hide resolved

Ali-Dalal added 2 commits August 26, 2019 21:29

remove comments from client.js

1ecd885

remove comments from mssql/index.js

a4df60d

kibertoad closed this Oct 15, 2019

kibertoad reopened this Oct 15, 2019

maximelkin reviewed Oct 15, 2019

View reviewed changes

Ali-Dalal added 3 commits October 15, 2019 17:28

Merge branch 'master' into connection-config-provider

1c6eff4

add await in oracledb dialects test

5484292

remove unused variables (clonedeeb) from client.js

2b6460d

fix eslint issues

42d7d41

Ali-Dalal added 3 commits October 16, 2019 09:48

add tests

7059d5f

update postgres unit test

2837c15

update postgres unit test

6d0da47

kibertoad reviewed Oct 16, 2019

View reviewed changes

oranoran mentioned this pull request Oct 25, 2019

Connection configuration provider - without changes to drivers #3497

Merged

kibertoad closed this Oct 27, 2019

Ali-Dalal deleted the connection-config-provider branch October 29, 2019 02:23

Connection config provider #3403

Connection config provider #3403

Conversation

Ali-Dalal commented Aug 19, 2019

kibertoad left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maximelkin Oct 15, 2019 • edited

Choose a reason for hiding this comment

kibertoad commented Oct 14, 2019

Ali-Dalal commented Oct 15, 2019 • edited

kibertoad commented Oct 15, 2019

kibertoad commented Oct 15, 2019

kibertoad commented Oct 15, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maximelkin commented Oct 15, 2019

Ali-Dalal commented Oct 15, 2019

kibertoad commented Oct 15, 2019

Ali-Dalal commented Oct 15, 2019

Ali-Dalal commented Oct 15, 2019

kibertoad Oct 16, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oranoran Oct 24, 2019 • edited

Choose a reason for hiding this comment

kibertoad commented Oct 27, 2019

maximelkin Oct 15, 2019 •

edited

Ali-Dalal commented Oct 15, 2019 •

edited

kibertoad Oct 16, 2019 •

edited

oranoran Oct 24, 2019 •

edited