Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: whenCurrentJobsFinished shouldn't initialize bclient #1347

Merged
merged 2 commits into from Jul 9, 2019

Conversation

gabegorelick
Copy link
Contributor

Fixes #1346

// Since connections are lazily initialized, we can't check queue.client
// without initializing a connection. So expose a boolean we can safely
// query.
queue[type + 'Initialized'] = true;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't you store this information in a separate data structure? furthermore, what other queue types would benefit from storing this data?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't you store this information in a separate data structure?

Not sure I follow. Do you mean you want a dedicated object on the queue to save this? Something like

queue._clientStatuses = {}; // in Queue constructor

queue._clientStatuses[type] = 'initialized'; // after initialization

Or are you thinking there's a more complex state machine that should be used?

what other queue types would benefit from storing this data?

Do you mean Redis connection types? There may be instances where client and eclient are also initialized unnecessarily, but I haven't tracked them down yet. That's also blocked by #1344 which means that client is always initialized.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@manast Have you had a chance to look at this?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, sorry. I meant a separate data structure.

@gabegorelick
Copy link
Contributor Author

The failing test appears to be an existing race condition:

it('should not close external connections', () => {
const client = new redis();
const subscriber = new redis();
const opts = {
createClient(type) {
switch (type) {
case 'client':
return client;
case 'subscriber':
return subscriber;
default:
return new redis();
}
}
};
const testQueue = utils.buildQueue('external connections', opts);
return testQueue
.isReady()
.then(() => {
return testQueue.add({ foo: 'bar' });
})
.then(() => {
expect(testQueue.client).to.be.eql(client);
expect(testQueue.eclient).to.be.eql(subscriber);
return testQueue.close();
})
.then(() => {
expect(client.status).to.be.eql('ready');
expect(subscriber.status).to.be.eql('ready');
return Promise.all([client.quit(), subscriber.quit()]);
});
});

isReady only waits for client to be ready, not the subscriber client. subscriber client is initialized in the Queue constructor when we call this.on('error'), so it's a race condition whether it's ready or not by the time the test gets to that assertion. I don't see anything in my PR that would change this.

@coveralls
Copy link

Coverage Status

Coverage increased (+0.08%) to 93.027% when pulling 2f9b580 on gabegorelick:pause-bclient into c0ee6be on OptimalBits:develop.

1 similar comment
@coveralls
Copy link

Coverage Status

Coverage increased (+0.08%) to 93.027% when pulling 2f9b580 on gabegorelick:pause-bclient into c0ee6be on OptimalBits:develop.

@manast manast merged commit 7baec95 into OptimalBits:develop Jul 9, 2019
@mgreenw
Copy link

mgreenw commented Jul 19, 2019

This is awesome, thanks for fixing this @gabegorelick!

@gabegorelick gabegorelick deleted the pause-bclient branch August 13, 2019 15:02
gabegorelick added a commit to gabegorelick/bull that referenced this pull request Nov 8, 2019
This is the same change as OptimalBits#1347 but for `pause(true, true)`.
yosi-ovadia added a commit to ActiveFence/bull that referenced this pull request Mar 1, 2020
* docs: fix typos

* test: fix mocha options

Having `test/test_*` in `mocha.opts` meant that it was not
possible to run an individual test in isolation.

This may have been caused by the recent mocha version upgrade.

* fix: typo in important notes section

* feat(sandbox): emulate job.progress function

* test: ban use of console

Logging is useful in tests to debug failures, but leaving it in
leads to noise.

* chore: lint JS files in root directory

Not a huge deal since these files are trivial, but important if
they ever grow.

* fix: queue.pause(true, true) doesn't pause queue

Fixes OptimalBits#1532

* chore: cleanup whenCurrentJobsFinished

Cleanup and document whenCurrentJobsFinished. And since it's now
official API, add some tests for it.

* chore: add eslint-plugin-mocha

This is useful to ensure something like `.only` isn't accidentally
introduced. It also includes a bunch of other useful rules.

* test: delete test_cluster.js

These tests have been disabled since 9a29725
was merged in 2017 (OptimalBits#472).

* chore: fix line endings

.eslintrc.yml got checked in with CLRF line endings. Every other
file has LF line endings, so let's make it consistent.

* docs: update CHANGELOG

* 3.12.0

* fix(pause): don't initialize bclient if not waiting for jobs to finish

This is the same change as OptimalBits#1347 but for `pause(true, true)`.

* chore: cleanup lockfiles

- Delete package-lock.json. Bull currently has both yarn.lock and
package-lock.json files. When both are present, TravisCI uses yarn,
so in practice npm has not been used by CI. But the presence of a
package-lock.json file meant that contributors were not sure which
lock file, if any, needed to be updated. Yarn also logs a scary
warning if it detects a package-lock.json file.

- Cache yarn dependencies in TravisCI. This should make CI jobs
somewhat faster.

- Update yarn.lock file and ensure that it remains in sync with
package.json changes by using `--frozen-lockfile`.

* fix: catch errors parsing invalid progress data

* docs: update CHANGELOG

* 3.12.1

* fix: whenCurrentJobsFinished should wait for all jobs

See https://github.com/OptimalBits/bull/pull/1542/files#r356810178

* queue.clean should clean job logs as well

OptimalBits#1599

* feat: add "preventParsingData" option of job to prevent data parsing

* VERY SMALL TYPO ADJUSTMENTS

Was reading through docs, thought these changes might be appreciated.

* ci: update changelog

* ci: upgrade dependencies

* 3.13.0

* Generate unique JobId for repeated jobs to prevent duplicated repeated jobs

Co-authored-by: sjoseph7 <17250762+sjoseph7@users.noreply.github.com>
Co-authored-by: Manuel Astudillo <manuel@optimalbits.com>
Co-authored-by: Gabe Gorelick <gabegorelick@gmail.com>
Co-authored-by: Abhishek Soni <abhisheksoni2720@gmail.com>
Co-authored-by: Doug Ayers <4142577+douglascayers@users.noreply.github.com>
Co-authored-by: Stanislav Mamontov <stanislav.mamontov@gmail.com>
Co-authored-by: Tom Grossman <tomg1988@gmail.com>
Co-authored-by: Siarhei Pakhuta <siarhei.pakhuta@mapbox.com>
Co-authored-by: Will Dembinski <wi-ski@users.noreply.github.com>
jtassin pushed a commit to jtassin/bull that referenced this pull request Jul 3, 2020
lavarsicious pushed a commit to lavarsicious/bull that referenced this pull request Feb 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

whenCurrentJobsFinished initializes bclient unnecessarily
4 participants