Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sanitized options #504

Merged
merged 14 commits into from Apr 23, 2017
5 changes: 5 additions & 0 deletions CHANGELOG.md
@@ -1,3 +1,8 @@
v.3.0.0-alpha.2
===============

- Eliminated possible memory leak #503

v.3.0.0-alpha.1
===============

Expand Down
72 changes: 37 additions & 35 deletions README.md
Expand Up @@ -78,10 +78,10 @@ Quick Guide
```javascript
var Queue = require('bull');

var videoQueue = Queue('video transcoding', 6379, '127.0.0.1');
var audioQueue = Queue('audio transcoding', 6379, '127.0.0.1');
var imageQueue = Queue('image transcoding', 6379, '127.0.0.1');
var pdfQueue = Queue('pdf transcoding', 6379, '127.0.0.1');
var videoQueue = Queue('video transcoding', {redis: {port: 6379, host: '127.0.0.1'}});
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be clearer to use the redis string version here, eg redis://127.0.0.1:6379, since the documentation lists the redisConnectionString as the second option (albeit optional - but some people might not notice that)

var audioQueue = Queue('audio transcoding', 'redis://127.0.0.1:6379');
var imageQueue = Queue('image transcoding');
var pdfQueue = new Queue('pdf transcoding');
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for the examples above too, for consistency.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. My idea was to show different ways to instantiate in the examples. I could clarify with a comment instead. I like to be able to instantiate without new, but maybe this is not a best practice anymore.


videoQueue.process(function(job, done){

Expand Down Expand Up @@ -247,7 +247,7 @@ var
cluster = require('cluster');

var numWorkers = 8;
var queue = Queue("test concurrent queue", 6379, '127.0.0.1');
var queue = Queue("test concurrent queue");

if(cluster.isMaster){
for (var i = 0; i < numWorkers; i++) {
Expand Down Expand Up @@ -290,19 +290,15 @@ This can be achieved using the "createClient" option in the queue constructor:
subscriber = new redis();

var opts = {
redis: {
opts: {
createClient: function(type){
switch(type){
case 'client':
return client;
case 'subscriber':
return subscriber;
default:
return new redis();
}
createClient: function(type, opts){
switch(type){
case 'client':
return client;
case 'subscriber':
return subscriber;
default:
return new redis(opts);
}
}
}
}
var queueFoo = new Queue('foobar', opts);
Expand Down Expand Up @@ -390,34 +386,40 @@ listened by some other service that stores the results in a database.

### Queue

```ts
Queue(queueName: string, redisPort: number, redisHost: string, redisOpts?: RedisOpts): Queue
```
```ts
Queue(queueName: string, redisConnectionString: string, redisOpts? RedisOpts): Queue
```typescript
Queue(queueName: string, redisConnectionString?: string, opts: QueueOptions): Queue
```

This is the Queue constructor. It creates a new Queue that is persisted in
Redis. Everytime the same queue is instantiated it tries to process all the
old jobs that may exist from a previous unfinished session.

If no connection string or options passed, the queue will use ioredis default connection
settings.

__Arguments__

```javascript
queueName {String} A unique name for this Queue.
redisPort {Number} A port where redis server is running.
redisHost {String} A host specified as IP or domain where redis is running.
redisOptions {Object} Options to pass to the redis client. https://github.com/luin/ioredis/blob/master/API.md#new-redisport-host-options
```typescript
queueName: string, // A unique name for this Queue.
redisConnectionString?: string, // string A connection string containing the redis server host, port and (optional) authentication.
opts?: QueueOptions, // Options to pass to the redis client. https://github.com/luin/ioredis/blob/master/API.md#new-redisport-host-options
```

Alternatively, it's possible to pass a connection string to create a new queue.

__Arguments__

```javascript
queueName {String} A unique name for this Queue.
redisConnectionString {String} A connection string containing the redis server host, port and (optional) authentication.
redisOptions {Object} Options to pass to the redis client. https://github.com/luin/ioredis/blob/master/API.md#new-redisport-host-options
```typescript
interface QueueOptions {
prefix?: string = 'bull',
redis : RedisOpts, // ioredis defaults
createClient?: (type: enum('client', 'subscriber'), redisOpts?: RedisOpts) => redisClient,

// Advanced settings
settings?: QueueSettings {
Copy link
Contributor

@bradvogel bradvogel Apr 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm working on a PR to document these in more detail

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lockDuration?: number = 5000,
Copy link
Contributor

@bradvogel bradvogel Apr 23, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about defaulting the lock to 30 seconds? My thoughts:

  • A lot of people complain about jobs being double processed currently. So there must be a lot of poorly written job processor code out there that stalls the event loop :). Or folks, like me, forget that we're running our code on tiny instances in the cloud where the CPU is so limited that a tiny bit of JS work will max the CPU. A 30sec timeout would give a bit more buffer. At least until we figure out a generic solution like [Feature] Support for running jobs in child processes #488.
  • An expired lock (due to event loop stalling) is quite fatal now that we check the lock prior to moving the job to the completed or failed (previously we would still move it if there wasn't a lock). So if a long running job (let's say 2min) stalls the event loop for even just 5sec, it means that the job is basically tombstoned at that point. It might still continue processing, but another worker would have likely picked it up as a stalled job and double-processed it. Or if it doesn't happen to get picked up as stalled, when it finally completes it still won't be moved to completed because it lost the lock at one time.
  • The tradeoff is that it will take longer for jobs to be considered 'stalled'. So instead waiting max 5sec to find out if a job was stalled, we'd wait 30sec. I think this is generally OK and that most people aren't running jobs that are that time-sensitive. Actual stalled jobs [due to process crashes] should be extremely rare anyways.

What do you think?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess it is ok to have a larger value as default, specially now when it can be configured. I also look forward #488, which for long duration jobs should work much better.

stalledInterval?: number = 5000,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per above comment, I think this should be moved to 30sec also. It doesn't do much good to make it a lot less than the lock time. Also note that since this is run once per process, if it was set to 30sec here, across 6+ worker processes it'd get called about every 5sec anyways.

maxStalledCount?: number = 1, // The maximum number of times a job can be recovered from the 'stalled' state
guardInterval?: number = 5000,
retryProcessDelay?: number = 5000
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should include lockRenewTime?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added this in #507

}
}
```

---------------------------------------
Expand Down