New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SNS client hanging indefinitely sending PublishCommand #6025
Comments
Hi @alesk20, Thanks for reaching out. The behavior is indeed odd. Since the return value from the await call to Without seeing more detailed logs it would be very difficult to diagnose. This could be due to different httphandler defaults with regard to connection management that you might need to change. For example, in the v2 SDK the default timeout was 60 seconds, in v3 we use the defaults provided by node's http client which is 0:
My guess is that this issue where the server hangs is also happening on v2, but the default behavior of the older version makes this more transparent. You might want to dial down the timeout to be more aggressive , perhaps at 60 seconds to align it with v2's behavior and see if this solves your issue. Thanks, |
Hi @RanVaknin, thank you for the response. I'll try setting the timeout explicitly to 60 seconds, but it's still strange that all the messages get published with V2 sdk and instead with V3 sdk they don't get published when sns client hangs. Thanks |
Hi @RanVaknin, I want to add another question after reading your response: in the V2 sdk what happens when the default requestTimeout is reached? An error is thrown or the promise is just resolved? The timeout of 180 seconds I mentioned in my first message was not set on client-sns, but as external timeout to drop the process and retry, so in my actual implementation, after what you said, I think the connection to SNS topic still hangs even if I drop the process. It still doesn't explain why V3 sdk has this slowdowns publishing messages to SNS topic, while the V2 sdk delivers them immediately, also under huge pressure, without missing any delivery. Thanks |
Hi @alesk20 ,
Not necessarily, the server might receive and process your request but it might not be responding with the status to inform the client that the message was / wasn't processed. It's hard to say why you are only experiencing this with v3. It might be because differences in connection management, or something you did differently in your code. Can you set up a minimal github repository that can reliably (intermittently reliably is also ok) reproduce this behavior? Thanks, |
Hi @RanVaknin, unfortunately it's very difficult to replicate this case, it only happens to me after 1-2 hours and only in production environment, where I have a lot of traffic on the sqs queue. I also tried to replicate it on a test environment myself, but couldn't manage to do it. As I said in the first message, I didn't change anything on the code, I just migrate V2 sdk to V3 sdk and upgraded Node.js 16 to Node.js 18, these two are the only things I changed. I don't think the problem is Node.js 18 version. Can you tell me what happens on V2 sdk when default requestTimeout is reached? The promise gets resolved or an error is thrown? Thanks. |
Hi @alesk20 ,
When v2 requestTimeout (or in its v2 name
I understand your concern, however I cannot point to a single point in the SDK and say "this is why your code is not working like it did in v2" There is about 8 years of development between when v2 was first introduced to when v3 was released, the architecture of the two is very different and evolved with the JS language itself and the Ecosystem's best practices. I tried to strip down all of the http configurations used by the v2 SDK and actually have found that the only http option we explicitly override is indeed console.log(sns.config.httpOptions)
// prints: { timeout: 120000 } I don't think it will be helpful for us to keep comparing the two, and instead we should try and focus how to help with your current setup. Are you running your application from something like a Docker container? I'm asking because Docker has decent support for I understand that your current repro code does not raise the reported behavior, but can you please share it anyway? Right now we are doing a lot of theorizing which is not helpful. By you sharing your code we can better visualize the architecture and do a simple visual check of certain things you might be missing to get this to work correctly (this is not to suggest that your code is wrong). If you have the v2 code handy, feel free to share that too. Thanks again for your cooperation. All the best, |
Checkboxes for prior research
Describe the bug
Hello, I have a problem I can't solve with SNS client. I have a server that receive a big amount of messages from an SQS queue (using SQS client), performs some internal operation and then send a notification with a json message body to an SNS topic, using the sdk method "sns.send" and the argument as instance of the class PublishCommand.
After some hour the server is running, depending on the amount of the data flowing through the sqs consumer, the "sns.send" method begin to hang indefinitely and never respond, and the notification is not being published.
I implemented a timeout of 180 seconds to stop the actual execution and retry the publication on the sns topic, and sometimes it works on the 2nd retry, sometimes on the 3rd and so on.
The problem is that as long as other messages are coming through the sqs queue, more and more messages start to have the same problem, until my server is completely blocked and needs to be restarted. After the restart the messages are succesfully elaborated and notifications are correctly published to the topic.
I have this problem only with aws-sdk v3, running aws-sdk v2 I never had this problem and the operations and logic of my server have remained the same. I tried different versions of the @aws-sdk/client-sns, included the last one, and the problem always occurs.
SDK version number
@aws-sdk/client-sns, @aws-sdk/sqs-consumer
Which JavaScript Runtime is this issue in?
Node.js
Details of the browser/Node.js/ReactNative version
Node.js 18
Reproduction Steps
const sns = new SNS({apiVersion: "2010-03-31", endpoint: options.endpointUrl});
const publishCommand = new PublishCommand({
...MessageData,
TopicArn: topic
});
await sns.send(publishCommand);
Observed Behavior
The command "await sns.send(publishCommand)" hangs undefinitely
Expected Behavior
The "sns.send" command should respond immediately or at least after reasonable time.
Possible Solution
No response
Additional Information/Context
No response
The text was updated successfully, but these errors were encountered: