Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RabbitPubSub Queue Connection Closes Unexpectedly with FailedPrecondition Error #3388

Open
omerfruk opened this issue Mar 1, 2024 · 2 comments
Labels
needs info Further discussion or clarification is necessary

Comments

@omerfruk
Copy link

omerfruk commented Mar 1, 2024

I've encountered an issue with the gocloud.dev/pubsub/rabbitpubsub project where the queue connection closes unexpectedly after a certain period. Despite the establishChannel function's logic to reopen a channel if it exists and is closed, the issue persists with the following error message:
error=pubsub (code=FailedPrecondition): Exception (504) Reason: "channel/connection is not open"

version: v1.8.1

I've reviewed the documentation and source code but haven't found a clear resolution to this issue. It seems like the connection/channel should be re-established by the establishChannel function if it detects that it's closed, but this doesn't appear to be happening in my case.

I am seeking guidance on how to properly handle this situation or if there's a potential bug in the establishChannel logic that might not be handling certain conditions correctly.

func (s *QueueService) Connect(uri string, queueNames []string) error {
	var err error
	s.rabbitConn, err = amqp091.Dial(uri) // "amqp://guest:guest@localhost:5672/"
	if err != nil {
		return err
	}
	ch, err := s.rabbitConn.Channel()
	if err != nil {
		return errors.Wrap(err, "rabbitConn.Channel")
	}
	defer ch.Close()

	for _, name := range queueNames {
		err = ch.ExchangeDeclare(name, "fanout", true, false, false, false, nil)
		if err != nil {
			return errors.Wrap(err, "ch.ExchangeDeclare")
		}

		_, err = ch.QueueDeclare(name, true, false, false, false, nil)
		if err != nil {
			return errors.Wrap(err, "ch.QueueDeclare")
		}

		err = ch.QueueBind(name, "", name, false, nil)
		if err != nil {
			return errors.Wrap(err, "ch.QueueBind")
		}
	}
	return err
}

Thank you for your assistance.

@vangent
Copy link
Contributor

vangent commented Mar 1, 2024

Can you clarify what you mean by "the issue persists"?

From looking at the code, it looks like we "refresh" the channel before using it, but there are definitely cases where the channel could close after that check, and you'd get an error from that operation (e.g., Send or Receive). But, retrying should end up calling establishChannel again, which should reset things unless your actual server is down.

Overall, you're likely going to have to debug this a bit on your own to figure out where the error is being generated. E.g., you could use https://github.com/rogpeppe/gohack to add some logging/printing to the existing code to see when establishChannel is being called, when the error is returned, and what happens when you retry.

HTH!

@vangent vangent added the needs info Further discussion or clarification is necessary label Mar 1, 2024
@omerfruk
Copy link
Author

omerfruk commented Mar 3, 2024

This is what I mean by the error persists: My code works on my server, but when the channel is not used for a certain period of time (about 1-2 weeks), when I want to send something to the channel again, I get ,

"error=pubsub (code=FailedPrecondition): Exception (504) Reason: "channel/connection is not open".

restarting the app fixes the error. But when it is not used again, it repeats the same bug.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs info Further discussion or clarification is necessary
Projects
None yet
Development

No branches or pull requests

2 participants