Rabbitmq queues dropped and recreated after a while

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Rabbitmq queues dropped and recreated after a while

srikanth tns
Hi

we are noticing this weird behaviour with rabbitmq from last 3 days . we are using rabbitmq as middleware for mcollective. The number of nodes we have connected to rabbitmq were 7000 last week, this week the number of nodes came up to 8000 , so we started seeing the connections dropping out from 8000 to 50 after a period of time and after 2 hours, the connections are back to 8000. Can we know the reason why the connections shutdown all of sudden and how do they reconnect back ? Also we noticed that till we restart one of the nodes mcollective services , the connections on rabbitmq dont come up.

Thanks for your help,
Srikanth

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Rabbitmq queues dropped and recreated after a while

Michael Klishin-2
On 5 April 2014 at 02:30:04, srikanth tns ([hidden email]) wrote:
> > The number of nodes we have connected to rabbitmq were 7000 last  
> week, this week the number of nodes came up to 8000 , so we started  
> seeing the connections dropping out from 8000 to 50 after a period  
> of time and after 2 hours, the connections are back to 8000. Can  
> we know the reason why the connections shutdown all of sudden  
> and how do they reconnect back ? Also we noticed that till we restart  
> one of the nodes mcollective services , the connections on rabbitmq  
> dont come up.

What’s in RabbitMQ log files? 
--  
MK  

Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Rabbitmq queues dropped and recreated after a while

srikanth tns
This is what we see in the logs..

ending STOMP connection <0.13770.0> from 10.233.243.155:55298

=INFO REPORT==== 6-Apr-2014::22:45:02 ===
ending STOMP connection <0.6415.0> from 10.233.243.156:33153

=INFO REPORT==== 6-Apr-2014::22:45:02 ===
ending STOMP connection <0.10014.0> from 10.233.243.157:36118

=INFO REPORT==== 6-Apr-2014::22:45:02 ===
ending STOMP connection <0.2265.0> from 10.233.243.158:45859

=INFO REPORT==== 6-Apr-2014::22:45:03 ===
ending STOMP connection <0.11893.0> from 10.233.243.159:44710

=INFO REPORT==== 6-Apr-2014::22:45:03 ===
ending STOMP connection <0.6531.0> from 10.233.243.169:60762



On Sun, Apr 6, 2014 at 6:33 AM, Michael Klishin <[hidden email]> wrote:
On 5 April 2014 at 02:30:04, srikanth tns ([hidden email]) wrote:
> > The number of nodes we have connected to rabbitmq were 7000 last
> week, this week the number of nodes came up to 8000 , so we started
> seeing the connections dropping out from 8000 to 50 after a period
> of time and after 2 hours, the connections are back to 8000. Can
> we know the reason why the connections shutdown all of sudden
> and how do they reconnect back ? Also we noticed that till we restart
> one of the nodes mcollective services , the connections on rabbitmq
> dont come up.

What’s in RabbitMQ log files? 
--
MK

Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Rabbitmq queues dropped and recreated after a while

michaelklishin
2014-04-07 3:01 GMT+04:00 srikanth tns <[hidden email]>:
=INFO REPORT==== 6-Apr-2014::22:45:02 ===
ending STOMP connection <0.6415.0> from 10.233.243.156:33153

It means that a client has disconnected or, possibly, an unrecoverable
error was received (but then there should be other messages in the log).

Can there be clients that disconnect at will or intermediaries (e.g. firewalls) that close
TCP connections, say due to inactivity?

Note: this particular error message was removed from RabbitMQ STOMP plugin around Jan 2012.
Consider upgrading.
--
MK

http://github.com/michaelklishin
http://twitter.com/michaelklishin

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Rabbitmq queues dropped and recreated after a while

srikanth tns
we have not seen any error messages on the logs , Do you think the firewall is dropping the connections if they are idle ? If the connections are dropping off , what trigger that connections to be re-established back ? and after how long does it start reconnecting ?


On Sun, Apr 6, 2014 at 11:35 PM, Michael Klishin <[hidden email]> wrote:
2014-04-07 3:01 GMT+04:00 srikanth tns <[hidden email]>:

=INFO REPORT==== 6-Apr-2014::22:45:02 ===
ending STOMP connection <0.6415.0> from 10.233.243.156:33153

It means that a client has disconnected or, possibly, an unrecoverable
error was received (but then there should be other messages in the log).

Can there be clients that disconnect at will or intermediaries (e.g. firewalls) that close
TCP connections, say due to inactivity?

Note: this particular error message was removed from RabbitMQ STOMP plugin around Jan 2012.
Consider upgrading.
--
MK

http://github.com/michaelklishin
http://twitter.com/michaelklishin

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss



_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Rabbitmq queues dropped and recreated after a while

Michael Klishin-2
On 8 April 2014 at 07:33:50, srikanth tns ([hidden email]) wrote:
> > If the connections are dropping off , what trigger that connections  
> to be re-established back ? and after how long does it start reconnecting  
> ?

Clients may choose to reconnect. Your apps may choose to reconnect. 
--  
MK  

Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss