Hanging connections in management interface (v3.0.4)

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Hanging connections in management interface (v3.0.4)

Allan Beaufour
I've for a long time had an issue where queues show hanging connections in the management interface. The queues show a number of perpetual unacked messages and even if I shut down all consumers, these remain. The ip+port for the consumers doesn't match any processes either. Is this a known bug?

I've had this issues for a while. We were running v2.7.1 perviously and was secretly hoping they would disappear in the upgrade :)

Thanks,

Allan

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Hanging connections in management interface (v3.0.4)

Simon MacMullen-2
There is a known issue with connections / channels on a cluster node not
being cleared out of mgmt if they were running when the node shuts down.
That's not fixed in 3.0.4 but is fixed in nightlies / will be in 3.1.0.
Does that sound like what you're seeing?

Cheers, Simon

On 29/04/13 16:23, Allan Beaufour wrote:

> I've for a long time had an issue where queues show hanging connections
> in the management interface. The queues show a number of perpetual
> unacked messages and even if I shut down all consumers, these remain.
> The ip+port for the consumers doesn't match any processes either. Is
> this a known bug?
>
> I've had this issues for a while. We were running v2.7.1 perviously and
> was secretly hoping they would disappear in the upgrade :)
>
> Thanks,
>
> Allan
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> [hidden email]
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>


--
Simon MacMullen
RabbitMQ, VMware
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Hanging connections in management interface (v3.0.4)

Allan Beaufour
No, this is a single rabbitmq node.


On Tue, Apr 30, 2013 at 5:44 AM, Simon MacMullen <[hidden email]> wrote:
There is a known issue with connections / channels on a cluster node not being cleared out of mgmt if they were running when the node shuts down. That's not fixed in 3.0.4 but is fixed in nightlies / will be in 3.1.0. Does that sound like what you're seeing?

Cheers, Simon

On 29/04/13 16:23, Allan Beaufour wrote:
I've for a long time had an issue where queues show hanging connections
in the management interface. The queues show a number of perpetual
unacked messages and even if I shut down all consumers, these remain.
The ip+port for the consumers doesn't match any processes either. Is
this a known bug?

I've had this issues for a while. We were running v2.7.1 perviously and
was secretly hoping they would disappear in the upgrade :)

Thanks,

Allan


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss



--
Simon MacMullen
RabbitMQ, VMware


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Hanging connections in management interface (v3.0.4)

Simon MacMullen-2
OK.

So you say "queues show hanging connections". Do you mean that the
management plugin shows connections under the "Connections" tab, or
consumers under queue details, or something else?

Do you see the same items in rabbitmqctl list_connections (or
list_queues or similar)? If so can you send the output of "rabbitmqctl
report"?

Also, do these items ever go away? Has the memory alarm / disk alarm
gone off at any point?

Cheers, Simon

On 30/04/13 14:17, Allan Beaufour wrote:

> No, this is a single rabbitmq node.
>
>
> On Tue, Apr 30, 2013 at 5:44 AM, Simon MacMullen <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     There is a known issue with connections / channels on a cluster node
>     not being cleared out of mgmt if they were running when the node
>     shuts down. That's not fixed in 3.0.4 but is fixed in nightlies /
>     will be in 3.1.0. Does that sound like what you're seeing?
>
>     Cheers, Simon
>
>     On 29/04/13 16:23, Allan Beaufour wrote:
>
>         I've for a long time had an issue where queues show hanging
>         connections
>         in the management interface. The queues show a number of perpetual
>         unacked messages and even if I shut down all consumers, these
>         remain.
>         The ip+port for the consumers doesn't match any processes either. Is
>         this a known bug?
>
>         I've had this issues for a while. We were running v2.7.1
>         perviously and
>         was secretly hoping they would disappear in the upgrade :)
>
>         Thanks,
>
>         Allan
>
>
>         _________________________________________________
>         rabbitmq-discuss mailing list
>         [hidden email]
>         <mailto:[hidden email]>
>         https://lists.rabbitmq.com/__cgi-bin/mailman/listinfo/__rabbitmq-discuss
>         <https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss>
>
>
>
>     --
>     Simon MacMullen
>     RabbitMQ, VMware
>
>


--
Simon MacMullen
RabbitMQ, VMware
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Hanging connections in management interface (v3.0.4)

Matthias Reik-2
I think we had a similar problem. Clients that died were still in the
list of connections. In our case this was slowing RabbitMQ (v2.6.x ? not
sure) down to a degree that we had to restart RabbitMQ.
We solved the problem by implementing the heartbeat and we haven't seen
any such issues since then.

Cheers
Maze



On 2013-04-30 16:04, Simon MacMullen wrote:

> OK.
>
> So you say "queues show hanging connections". Do you mean that the
> management plugin shows connections under the "Connections" tab, or
> consumers under queue details, or something else?
>
> Do you see the same items in rabbitmqctl list_connections (or
> list_queues or similar)? If so can you send the output of "rabbitmqctl
> report"?
>
> Also, do these items ever go away? Has the memory alarm / disk alarm
> gone off at any point?
>
> Cheers, Simon
>
> On 30/04/13 14:17, Allan Beaufour wrote:
>> No, this is a single rabbitmq node.
>>
>>
>> On Tue, Apr 30, 2013 at 5:44 AM, Simon MacMullen <[hidden email]
>> <mailto:[hidden email]>> wrote:
>>
>>     There is a known issue with connections / channels on a cluster node
>>     not being cleared out of mgmt if they were running when the node
>>     shuts down. That's not fixed in 3.0.4 but is fixed in nightlies /
>>     will be in 3.1.0. Does that sound like what you're seeing?
>>
>>     Cheers, Simon
>>
>>     On 29/04/13 16:23, Allan Beaufour wrote:
>>
>>         I've for a long time had an issue where queues show hanging
>>         connections
>>         in the management interface. The queues show a number of
>> perpetual
>>         unacked messages and even if I shut down all consumers, these
>>         remain.
>>         The ip+port for the consumers doesn't match any processes
>> either. Is
>>         this a known bug?
>>
>>         I've had this issues for a while. We were running v2.7.1
>>         perviously and
>>         was secretly hoping they would disappear in the upgrade :)
>>
>>         Thanks,
>>
>>         Allan
>>
>>
>>         _________________________________________________
>>         rabbitmq-discuss mailing list
>>         [hidden email]
>>         <mailto:[hidden email]>
>> https://lists.rabbitmq.com/__cgi-bin/mailman/listinfo/__rabbitmq-discuss
>> <https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss>
>>
>>
>>
>>     --
>>     Simon MacMullen
>>     RabbitMQ, VMware
>>
>>
>
>

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Hanging connections in management interface (v3.0.4)

Ben Hood
Could this not be due to a TCP half close?

On Tuesday, 30 April 2013 at 20:35, Maze wrote:

I think we had a similar problem. Clients that died were still in the
list of connections. In our case this was slowing RabbitMQ (v2.6.x ? not
sure) down to a degree that we had to restart RabbitMQ.
We solved the problem by implementing the heartbeat and we haven't seen
any such issues since then.

Cheers
Maze



On 2013-04-30 16:04, Simon MacMullen wrote:
OK.

So you say "queues show hanging connections". Do you mean that the
management plugin shows connections under the "Connections" tab, or
consumers under queue details, or something else?

Do you see the same items in rabbitmqctl list_connections (or
list_queues or similar)? If so can you send the output of "rabbitmqctl
report"?

Also, do these items ever go away? Has the memory alarm / disk alarm
gone off at any point?

Cheers, Simon

On 30/04/13 14:17, Allan Beaufour wrote:
No, this is a single rabbitmq node.


On Tue, Apr 30, 2013 at 5:44 AM, Simon MacMullen <[hidden email]
<[hidden email]>> wrote:

There is a known issue with connections / channels on a cluster node
not being cleared out of mgmt if they were running when the node
shuts down. That's not fixed in 3.0.4 but is fixed in nightlies /
will be in 3.1.0. Does that sound like what you're seeing?

Cheers, Simon

On 29/04/13 16:23, Allan Beaufour wrote:

I've for a long time had an issue where queues show hanging
connections
in the management interface. The queues show a number of
perpetual
unacked messages and even if I shut down all consumers, these
remain.
The ip+port for the consumers doesn't match any processes
either. Is
this a known bug?

I've had this issues for a while. We were running v2.7.1
perviously and
was secretly hoping they would disappear in the upgrade :)

Thanks,

Allan


_________________________________________________
rabbitmq-discuss mailing list



--
Simon MacMullen
RabbitMQ, VMware

_______________________________________________
rabbitmq-discuss mailing list


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Hanging connections in management interface (v3.0.4)

Allan Beaufour
In reply to this post by Simon MacMullen-2

On Tue, Apr 30, 2013 at 10:04 AM, Simon MacMullen <[hidden email]> wrote:
So you say "queues show hanging connections". Do you mean that the management plugin shows connections under the "Connections" tab, or consumers under queue details, or something else?

It shows up under the consumers in queue details.
 
Do you see the same items in rabbitmqctl list_connections (or list_queues or similar)?

Yes, it shows up there. I just forcefully closed one of the:
[[
# rabbitmqctl list_channels name connection | grep 44478
client:44478 -> server:5672 (1) <rabbit@rabbitmq02.2.10377.40>
# rabbitmqctl close_connection "<rabbit@rabbitmq02.2.10377.40>" "close hanging connection"
Closing connection "<rabbit@rabbitmq02.2.10377.40>" ...
...done.
]]

This cleared that connection. The really weird thing is that lsof only shows that connection on the server, not on the client. I.e. I could find this on the server:
[[
beam.smp 29152    rabbitmq   19u  IPv6 523865701      0t0  TCP server:5672->client:44478 (ESTABLISHED)
]]
but not on the client side. That it's IPv6 confuses me "a little".

 
If so can you send the output of "rabbitmqctl report"?

I'll send that off list. 

Also, do these items ever go away?

I can't say for sure, as I haven't checked to see if some of them go away, and new ones "replace" them, but the number of hanging connections doesn't ever seem to go down, no.
 
Has the memory alarm / disk alarm gone off at any point?

No.

Allan

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Hanging connections in management interface (v3.0.4)

Simon MacMullen-2
On 01/05/13 17:35, Allan Beaufour wrote:
> I'll send that off list.

Thanks.

> This cleared that connection. The really weird thing is that lsof only
> shows that connection on the server, not on the client. I.e. I could
> find this on the server:

Hmm. That's very suspicious.

So this is a very different scenario to the known issue I initially
suggested. These connections really are there from the point of view of
the server. If lsof is giving contradictory results on client vs server
then that implies the problem is at a TCP level rather than AMQP.

Could there be a firewall or something which is interfering with the way
TCP connections are torn down?

Cheers, Simon

--
Simon MacMullen
RabbitMQ, Pivotal
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Hanging connections in management interface (v3.0.4)

Allan Beaufour

On Fri, May 3, 2013 at 8:10 AM, Simon MacMullen <[hidden email]> wrote:

This cleared that connection. The really weird thing is that lsof only
shows that connection on the server, not on the client. I.e. I could
find this on the server:

Hmm. That's very suspicious.

So this is a very different scenario to the known issue I initially suggested. These connections really are there from the point of view of the server. If lsof is giving contradictory results on client vs server then that implies the problem is at a TCP level rather than AMQP.

Yes, they are indeed there on the server. It makes no sense to me how they can be, while not being on the server. I find it hard to believe myself.
 
Could there be a firewall or something which is interfering with the way TCP connections are torn down?

It's running on Amazon EC2, so who knows. But now that I think of it I should probably take it up with them, maybe that's where the issue really is. But since there's no IPv6 support on Amazon, we should probably just disable it altogether and that might take care of the issue.

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Hanging connections in management interface (v3.0.4)

Nathan Shirlberg
In reply to this post by Allan Beaufour
We are running  RabbitMQ 2.8.7 on Erlang R14B04 and see similar behavior.  Usually around the same time as network glitches.  We use an F5 load balancer that we believe is keeping the connection open even though the client was (sometimes forcefully) severed.  We either Force the connection closed through API, or just let it sit long enough that the connections drop, the conumsers disappear, and the messages get requeued.

On Monday, April 29, 2013 10:23:43 AM UTC-5, Allan Beaufour wrote:
I've for a long time had an issue where queues show hanging connections in the management interface. The queues show a number of perpetual unacked messages and even if I shut down all consumers, these remain. The ip+port for the consumers doesn't match any processes either. Is this a known bug?

I've had this issues for a while. We were running v2.7.1 perviously and was secretly hoping they would disappear in the upgrade :)

Thanks,

Allan

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss