Rabbit crash (1.7.0)

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Rabbit crash (1.7.0)

jdunck
Hey all,
  We've had a Rabbit 1.7.0 daemon running since January, pretty much
the same app load, no changes in client code which uses Rabbit.

  Today it crashed, complaining about too many open processes.  I've
zipped up the rabbit.log and the erlang crash log.

  Can anyone see an issue?  How can I help troubleshoot the cause?

  Log files here:
  http://radiotime-logs.s3.amazonaws.com/rabbit-crash-2010-04-14.zip

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Rabbit crash (1.7.0)

Matthias Radestock-3
Jeremy,

Jeremy Dunck wrote:
>   We've had a Rabbit 1.7.0 daemon running since January, pretty much
> the same app load, no changes in client code which uses Rabbit.
>
>   Today it crashed, complaining about too many open processes.  I've
> zipped up the rabbit.log and the erlang crash log.

Please accept the rabbit team's condolences on the death of your bunny.
According to the logs it seemed to be in good health and spirit and then
it suddenly collapsed.

There is nothing in the logs that points at the cause for reaching the
process limit. It's possible that there is a very slow leak somewhere,
but I cannot think of any place where that would be, unless your clients
sometimes do something strange like create a queue that gets left
behind. Hence it's probably worth keeping an eye in the number of queues
with
   rabbitmqctl list_queues

To check whether there is a slow leak of processes, periodically run
   erl_call -sname rabbit@localhost -a 'erlang system_info [process_count]'
which returns the current process count. Depending on how you installed
rabbit you may need the -c option to set the Erlang cookie, e.g.
   erl_call -c `cat /var/lib/rabbitmq/.erlang.cookie` -sname ...

The count will fluctuate as rabbit does its work, but if you see a
general upward trend then please let us know.

Also, please upgrade to the latest version of RabbitMQ if it's not too
much hassle. I don't think that will solve the problem, but it will make
investigation easier if the problem reoccurs.

Here's hoping that the next incarnation of your bunny lives longer.


Regards,

Matthias.

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Rabbit crash (1.7.0)

Alvaro Videla-2
Our RabbitMQ server just died in the same way.

We will add a script to get the number of processes and add that to Graphite, maybe in a week we can have a nice graph with those numbers and see what's the behavior is.

Regards,

Alvaro

On Apr 15, 2010, at 6:07 AM, Matthias Radestock wrote:

> Jeremy,
>
> Jeremy Dunck wrote:
>>  We've had a Rabbit 1.7.0 daemon running since January, pretty much
>> the same app load, no changes in client code which uses Rabbit.
>>
>>  Today it crashed, complaining about too many open processes.  I've
>> zipped up the rabbit.log and the erlang crash log.
>
> Please accept the rabbit team's condolences on the death of your bunny.
> According to the logs it seemed to be in good health and spirit and then
> it suddenly collapsed.
>
> There is nothing in the logs that points at the cause for reaching the
> process limit. It's possible that there is a very slow leak somewhere,
> but I cannot think of any place where that would be, unless your clients
> sometimes do something strange like create a queue that gets left
> behind. Hence it's probably worth keeping an eye in the number of queues
> with
>   rabbitmqctl list_queues
>
> To check whether there is a slow leak of processes, periodically run
>   erl_call -sname rabbit@localhost -a 'erlang system_info [process_count]'
> which returns the current process count. Depending on how you installed
> rabbit you may need the -c option to set the Erlang cookie, e.g.
>   erl_call -c `cat /var/lib/rabbitmq/.erlang.cookie` -sname ...
>
> The count will fluctuate as rabbit does its work, but if you see a
> general upward trend then please let us know.
>
> Also, please upgrade to the latest version of RabbitMQ if it's not too
> much hassle. I don't think that will solve the problem, but it will make
> investigation easier if the problem reoccurs.
>
> Here's hoping that the next incarnation of your bunny lives longer.
>
>
> Regards,
>
> Matthias.
>
> _______________________________________________
> rabbitmq-discuss mailing list
> [hidden email]
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Rabbit crash (1.7.0)

Alvaro Videla-2
In reply to this post by Matthias Radestock-3
Well, I found the problem in our situation,

One of our publishers was opening too many channels.

For the library that we are using, if you don't specify the channel_id, then it will open a new one. Our library is a PHP one, I don't know in Jeremy's case.

Regards,

Alvaro

On Apr 15, 2010, at 6:07 AM, Matthias Radestock wrote:

> Jeremy,
>
> Jeremy Dunck wrote:
>>  We've had a Rabbit 1.7.0 daemon running since January, pretty much
>> the same app load, no changes in client code which uses Rabbit.
>>
>>  Today it crashed, complaining about too many open processes.  I've
>> zipped up the rabbit.log and the erlang crash log.
>
> Please accept the rabbit team's condolences on the death of your bunny.
> According to the logs it seemed to be in good health and spirit and then
> it suddenly collapsed.
>
> There is nothing in the logs that points at the cause for reaching the
> process limit. It's possible that there is a very slow leak somewhere,
> but I cannot think of any place where that would be, unless your clients
> sometimes do something strange like create a queue that gets left
> behind. Hence it's probably worth keeping an eye in the number of queues
> with
>   rabbitmqctl list_queues
>
> To check whether there is a slow leak of processes, periodically run
>   erl_call -sname rabbit@localhost -a 'erlang system_info [process_count]'
> which returns the current process count. Depending on how you installed
> rabbit you may need the -c option to set the Erlang cookie, e.g.
>   erl_call -c `cat /var/lib/rabbitmq/.erlang.cookie` -sname ...
>
> The count will fluctuate as rabbit does its work, but if you see a
> general upward trend then please let us know.
>
> Also, please upgrade to the latest version of RabbitMQ if it's not too
> much hassle. I don't think that will solve the problem, but it will make
> investigation easier if the problem reoccurs.
>
> Here's hoping that the next incarnation of your bunny lives longer.
>
>
> Regards,
>
> Matthias.
>
> _______________________________________________
> rabbitmq-discuss mailing list
> [hidden email]
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Rabbit crash (1.7.0)

Matthew Sackman
On Thu, Apr 15, 2010 at 03:12:38PM +0800, Alvaro Videla wrote:
> Well, I found the problem in our situation,
>
> One of our publishers was opening too many channels.
>
> For the library that we are using, if you don't specify the channel_id, then it will open a new one. Our library is a PHP one, I don't know in Jeremy's case.

Yes, you only have 65536 channels max per connection and the client
should reuse them for you automatically if it does assignment of
channels automatically. Both our .Net and Java clients have channel
allocators which cope nicely with holes in the range of assigned
channels and so forth.

Matthew

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss