Workpool memory leak on 3.3.1?

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Workpool memory leak on 3.3.1?

gui.balde
Good morning,

I know this was a known issue back with 2.7.1

We've been profiling one of our modules running on an ec2 instance consuming from a rabbit queue.
The message consumer simply queues the delivered object as a task to be consumed by an internal thread, no hassle.
This internal thread takes each item from the internal queue and goes over the wire to the database. This takes time due to net i/o and transactions. This makes our consumer a slow consumer ;)

In this scenario we expected to see messages queueing up on "rabbitmqctl list_queues", but instead, we see none. What we see is the workPool (inside com.rabbitmq.client.impl.ConsumerWorkService) filling up to 780MB of items in the 'pool' HashMap. Which then leads to an OutOfMemoryError when under a heavy load.

We currently have 90 consumers synchronizing that one internal queue I referred to.

Does anyone see any problems here?

Thanks in advance.

Guilherme.
Reply | Threaded
Open this post in threaded view
|

Re: Workpool memory leak on 3.3.1?

Michael Klishin-2
 On 2 June 2014 at 18:46:07, gui.balde ([hidden email]) wrote:

> > In this scenario we expected to see messages queueing up on "rabbitmqctl  
> list_queues", but instead, we see none. What we see is the workPool  
> (inside
> com.rabbitmq.client.impl.ConsumerWorkService) filling  
> up to 780MB of items
> in the 'pool' HashMap. Which then leads to an OutOfMemoryError  
> when under a
> heavy load.
>  
> We currently have 90 consumers synchronizing that one internal  
> queue I
> referred to.

Are you using QueueingConsumer by any chance? Do you use default basicQos value?
--  
MK  

Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Workpool memory leak on 3.3.1?

gui.balde
Hi Michael,

We're using spring-amqp 1.3.4. It uses a inner class InternalConsumer that extends rabbitmq-client's DefaultConsumer (found at spring-amqp's BlockingQueueConsumer). It handles devilery by putting items on a BlockingQueue.

Since we're using AckMode.NONE on spring-amqp, I believe we use the default basicQoS value.

Should I try a different basicQoS?

Thank you very much.

Guilherme.


Reply | Threaded
Open this post in threaded view
|

Re: Workpool memory leak on 3.3.1?

Michael Klishin-2
On 2 June 2014 at 20:09:44, gui.balde ([hidden email]) wrote:

> > We're using spring-amqp 1.3.4. It uses a inner class InternalConsumer  
> that
> extends rabbitmq-client's DefaultConsumer (found at spring-amqp's  
> BlockingQueueConsumer). It handles devilery by putting items  
> on a
> BlockingQueue.
>  
> Since we're using AckMode.NONE on spring-amqp, I believe we  
> use the default
> basicQoS value.
>  
> Should I try a different basicQoS?

By default there is no limit on how many messages consumers can be given.
Plus you use automatic acknowledgements. This means that RabbitMQ will
try to deliver messages to your consumer as fast as possible,
and if the consumer cannot keep up, its internal BlockingQueue will grow
and grow until you run out of heap. 

This is not a memory leak. You need to use manual acknowledgements
and a non-unlimited basic.qos value (equal to roughly how many messages
you expect to be processed at a time).
--  
MK  

Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Workpool memory leak on 3.3.1?

gui.balde
Hi Michael,

I just shot for 'memory leak' cause I saw it was an issue in the past. I've been trying to improve our consumer logic to make it faster/more effective. It was good refactoring, but it can only take us too far before we run out of memory again.

I'll look into the QoS and manual acks. Those were good tips. Should I have any questions, I'll post back.

Thank you very much for your time.

Guilherme.
Reply | Threaded
Open this post in threaded view
|

Re: Workpool memory leak on 3.3.1?

Gary Russell-2
For the most part, with Spring-AMQP, you can avoid having to do manual acks - use AcknowledgeMode.AUTO (default), set the prefetch (basic.qos) and txSize to, say, 1000, and the container will send the ack each time 1000 messages are processed (per thread).



On Mon, Jun 2, 2014 at 12:24 PM, gui.balde <[hidden email]> wrote:
Hi Michael,

I just shot for 'memory leak' cause I saw it was an issue in the past. I've
been trying to improve our consumer logic to make it faster/more effective.
It was good refactoring, but it can only take us too far before we run out
of memory again.

I'll look into the QoS and manual acks. Those were good tips. Should I have
any questions, I'll post back.

Thank you very much for your time.

Guilherme.



--
View this message in context: http://rabbitmq.1065348.n5.nabble.com/Workpool-memory-leak-on-3-3-1-tp35984p35988.html
Sent from the RabbitMQ mailing list archive at Nabble.com.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Workpool memory leak on 3.3.1?

gui.balde
Hi Gary,

I just tried it. Worked like a charm ;)
Now I see the queue filling up and holding those messages for us. So our component is not flooded anymore.

Thanks Gary,

Guilherme.