Memory leak in STOMP connector or federation?

classic Classic list List threaded Threaded
26 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Memory leak in STOMP connector or federation?

Tomas Doran
Hiya

I’m seeing issues with what appears to be gradual memory leaking in RabbitMQ.

We’re currently running 3.2.1 in production, but I’ve also reproduced this issue with 3.3.1.

The memory is all in the ‘binary’ section, which would imply it’s message bodies or similar content. gcore on the rabbitmq process, followed by strings and grep also implies it’s holding onto message bodies that it shouldn’t be.

From the admin console or rabbitmqctl report, I can’t see any cause of the message bodies being kept (i.e. there is no significant amount of queued messages).

I’ve attached the rabbitmqctl report in the hope that someone with more clues than me will be able to read the runes better and point out what may be wrong or give me any hints for how to further investigate.

TIA
Tom



_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

server_report.txt.gz (410K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory leak in STOMP connector or federation?

Michael Klishin-2
 

On May 16, 2014 at 7:26:33 PM, Tomas Doran ([hidden email]) wrote:
> > I’ve attached the rabbitmqctl report in the hope that someone  
> with more clues than me will be able to read the runes better and  
> point out what may be wrong or give me any hints for how to further  
> investigate.

The title says you suspect STOMP plugin or Federation. Are those (and management)
are the only plugins enabled? Do you publish messages over STOMP and use federation? 
--  
MK  

Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory leak in STOMP connector or federation?

Tomas Doran

On May 16, 2014, at 4:29 PM, Michael Klishin <[hidden email]> wrote:

>  
>
> On May 16, 2014 at 7:26:33 PM, Tomas Doran ([hidden email]) wrote:
>>> I’ve attached the rabbitmqctl report in the hope that someone  
>> with more clues than me will be able to read the runes better and  
>> point out what may be wrong or give me any hints for how to further  
>> investigate.
>
> The title says you suspect STOMP plugin or Federation. Are those (and management)
> are the only plugins enabled? Do you publish messages over STOMP and use federation?

Correct, both of these things - I’m using mcollective a lot (this is the vast bulk of the connections and traffic to the message queue).

Messages are published via STOMP to explicit rabbitmq exchanges, and these exchanges are federated to a couple of other brokers in a star distribution.

We seem to be seeing the most/fastest leaking on the machine which gets the most connections (and disconnections) - at the rate of a few a second (see end), however machines with less clients (and less ephemeral clients) are also leaking, just at a reduced rate.


Cheers
Tom

E.g.

=INFO REPORT==== 16-May-2014::08:33:09 ===
closing STOMP connection <0.1169.2> (10.56.6.52:46849 -> 10.56.1.15:61614)

=INFO REPORT==== 16-May-2014::08:33:09 ===
accepting STOMP connection <0.1184.2> (10.56.6.52:46850 -> 10.56.1.15:61614)

=INFO REPORT==== 16-May-2014::08:33:09 ===
closing STOMP connection <0.1184.2> (10.56.6.52:46850 -> 10.56.1.15:61614)

=INFO REPORT==== 16-May-2014::08:33:11 ===
accepting STOMP connection <0.1202.2> (10.56.5.51:51111 -> 10.56.1.15:61614)

=INFO REPORT==== 16-May-2014::08:33:11 ===
accepting STOMP connection <0.1207.2> (10.56.5.51:51112 -> 10.56.1.15:61614)

=INFO REPORT==== 16-May-2014::08:33:11 ===
closing STOMP connection <0.1202.2> (10.56.5.51:51111 -> 10.56.1.15:61614)

=INFO REPORT==== 16-May-2014::08:33:11 ===
closing STOMP connection <0.1207.2> (10.56.5.51:51112 -> 10.56.1.15:61614)

=INFO REPORT==== 16-May-2014::08:33:11 ===
accepting STOMP connection <0.1232.2> (10.56.5.51:51113 -> 10.56.1.15:61614)

=INFO REPORT==== 16-May-2014::08:33:11 ===
accepting STOMP connection <0.1240.2> (10.56.5.51:51114 -> 10.56.1.15:61614)

=INFO REPORT==== 16-May-2014::08:33:11 ===
closing STOMP connection <0.1232.2> (10.56.5.51:51113 -> 10.56.1.15:61614)

=INFO REPORT==== 16-May-2014::08:33:11 ===
closing STOMP connection <0.1240.2> (10.56.5.51:51114 -> 10.56.1.15:61614)
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory leak in STOMP connector or federation?

Simon MacMullen-2
In reply to this post by Tomas Doran
On 16/05/14 16:24, Tomas Doran wrote:
> I’ve attached the rabbitmqctl report in the hope that someone with
> more clues than me will be able to read the runes better and point
> out what may be wrong or give me any hints for how to further
> investigate.

That definitely looks wrong!

Initial suspicion should fall on the management database since your
queues are all empty and there shouldn't be a lot of other state
anywhere. What happens if you invoke:

$ rabbitmqctl eval
'application:stop(rabbitmq_management),application:start(rabbitmq_management).'

?

Cheers, Simon

--
Simon MacMullen
RabbitMQ, Pivotal
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory leak in STOMP connector or federation?

Tomas Doran

On May 16, 2014, at 5:02 PM, Simon MacMullen <[hidden email]> wrote:

> On 16/05/14 16:24, Tomas Doran wrote:
>> I’ve attached the rabbitmqctl report in the hope that someone with
>> more clues than me will be able to read the runes better and point
>> out what may be wrong or give me any hints for how to further
>> investigate.
>
> That definitely looks wrong!
>
> Initial suspicion should fall on the management database since your queues are all empty and there shouldn't be a lot of other state anywhere. What happens if you invoke:
>
> $ rabbitmqctl eval 'application:stop(rabbitmq_management),application:start(rabbitmq_management).’


Exactly nothing.

Testing on an instance that’s currently using 5.9Gb before I restart it:

 - Nothing logged
 - No change in memory use in the management console.

On the off chance, I also tried stopping and starting rabbitmq_federation, and similarly saw no changes.

I then restarted rabbitmq_stomp, and bingo - memory use has dropped to 225Mb (at the penalty of having bounced ~400 clients).

So I assume that this indicates the STOMP adaptor is somehow leaking memory?

Cheers
Tom


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory leak in STOMP connector or federation?

Matthias Radestock-3
On 16/05/14 23:09, Tomas Doran wrote:
> I then restarted rabbitmq_stomp, and bingo - memory use has dropped
> to 225Mb (at the penalty of having bounced ~400 clients).
>
> So I assume that this indicates the STOMP adaptor is somehow leaking
> memory?

Are these connections very long-lived and publish to / consume from lots
of queue destinations? iirc one of the Erlang processes associated with
a stomp connection keeps a set of all queue names it has ever
encountered so it can avoid (re)declaring queues.

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory leak in STOMP connector or federation?

Tomas Doran

On May 16, 2014, at 11:40 PM, Matthias Radestock <[hidden email]> wrote:

> On 16/05/14 23:09, Tomas Doran wrote:
>> I then restarted rabbitmq_stomp, and bingo - memory use has dropped
>> to 225Mb (at the penalty of having bounced ~400 clients).
>>
>> So I assume that this indicates the STOMP adaptor is somehow leaking
>> memory?
>
> Are these connections very long-lived and publish to / consume from lots of queue destinations? iirc one of the Erlang processes associated with a stomp connection keeps a set of all queue names it has ever encountered so it can avoid (re)declaring queues.


The connections are (or at least can be) very long lived. I can/will try resetting all TCP connections to the broker though (tomorrow) and see if that causes the memory use to decrease...

However I’m guessing not, as no re queues - it’s all mcollective, so there is a persistent server process that listens to client messages:

all messages to a ‘broadcast’ queue
messages with a specific routing key to a ‘direct’ queue

and replies to a specific ‘replies’ queue that the client uses it’s name as a routing key on.

Clients are short lived..

Cheers
Tom



_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory leak in STOMP connector or federation?

Matthias Radestock-3
On 16/05/14 23:53, Tomas Doran wrote:

> The connections are (or at least can be) very long lived. I can/will
> try resetting all TCP connections to the broker though (tomorrow) and
> see if that causes the memory use to decrease...
>
> However I’m guessing not, as no re queues - it’s all mcollective, so
> there is a persistent server process that listens to client
> messages:
>
> all messages to a ‘broadcast’ queue messages with a specific routing
> key to a ‘direct’ queue
>
> and replies to a specific ‘replies’ queue that the client uses it’s
> name as a routing key on.
>
> Clients are short lived..

Hmm. Your report shows more than 3000 exclusive, auto-delete queues.
Does that correspond to the number of concurrent client connections?

I see the memory alarm has gone off, so connections are blocked, which
may skew the result somewhat since such connections won't be closable.
It would be useful to get a report from a time when rabbit is getting
close to the memory alarm threshold but hasn't reached it yet.

The reply queues you mention above... Does a new reply queue get created
for every connecting client, and will the server end up publishing
(reply) messages to that queue? If so then you have exactly the scenario
I described, i.e. a stomp connections that over its lifetime publishes
to an increasing number of queues.

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory leak in STOMP connector or federation?

Michael Klishin-2
In reply to this post by Matthias Radestock-3
 On 17 May 2014 at 02:41:16, Matthias Radestock ([hidden email]) wrote:
> > iirc one of the Erlang processes associated with
> a stomp connection keeps a set of all queue names it has ever
> encountered so it can avoid (re)declaring queues

This is correct. There needs to be a way to distinguish one-off
reply queues so that those are always declared (and not tracked
with the rest of the queues).
--  
MK  

Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory leak in STOMP connector or federation?

Michael Klishin-2
In reply to this post by Tomas Doran
 On 17 May 2014 at 02:55:13, Tomas Doran ([hidden email]) wrote:
> > However I’m guessing not, as no re queues - it’s all mcollective,  
> so there is a persistent server process that listens to client  
> messages:
>  
> all messages to a ‘broadcast’ queue
> messages with a specific routing key to a ‘direct’ queue
>  
> and replies to a specific ‘replies’ queue that the client uses  
> it’s name as a routing key on.

Tomas,

We are not mcollective experts. Can you please post a code snippet
(in any language) that demonstrates what exactly mcollective does for request/reply
over STOMP?

This would be a great way for us to actually reproduce the problem.
--  
MK  

Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory leak in STOMP connector or federation?

Tomas Doran

On May 17, 2014, at 5:28 AM, Michael Klishin <[hidden email]> wrote:

> On 17 May 2014 at 02:55:13, Tomas Doran ([hidden email]) wrote:
>>> However I’m guessing not, as no re queues - it’s all mcollective,  
>> so there is a persistent server process that listens to client  
>> messages:
>>
>> all messages to a ‘broadcast’ queue
>> messages with a specific routing key to a ‘direct’ queue
>>
>> and replies to a specific ‘replies’ queue that the client uses  
>> it’s name as a routing key on.
>
> Tomas,
>
> We are not mcollective experts.

Ack - thanks for taking the time to look at this given my non-specific problem description!

> Can you please post a code snippet
> (in any language) that demonstrates what exactly mcollective does for request/reply
> over STOMP?
>
> This would be a great way for us to actually reproduce the problem.


This is not completely trivial to do (a snippet), as I’m not sure if it’s the server to client or client to server messaging which is causing the leak (or something else, like reconnections).

However, the source for the connector I’m using is:

https://github.com/puppetlabs/marionette-collective/blob/master/plugins/mcollective/connector/rabbitmq.rb

Every one of the messaging patterns in there gets used in our setup, except that notably, we are setting the ‘rabbitmq.use_reply_exchange’ setting (https://github.com/puppetlabs/marionette-collective/blob/master/plugins/mcollective/connector/rabbitmq.rb#L335) to true.

I think that the default pattern  is to use temporary queues for server => client responses, which would cause the effect you’re describing - however with rabbitmq.use_reply_exchange = true, my reply paths are /exchange/mcollective_reply/…, which wouldn’t suffer from the problem mentioned in this thread?

I’m trying to work up a Vagrant box which reproduces my setup faithfully enough to replicate the leak I’m seeing - I’ll share my results as soon as I can get there, but whilst I have a very similar setup running, I haven’t (yet) successfully been able to reproduce the leak.

Cheers
Tom






_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory leak in STOMP connector or federation?

Tomas Doran

On May 20, 2014, at 1:32 AM, Tomas Doran <[hidden email]> wrote:
>
> I’m trying to work up a Vagrant box which reproduces my setup faithfully enough to replicate the leak I’m seeing - I’ll share my results as soon as I can get there, but whilst I have a very similar setup running, I haven’t (yet) successfully been able to reproduce the leak.

https://github.com/bobtfish/rabbitmq-leak

This box can be spun up easily and (slowly) demonstrates the leak.

I’m working on optimizing it to be smaller/more specific to try and identify exactly which part of the messaging is causing the issues.

Cheers
Tom

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory leak in STOMP connector or federation?

Michael Klishin-2

On 21 May 2014 at 00:09:48, Tomas Doran ([hidden email]) wrote:
> > https://github.com/bobtfish/rabbitmq-leak
>  
> This box can be spun up easily and (slowly) demonstrates the leak.  
>  
> I’m working on optimizing it to be smaller/more specific to try  
> and identify exactly which part of the messaging is causing the  
> issues.

Thank you, this is very helpful!
--  
MK  

Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory leak in STOMP connector or federation?

Michael Klishin-2
In reply to this post by Tomas Doran
On 21 May 2014 at 00:09:48, Tomas Doran ([hidden email]) wrote:
> > https://github.com/bobtfish/rabbitmq-leak
>  
> This box can be spun up easily and (slowly) demonstrates the leak.  

Started looking into reproducing this issue. What box (OS, arch) do you use? 
--  
MK  

Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory leak in STOMP connector or federation?

Michael Klishin-2
On 21 May 2014 at 14:42:31, Michael Klishin ([hidden email]) wrote:
> > Started looking into reproducing this issue. What box (OS,  
> arch) do you use?

Ubuntu Precise 64 bit worked like a charm.

So far I've been running the VM for 1 hour, with RAM use floating between 54
and 70 or so MB. GC kicks in from time to time as expected, mgmt_db RAM usage
(historically the most common source of leaks) also floats around 4-5 MB
without growing monotonically. All of this at message rates between 200 and 250
msg/s, published over 20-30 temporary STOMP connections.

Will leave the VM running for 6-8 hours and report what I find.  
--  
MK  

Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory leak in STOMP connector or federation?

Tomas Doran

On May 21, 2014, at 12:31 PM, Michael Klishin <[hidden email]> wrote:

> On 21 May 2014 at 14:42:31, Michael Klishin ([hidden email]) wrote:
>>> Started looking into reproducing this issue. What box (OS,  
>> arch) do you use?
>
> Ubuntu Precise 64 bit worked like a charm.

That’s whatI’m running - just the standard vagrant ubuntu precise :)

(My production is lucid, but)

>
> So far I've been running the VM for 1 hour, with RAM use floating between 54
> and 70 or so MB. GC kicks in from time to time as expected, mgmt_db RAM usage
> (historically the most common source of leaks) also floats around 4-5 MB
> without growing monotonically. All of this at message rates between 200 and 250
> msg/s, published over 20-30 temporary STOMP connections.
>
> Will leave the VM running for 6-8 hours and report what I find.  

I’d expect it to be at ~ 100Mb after 6-8 hours. It’s sloooow. :(

I’m seeing _much faster_ leak rates in production (probably due to the fact I have 400+ clients, not 10).

Cheers
Tom


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory leak in STOMP connector or federation?

Michael Klishin-2
On 21 May 2014 at 16:20:29, Tomas Doran ([hidden email]) wrote:
> > I’d expect it to be at ~ 100Mb after 6-8 hours. It’s sloooow. :(  
>  
> I’m seeing _much faster_ leak rates in production (probably  
> due to the fact I have 400+ clients, not 10).

OK, can you tell me how I can increase the # of mcollectives in the VM
to 100 (or however many is realistic to run in a 2-4 GB VM)?
I'm a Chef guy, so unfortunately I know next to nothing about Puppet. Is there
an attribute I can set via Vagrantfile?

Have you tried inspecting memory breakdown via sudo rabbitmqctl status
to see what section grows fastest/largest? 
--  
MK  

Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory leak in STOMP connector or federation?

Tomas Doran

On May 21, 2014, at 1:24 PM, Michael Klishin <[hidden email]> wrote:

> On 21 May 2014 at 16:20:29, Tomas Doran ([hidden email]) wrote:
>>> I’d expect it to be at ~ 100Mb after 6-8 hours. It’s sloooow. :(  
>>
>> I’m seeing _much faster_ leak rates in production (probably  
>> due to the fact I have 400+ clients, not 10).
>
> OK, can you tell me how I can increase the # of mcollectives in the VM
> to 100 (or however many is realistic to run in a 2-4 GB VM)?

https://github.com/bobtfish/rabbitmq-leak/blob/master/manifests/site.pp#L1

That’s the number of servers (listening for requests, and replying)

https://github.com/bobtfish/rabbitmq-leak/blob/master/manifests/site.pp#L2

That’s the number of clients doing a message to all servers (and getting a reply from each) in a while (true) type loop.

> I'm a Chef guy, so unfortunately I know next to nothing about Puppet. Is there
> an attribute I can set via Vagrantfile?

No sweat, sorry for not making that more clear in the README

> Have you tried inspecting memory breakdown via sudo rabbitmqctl status
> to see what section grows fastest/largest?
> —  

Yes - the ‘binary’ section is what appears to be growing (I think I initially attached a server_report.txt from a prod server at the start of this thread where it had grown _massive_?)

Another (from a VM as above, with a more modest leak) is attached:

Initial Memory: 35 MB
Memory after 17 Hours running: 90 MB

Cheers
Tom





_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

server_report.txt (74K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory leak in STOMP connector or federation?

Michael Klishin-2
 On 21 May 2014 at 17:03:37, Tomas Doran ([hidden email]) wrote:
> > Yes - the ‘binary’ section is what appears to be growing (I think  
> I initially attached a server_report.txt from a prod server  
> at the start of this thread where it had grown _massive_?)
>  
> Another (from a VM as above, with a more modest leak) is attached:  
>  
> Initial Memory: 35 MB
> Memory after 17 Hours running: 90 MB

but only ~ 4 MB is used by binaries. Management DB uses ~ 23 MB, message
index: 31 MB.

(message index size depend on how many messages there are enqueued)

I'm collecting memory breakdown by the hour, will plot once I have at least
5-6 data points.
--  
MK  

Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Memory leak in STOMP connector or federation?

Tomas Doran

On May 21, 2014, at 2:19 PM, Michael Klishin <[hidden email]> wrote:

> On 21 May 2014 at 17:03:37, Tomas Doran ([hidden email]) wrote:
>>> Yes - the ‘binary’ section is what appears to be growing (I think  
>> I initially attached a server_report.txt from a prod server  
>> at the start of this thread where it had grown _massive_?)
>>
>> Another (from a VM as above, with a more modest leak) is attached:  
>>
>> Initial Memory: 35 MB
>> Memory after 17 Hours running: 90 MB
>
> but only ~ 4 MB is used by binaries. Management DB uses ~ 23 MB, message
> index: 31 MB.
>
> (message index size depend on how many messages there are enqueued)
>
> I'm collecting memory breakdown by the hour, will plot once I have at least
> 5-6 data points.

Well dammit, I should have looked more closely before certifying this was my production leak.

I’m pretty sure that we’re _also_ leaking memory in this case somehow, but the symptom of the initial issue I was seeing was massive ‘binary’ memory use, but no (or very few) queued messages.

I’ll continue to dig into this further, thanks for all the help :)

Cheers
Tom

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
12
Loading...