RabbitMQ 3.0, SSL and rc4

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

RabbitMQ 3.0, SSL and rc4

carlhoerberg
If you specify this chipers: {ciphers,[{rsa,rc4_128,sha},{rsa,aes_128_cbc,sha}]}]}
and a client connects with rsa,rc4_128,sha, RabbitMQ 3.0 memory usage will grow indefinitely until i crashes.
Reply | Threaded
Open this post in threaded view
|

Re: RabbitMQ 3.0, SSL and rc4

Matthias Radestock-3
On 16/03/13 16:37, carlhoerberg wrote:
> If you specify this chipers:
> {ciphers,[{rsa,rc4_128,sha},{rsa,aes_128_cbc,sha}]}]}
> and a client connects with rsa,rc4_128,sha, RabbitMQ 3.0 memory usage will
> grow indefinitely until i crashes.

Works fine for me:

$ ./scripts/rabbitmqctl status
Status of node rabbit@i ...
[{pid,5015},
  {running_applications,[{rabbit,"RabbitMQ","%%VSN%%"},
                         {ssl,"Erlang/OTP SSL application","5.2.1"},
                         {public_key,"Public key infrastructure","0.18"},
                         {crypto,"CRYPTO version 2","2.3"},
                         {mnesia,"MNESIA  CXC 138 12","4.8"},
                         {os_mon,"CPO  CXC 138 46","2.2.11"},
                         {xmerl,"XML parser","1.3.3"},
                         {sasl,"SASL  CXC 138 11","2.3.1"},
                         {stdlib,"ERTS  CXC 138 10","1.19.1"},
                         {kernel,"ERTS  CXC 138 10","2.16.1"}]},
  {os,{unix,linux}},
  {erlang_version,"Erlang R16B (erts-5.10.1) [source] [64-bit] [smp:8:8]
[async-threads:30] [hipe] [kernel-poll:true]\n"},
...]
...done.

$ ./scripts/rabbitmqctl environment
Application environment of node rabbit@i ...
[...
  {ssl_options,[{cacertfile,"certs/testca/cacert.pem"},
                {certfile,"certs/server/cert.pem"},
                {keyfile,"certs/server/key.pem"},
                {verify_code,1},
                {ciphers,[{rsa,rc4_128,sha},{rsa,aes_128_cbc,sha}]}]},
  ...]
...done.

$ ./scripts/rabbitmqctl list_connections name ssl_key_exchange
ssl_cipher ssl_hash
Listing connections ...
127.0.0.1:51467 -> 127.0.0.1:5671 rsa rc4_128 sha
...done.

That connection is from an Erlang client.


So presumably there is something different in your environment. Any idea
what that might be?


Regards,

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: RabbitMQ 3.0, SSL and rc4

carlhoerberg
Actually all ciphers gives the same result, memory just grows and grows. But by rejecting port 5671 it immediately  drops down to normal level.

Ubuntu 12.04, RabbitMQ 3.0.4, Erlang R14B04

[
  {rabbit, [
     {log_levels, [{connection, error}]},
     {tcp_listeners, [{"0.0.0.0", 5672}]},
     {ssl_listeners, [{"0.0.0.0", 5671}]},
     {ssl_options, [{cacertfile,"/etc/rabbitmq/ca.pem"},
                    {certfile,"/etc/rabbitmq/key.pem"}
                   ]}
   ]},
{rabbitmq_management,
  [{listener, [{port, 15672},
               {ip, "0.0.0.0"},
                {ssl,      true}
              ]}
  ]}
].
Reply | Threaded
Open this post in threaded view
|

Re: RabbitMQ 3.0, SSL and rc4

Matthias Radestock-3
On 16/03/13 17:34, carlhoerberg wrote:
> Actually all ciphers gives the same result, memory just grows and grows. But
> by rejecting port 5671 it immediately  drops down to normal level.
>
> Ubuntu 12.04, RabbitMQ 3.0.4, Erlang R14B04

Give R16B a try.

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: RabbitMQ 3.0, SSL and rc4

carlhoerberg
Yes, a non clustered 3.0.1 works fine with ssl, but not this clustered 3.0.4 :( can try with R16B tmrw.



On Sunday 17 March 2013 at 01:40, Matthias Radestock-3 [via RabbitMQ] wrote:

> On 16/03/13 17:34, carlhoerberg wrote:
> > Actually all ciphers gives the same result, memory just grows and grows. But
> > by rejecting port 5671 it immediately drops down to normal level.
> >
> > Ubuntu 12.04, RabbitMQ 3.0.4, Erlang R14B04
>
> Give R16B a try.
>
> Matthias.
> _______________________________________________
> rabbitmq-discuss mailing list
> [hidden email] (/user/SendEmail.jtp?type=node&node=25486&i=0)
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
> If you reply to this email, your message will be added to the discussion below: http://rabbitmq.1065348.n5.nabble.com/RabbitMQ-3-0-SSL-and-rc4-tp25483p25486.html 
> To unsubscribe from RabbitMQ 3.0, SSL and rc4, click here (
> NAML (
http://rabbitmq.1065348.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml)



Reply | Threaded
Open this post in threaded view
|

Re: RabbitMQ 3.0, SSL and rc4

Matthias Radestock-3
On 16/03/13 17:48, carlhoerberg wrote:
> Yes, a non clustered 3.0.1 works fine with ssl, but not this clustered
> 3.0.4 :( can try with R16B tmrw.

Can't see why clustering would make a difference, unless you are running
distributed Erlang over SSL, which presumably you aren't.

FWIW, I have a clustered 3.0.4 running here, on R14B04 on Ubuntu 12.04,
and SSL connections work just fine.

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: RabbitMQ 3.0, SSL and rc4

carlhoerberg
You're right, it's a user, which happened to use SSL, that's is doing something that makes the server use a lot of memory, within minutes it makes the memory consumption go up to over 7GB. As soon as that connection is closed the memory falls back to normal levels, ~200mb. Will try to figure out what that user is doing..



On Sunday 17 March 2013 at 02:30, Matthias Radestock-3 [via RabbitMQ] wrote:

> On 16/03/13 17:48, carlhoerberg wrote:
> > Yes, a non clustered 3.0.1 works fine with ssl, but not this clustered
> > 3.0.4 :( can try with R16B tmrw.
>
>
> Can't see why clustering would make a difference, unless you are running
> distributed Erlang over SSL, which presumably you aren't.
>
> FWIW, I have a clustered 3.0.4 running here, on R14B04 on Ubuntu 12.04,
> and SSL connections work just fine.
>
> Matthias.
> _______________________________________________
> rabbitmq-discuss mailing list
> [hidden email] (/user/SendEmail.jtp?type=node&node=25488&i=0)
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
> If you reply to this email, your message will be added to the discussion below: http://rabbitmq.1065348.n5.nabble.com/RabbitMQ-3-0-SSL-and-rc4-tp25483p25488.html 
> To unsubscribe from RabbitMQ 3.0, SSL and rc4, click here (
> NAML (
http://rabbitmq.1065348.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml)



Reply | Threaded
Open this post in threaded view
|

Re: RabbitMQ 3.0, SSL and rc4

carlhoerberg
In reply to this post by Matthias Radestock-3
The client in question is using Langohr 1.0.0-beta10-SNAPSHOT.

The users says he's just consuming, but the mgmt interface shows that's he's both sending and receiving traffic in ~20kb/s, but only receiving 0.5msg/sec and not delivering any messages. When the connection if force closed this is returned:

{exit, {channel_termination_timeout, {gen_server,call, [<0.27132.1>,{shutdown,"Closed via management plugin"},infinity]}}, [{gen_server,call,3}, {rabbit_mgmt_wm_connection,delete_resource,2}, {webmachine_resource,resource_call,3}, {webmachine_resource,do,3}, {webmachine_decision_core,resource_call,1}, {webmachine_decision_core,decision,1}, {webmachine_decision_core,handle_request,2}, {rabbit_webmachine,'-makeloop/1-fun-0-',2}]}

We hadn't this problems with 2.8.7, but maybe it's a user error. I've encouraged the user to upgrade to the latest langohr.  


On Sunday 17 March 2013 at 07:47, Carl Hörberg wrote:

> You're right, it's a user, which happened to use SSL, that's is doing something that makes the server use a lot of memory, within minutes it makes the memory consumption go up to over 7GB. As soon as that connection is closed the memory falls back to normal levels, ~200mb. Will try to figure out what that user is doing..  
>  
>  
>  
> On Sunday 17 March 2013 at 02:30, Matthias Radestock-3 [via RabbitMQ] wrote:
>  
> > On 16/03/13 17:48, carlhoerberg wrote:  
> > > Yes, a non clustered 3.0.1 works fine with ssl, but not this clustered  
> > > 3.0.4 :( can try with R16B tmrw.  
> >  
> >  
> >  
> >  
> > Can't see why clustering would make a difference, unless you are running  
> > distributed Erlang over SSL, which presumably you aren't.  
> >  
> > FWIW, I have a clustered 3.0.4 running here, on R14B04 on Ubuntu 12.04,  
> > and SSL connections work just fine.  
> >  
> > Matthias.  
> > _______________________________________________  
> > rabbitmq-discuss mailing list  
> > [hidden email] (/user/SendEmail.jtp?type=node&node=25488&i=0)  
> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> >  
> >  
> > If you reply to this email, your message will be added to the discussion below: http://rabbitmq.1065348.n5.nabble.com/RabbitMQ-3-0-SSL-and-rc4-tp25483p25488.html 
> > To unsubscribe from RabbitMQ 3.0, SSL and rc4, click here (
> > NAML (
http://rabbitmq.1065348.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml)  
>  



Reply | Threaded
Open this post in threaded view
|

Re: RabbitMQ 3.0, SSL and rc4

Matthias Radestock-3
On 17/03/13 03:05, carlhoerberg wrote:
> The users says he's just consuming, but the mgmt interface shows that's
> he's both sending and receiving traffic in ~20kb/s, but only receiving
> 0.5msg/sec and not delivering any messages.

Are you saying there's 20kb/s traffic each way on the connection, but
the publishing rate is 0 msg/s and the consuming rate is 0.5 msg/s?

Perhaps there is some weird ssl chatter going on. Could you ask the user
to connect without SSL?

It would be good to see a screen shot of the connection details for the
connection(s) by that user, so we can see both the data rates on the
connection and message rates on that connection's channels all on one
screen.

Also, you said before that "memory usage will grow indefinitely until
[rabbit] crashes". Please post the output of 'rabbitmqctl status' when
you see the high memory usage, which should give us some idea where the
memory is going.

> When the connection if force closed this is returned:
>
> {exit, {channel_termination_timeout, {gen_server,call,
> [<0.27132.1>,{shutdown,"Closed via management plugin"},infinity]}},
> [{gen_server,call,3}, {rabbit_mgmt_wm_connection,delete_resource,2},
> {webmachine_resource,resource_call,3}, {webmachine_resource,do,3},
> {webmachine_decision_core,resource_call,1},
> {webmachine_decision_core,decision,1},
> {webmachine_decision_core,handle_request,2},
> {rabbit_webmachine,'-makeloop/1-fun-0-',2}]}

Is that on a really busy server? There is a grace period of 3 seconds x
number_of_channels during which rabbit tries to terminate connection
gracefully. If that time threshold is exceeded then the connection is
terminated regardless. The above message is a symptom of that. I have
filed a a bug to clean up the error reporting in the management UI for that.

Regards,

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: RabbitMQ 3.0, SSL and rc4

carlhoerberg
The same thing even without SSL. Screenhosts of both a user connection and the node memory status page:

https://www.dropbox.com/s/4xoye3uilej53z9/Screen%20Shot%202013-03-16%20at%2022.48.50.png
https://www.dropbox.com/s/dbousjdaccvqgj8/Screen%20Shot%202013-03-18%20at%2012.22.32.png


On Sunday 17 March 2013 at 17:19, Matthias Radestock-3 [via RabbitMQ] wrote:

> On 17/03/13 03:05, carlhoerberg wrote:
> > The users says he's just consuming, but the mgmt interface shows that's
> > he's both sending and receiving traffic in ~20kb/s, but only receiving
> > 0.5msg/sec and not delivering any messages.
> >
>
>
> Are you saying there's 20kb/s traffic each way on the connection, but
> the publishing rate is 0 msg/s and the consuming rate is 0.5 msg/s?
>
> Perhaps there is some weird ssl chatter going on. Could you ask the user
> to connect without SSL?
>
> It would be good to see a screen shot of the connection details for the
> connection(s) by that user, so we can see both the data rates on the
> connection and message rates on that connection's channels all on one
> screen.
>
> Also, you said before that "memory usage will grow indefinitely until
> [rabbit] crashes". Please post the output of 'rabbitmqctl status' when
> you see the high memory usage, which should give us some idea where the
> memory is going.
>
> > When the connection if force closed this is returned:
> >
> > {exit, {channel_termination_timeout, {gen_server,call,
> > [<0.27132.1>,{shutdown,"Closed via management plugin"},infinity]}},
> > [{gen_server,call,3}, {rabbit_mgmt_wm_connection,delete_resource,2},
> > {webmachine_resource,resource_call,3}, {webmachine_resource,do,3},
> > {webmachine_decision_core,resource_call,1},
> > {webmachine_decision_core,decision,1},
> > {webmachine_decision_core,handle_request,2},
> > {rabbit_webmachine,'-makeloop/1-fun-0-',2}]}
> >
>
> Is that on a really busy server? There is a grace period of 3 seconds x
> number_of_channels during which rabbit tries to terminate connection
> gracefully. If that time threshold is exceeded then the connection is
> terminated regardless. The above message is a symptom of that. I have
> filed a a bug to clean up the error reporting in the management UI for that.
>
> Regards,
>
> Matthias.
> _______________________________________________
> rabbitmq-discuss mailing list
> [hidden email] (/user/SendEmail.jtp?type=node&node=25492&i=0)
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
> If you reply to this email, your message will be added to the discussion below: http://rabbitmq.1065348.n5.nabble.com/RabbitMQ-3-0-SSL-and-rc4-tp25483p25492.html 
> To unsubscribe from RabbitMQ 3.0, SSL and rc4, click here (
> NAML (
http://rabbitmq.1065348.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml)
>



Reply | Threaded
Open this post in threaded view
|

Re: RabbitMQ 3.0, SSL and rc4

Matthias Radestock-3
On 18/03/13 11:29, carlhoerberg wrote:
> The same thing even without SSL. Screenhosts of both a user
> connection and the node memory status page

Can you get us a full 'rabbitmqctl report' when the above is happening?
Please send it off-list to [hidden email].

Also, it would be very helpful to find out what traffic is passing over
those connections. Can you insert the tracer
(http://www.rabbitmq.com/api-guide.html#tracer) in the connection path?
Or, failing that, get a wireshark capture?

Regards,

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: RabbitMQ 3.0, SSL and rc4

carlhoerberg
The trace log is sent. The weird thing is that the problem with memory explosion didn't occur when connecting through the trace proxy!



On Monday 18 March 2013 at 20:03, Matthias Radestock-3 [via RabbitMQ] wrote:

> On 18/03/13 11:29, carlhoerberg wrote:
> > The same thing even without SSL. Screenhosts of both a user
> > connection and the node memory status page
>
>
> Can you get us a full 'rabbitmqctl report' when the above is happening?
> Please send it off-list to [hidden email] (/user/SendEmail.jtp?type=node&node=25502&i=0).
>
> Also, it would be very helpful to find out what traffic is passing over
> those connections. Can you insert the tracer
> (http://www.rabbitmq.com/api-guide.html#tracer) in the connection path?
> Or, failing that, get a wireshark capture?
>
> Regards,
>
> Matthias.
> _______________________________________________
> rabbitmq-discuss mailing list
> [hidden email] (/user/SendEmail.jtp?type=node&node=25502&i=1)
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
> If you reply to this email, your message will be added to the discussion below: http://rabbitmq.1065348.n5.nabble.com/RabbitMQ-3-0-SSL-and-rc4-tp25483p25502.html 
> To unsubscribe from RabbitMQ 3.0, SSL and rc4, click here (
> NAML (
http://rabbitmq.1065348.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml)



Reply | Threaded
Open this post in threaded view
|

Re: RabbitMQ 3.0, SSL and rc4

Matthias Radestock-3
On 19/03/13 07:30, carlhoerberg wrote:
> The trace log is sent. The weird thing is that the problem with memory
> explosion didn't occur when connecting through the trace proxy!

Cheers.

Looking at the trace, we can see that there are two connections with one
channel each that create *lots* of consumers, all on the same two
queues. More than 20,000 consumers in ~15 minutes.

These consumers are never cancelled; the client just keeps creating more
and more of them.

You should be able to confirm this by looking at the consumer count in
the management UI overview.

This looks like a bug in the application or client library, and explains
why rabbit is consuming increasing amounts of memory, since each of
these consumers requires some bookkeeping.

Regards,

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: RabbitMQ 3.0, SSL and rc4

carlhoerberg
Thanks a lot Matthias! Yes, I can see it now. Weird though that it don't happens when connected through the tracer proxy or with rabbitmq 2.8.7. Continuing the discussion with the user and the langohr ppl.



On Tuesday 19 March 2013 at 18:04, Matthias Radestock-3 [via RabbitMQ] wrote:

> On 19/03/13 07:30, carlhoerberg wrote:
> > The trace log is sent. The weird thing is that the problem with memory
> > explosion didn't occur when connecting through the trace proxy!
>
>
> Cheers.
>
> Looking at the trace, we can see that there are two connections with one
> channel each that create *lots* of consumers, all on the same two
> queues. More than 20,000 consumers in ~15 minutes.
>
> These consumers are never cancelled; the client just keeps creating more
> and more of them.
>
> You should be able to confirm this by looking at the consumer count in
> the management UI overview.
>
> This looks like a bug in the application or client library, and explains
> why rabbit is consuming increasing amounts of memory, since each of
> these consumers requires some bookkeeping.
>
> Regards,
>
> Matthias.
> _______________________________________________
> rabbitmq-discuss mailing list
> [hidden email] (/user/SendEmail.jtp?type=node&node=25542&i=0)
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
> If you reply to this email, your message will be added to the discussion below: http://rabbitmq.1065348.n5.nabble.com/RabbitMQ-3-0-SSL-and-rc4-tp25483p25542.html 
> To unsubscribe from RabbitMQ 3.0, SSL and rc4, click here (
> NAML (
http://rabbitmq.1065348.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml)



Reply | Threaded
Open this post in threaded view
|

Re: RabbitMQ 3.0, SSL and rc4

Matthias Radestock-3
On 19/03/13 11:28, carlhoerberg wrote:
> Thanks a lot Matthias! Yes, I can see it now. Weird though that it don't
> happens when connected through the tracer proxy or with rabbitmq 2.8.7.
> Continuing the discussion with the user and the langohr ppl.

Could just differences in performance. Certainly if the management UI /
'rabbitmqctl list_consumers' show a growing number of consumers then
memory usage will grow correspondingly. Running through the tracer
introduces some latency, so will slow things down a bit and hence reduce
the growth rate. And 2.8.7 is generally slower than 3.0, so will exhibit
a slower growth rate.

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss