badmatch when unbinding queue in web management

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

badmatch when unbinding queue in web management

Gotthard, Petr

Hello,

There is a strange queue in our RabbitMQ 3.2.2 (Erlang R16B01):

 

1) Web management says “Consumers: 1” in the Details section, but then “…no consumers…” in the Consumers section.

 

2) When I do “Delete / Purge” nothing happens. The Web management stops responding and no error appears in the log file.

 

3) When I try to unbind this queue nothing happens too, but there is a badmatch in the rabbitmq.log

 

=ERROR REPORT==== 28-Mar-2014::08:46:40 ===

** Generic server <0.7525.75> terminating

** Last message in was {'$gen_cast',

                           {method,

                               {'queue.unbind',0,<<"efb.ok-3">>,<<"update">>,

                                   <<"aircraft.OK-3">>,

                                   [{<<"x-ground-bind">>,bool,true},

                                    {<<"x-ground-cos">>,long,3},

                                    {<<"x-ground-exchange">>,longstr,

                                     <<"update">>},

                                    {<<"x-reference">>,longstr,

                                     <<"Ground bind">>},

                                    {<<"x-result-x">>,longstr,

                                     <<"efb.ok-3">>}]},

                               none,noflow}}

** When Server state == {ch,running,rabbit_framing_amqp_0_9_1,1,<0.7520.75>,

                            <0.7520.75>,<0.7518.75>,

                            <<"<rabbit@ground.3.7518.75>">>,

                            {lstate,<0.7526.75>,false,false},

                            none,1,

                            {[],[]},

                            {user,<<"swim">>,

                                [administrator,impersonator],

                                rabbit_auth_backend_internal,

                                {internal_user,<<"swim">>,

                                    <<139,60,61,83,54,226,9,72,9,158,69,131,

                                      112,210,248,0,214,125,108,114>>,

                                    [administrator,impersonator]}},

                            <<"pavel">>,<<>>,

                            {dict,0,16,16,8,80,48,

                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                 []},

                                {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                  [],[]}}},

                            {state,

                                {dict,0,16,16,8,80,48,

                                    {[],[],[],[],[],[],[],[],[],[],[],[],[],

                                     [],[],[]},

                                    {{[],[],[],[],[],[],[],[],[],[],[],[],[],

                                      [],[],[]}}},

                                erlang},

                            {dict,0,16,16,8,80,48,

                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                 []},

                                {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                  [],[]}}},

                            {set,0,16,16,8,80,48,

                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                 []},

                                {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                  [],[]}}},

                            {dict,0,16,16,8,80,48,

                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                 []},

                                {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                  [],[]}}},

                            {set,0,16,16,8,80,48,

                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                 []},

                                {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                  [],[]}}},

                            <0.7515.75>,

                            {state,fine,5000,#Ref<0.0.124.120869>},

                            false,1,

                            {{0,nil},{0,nil}},

                            [],

                            [{<<"publisher_confirms">>,bool,true},

                             {<<"exchange_exchange_bindings">>,bool,true},

                             {<<"basic.nack">>,bool,true},

                             {<<"consumer_cancel_notify">>,bool,true},

                             {<<"connection.blocked">>,bool,true},

                             {<<"authentication_failure_close">>,bool,true}],

                            {exchange,

                                {resource,<<"pavel">>,exchange,

                                    <<"amq.rabbitmq.trace">>},

                                topic,true,false,false,[],undefined,undefined,

                                {[],[]}}}

** Reason for termination ==

** {{error,{{badmatch,{error,[{root,none}],["aircraft","OK-3"]}},

            [{rabbit_exchange_type_topic,follow_down_get_path,2,[]},

             {rabbit_exchange_type_topic,'-remove_bindings/3-lc$^1/1-1-',1,[]},

             {rabbit_exchange_type_topic,remove_bindings,3,[]},

             {rabbit_binding,x_callback,4,[]},

             {rabbit_binding,'-process_deletions/1-fun-0-',2,[]},

             {dict,map_bucket,2,[{file,"dict.erl"},{line,460}]},

             {dict,map_bkt_list,2,[{file,"dict.erl"},{line,456}]},

             {dict,map_bkt_list,2,[{file,"dict.erl"},{line,456}]}]}},

    [{rabbit_misc,execute_mnesia_transaction,1,[]},

    {rabbit_misc,execute_mnesia_tx_with_tail,1,[]},

     {rabbit_channel,binding_action,9,[]},

     {rabbit_channel,handle_cast,2,[]},

     {gen_server2,handle_msg,2,[]},

     {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}

 

 

Do you have any idea what could be wrong?

 

Thanks,

Petr

 


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: badmatch when unbinding queue in web management

Simon MacMullen-2
On 28/03/14 08:37, Gotthard, Petr wrote:
> Do you have any idea what could be wrong?

Yes, there's a bug in topic exchanges which is seen when they have
multiple bindings which have identical source, destination and routing
key but differ in arguments.

This bug was introduced in 2.4.0 with fast topic routing, and has been
fixed in hg but has not made it into a release yet - it will be fixed in
3.3.0 when that comes out.

Cheers, Simon

--
Simon MacMullen
RabbitMQ, Pivotal
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: badmatch when unbinding queue in web management

Matthias Radestock-3
On 28/03/14 10:25, Simon MacMullen wrote:
> This bug was introduced in 2.4.0 with fast topic routing, and has been
> fixed in hg but has not made it into a release yet - it will be fixed in
> 3.3.0 when that comes out.

...and meanwhile you could try the nightly build.

Matthias.

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: badmatch when unbinding queue in web management

Gotthard, Petr
In reply to this post by Simon MacMullen-2
I'm not sure this is the case; we don't have multiple identical bindings with different arguments.


In the meantime we discovered the scenario that causes the web unbinds to fail:
 -- client creates a binding with some parameters
 -- clients unbinds with different parameters (which will fail) and disconnects immediately

It seems that if I disconnect before the failed unbind is fully processed, the queue is left in some strange state. In this state the queue remains "active" forever and cannot be deleted.
Even the standard broker shutdown doesn't work and it is necessary to kill the broker by brute force (-9). After the broker is restarted, the queue can be deleted without any troubles. It almost looks as a way to cripple the broker using a standard AMQP client (a possible DOS attack?).

On the other hand, if there is some wait/sleep between the wrong unbind and the disconnection, everything works fine. This makes me believe there is some race condition in the "wrong unbind-disconnect" scenario.


Petr

-----Original Message-----
From: Simon MacMullen [mailto:[hidden email]]
Sent: 28. března 2014 11:25
To: Discussions about RabbitMQ
Cc: Gotthard, Petr
Subject: Re: [rabbitmq-discuss] badmatch when unbinding queue in web management

On 28/03/14 08:37, Gotthard, Petr wrote:
> Do you have any idea what could be wrong?

Yes, there's a bug in topic exchanges which is seen when they have
multiple bindings which have identical source, destination and routing
key but differ in arguments.

This bug was introduced in 2.4.0 with fast topic routing, and has been
fixed in hg but has not made it into a release yet - it will be fixed in
3.3.0 when that comes out.

Cheers, Simon

--
Simon MacMullen
RabbitMQ, Pivotal
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: badmatch when unbinding queue in web management

Matthias Radestock-3
Petr,

On 28/03/14 11:36, Gotthard, Petr wrote:
> I'm not sure this is the case; we don't have multiple identical
> bindings with different arguments.
>
>
> In the meantime we discovered the scenario that causes the web
> unbinds to fail: -- client creates a binding with some parameters --
> clients unbinds with different parameters (which will fail) and
> disconnects immediately

We've had no joy reproducing this.

If this is easily reproducible for you, would you mind grabbing a
nightly build and repeating your experiment?

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: badmatch when unbinding queue in web management

Gotthard, Petr
Hi Matthias,

It seems that the 3.3.0 does fix this issue for me.

Thanks,
Petr

-----Original Message-----
From: Matthias Radestock [mailto:[hidden email]]
Sent: 31. března 2014 13:05
To: Discussions about RabbitMQ
Cc: Gotthard, Petr
Subject: Re: [rabbitmq-discuss] badmatch when unbinding queue in web management

Petr,

On 28/03/14 11:36, Gotthard, Petr wrote:
> I'm not sure this is the case; we don't have multiple identical
> bindings with different arguments.
>
>
> In the meantime we discovered the scenario that causes the web
> unbinds to fail: -- client creates a binding with some parameters --
> clients unbinds with different parameters (which will fail) and
> disconnects immediately

We've had no joy reproducing this.

If this is easily reproducible for you, would you mind grabbing a
nightly build and repeating your experiment?

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss