Queue creation error

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Queue creation error

dave sinclair
Hi all,

I am currently running a 2 node cluster with Rabbit 3.1.5. We have policies setup to ensure that certain queues are mirrored on all the nodes in a cluster. 

Operations was configuring a new cluster and put the hipe_compile flag to true (normally set to false). When they did this, queue creation was failing for queues that needed to be mirrored. The error is below

=INFO REPORT==== 10-Jun-2014::19:19:52 ===
Adding mirror of queue 'title-server-test' in vhost '/test' on node 'rabbit@udbamq-cmc-02': <14717.5271.0>

=ERROR REPORT==== 10-Jun-2014::19:19:53 ===
** Generic server <0.5257.0> terminating
** Last message in was pre_hibernate
** When Server state == {q,
                         {amqqueue,
                          {resource,<<"/test">>,queue,
                           <<"title-server-test">>},
                          true,false,none,[],<0.5257.0>,[],[],
                          [{vhost,<<"/test">>},
                           {name,<<"unu-ha-all">>},
                           {pattern,
                            <<"menu-item-server.*|title-server.*|unu-server.*">>},
                           {definition,[{<<"ha-mode">>,<<"all">>}]},
                           {priority,0}],
                          []},
                         none,false,rabbit_mirror_queue_master,
                         {state,
                          {resource,<<"/test">>,queue,
                           <<"title-server-test">>},
                          <0.5259.0>,<0.5258.0>,rabbit_variable_queue,
                          {vqstate,
                           {0,{[],[]}},
                           {0,{[],[]}},
                           {delta,undefined,0,undefined},
                           {0,{[],[]}},
                           {0,{[],[]}},
                           0,
                           {0,nil},
                           {0,nil},
                           {qistate,
                            "/var/lib/rabbitmq/mnesia/rabbit@udbamq-cmc-01/queues/C3QO8FHBOV9KWRA6JK601BOPG",
                            {{dict,0,16,16,8,80,48,
                              {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                               []},
                              {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                []}}},
                             []},
                            undefined,0,65536,
                            #Fun<rabbit_variable_queue.2.81334491>,
                            {0,nil}},
                           {{client_msstate,msg_store_persistent,
                             <<155,86,72,191,145,120,119,132,195,47,38,124,147,
                               109,237,88>>,
                             {dict,0,16,16,8,80,48,
                              {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                               []},
                              {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                []}}},
                             {state,35409996,
                              "/var/lib/rabbitmq/mnesia/rabbit@udbamq-cmc-01/msg_store_persistent"},
                             rabbit_msg_store_ets_index,
                             "/var/lib/rabbitmq/mnesia/rabbit@udbamq-cmc-01/msg_store_persistent",
                             <0.4760.0>,35414093,35405899,35418190,35422287},
                            {client_msstate,msg_store_transient,
                             <<167,204,117,67,245,155,36,187,165,217,239,159,
                               54,180,2,199>>,
                             {dict,0,16,16,8,80,48,
                              {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                               []},
                              {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                []}}},
                             {state,35389511,
                              "/var/lib/rabbitmq/mnesia/rabbit@udbamq-cmc-01/msg_store_transient"},
                             rabbit_msg_store_ets_index,
                             "/var/lib/rabbitmq/mnesia/rabbit@udbamq-cmc-01/msg_store_transient",
                             <0.4753.0>,35393608,35385408,35397705,35401802}},
                           true,0,0,0,infinity,0,0,0,0,0,
                           {rates,
                            {{1402,427992,812678},0},
                            {{1402,427992,812678},0},
                            0.0,0.0,
                            {1402,427992,812678}},
                           {0,nil},
                           {0,nil},
                           {0,nil},
                           {0,nil},
                           0,0,
                           {rates,
                            {{1402,427992,812678},0},
                            {{1402,427992,812678},0},
                            0.0,0.0,
                            {1402,427992,812678}}},
                          {dict,0,16,16,8,80,48,
                           {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                           {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                             []}}},
                          [],
                          {set,0,16,16,8,80,48,
                           {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                           {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                             []}}}},
                         {[],[]},
                         undefined,undefined,#Ref<0.0.0.146892>,undefined,
                         {state,fine,5000,#Ref<0.0.0.146893>},
                         {0,nil},
                         undefined,undefined,undefined,
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         undefined,undefined,undefined,running}
** Reason for termination ==
** {badarith,[{rabbit_variable_queue,ram_duration,1},
              {rabbit_mnesia,node_info,0},
              {rabbit_amqqueue_process,handle_pre_hibernate,1},
              {gen_server2,pre_hibernate,1},
              {rabbit_mnesia,node_info,0}]}

From the sasl log

=CRASH REPORT==== 10-Jun-2014::19:19:53 ===
  crasher:
    initial call: gen:init_it/6
    pid: <0.5257.0>
    registered_name: []
    exception exit: {badarith,
                        [{rabbit_variable_queue,ram_duration,1},
                         {rabbit_mnesia,node_info,0},
                         {rabbit_amqqueue_process,handle_pre_hibernate,1},
                         {gen_server2,pre_hibernate,1},
                         {rabbit_mnesia,node_info,0}]}
      in function  gen_server2:terminate/3
      in call from rabbit_mnesia:node_info/0
    ancestors: [rabbit_amqqueue_sup,rabbit_sup,<0.4622.0>]
    messages: [{'EXIT',<0.5258.0>,{shutdown,ring_shutdown}}]
    links: [<0.4761.0>]
    dictionary: [{{xtype_to_module,direct},rabbit_exchange_type_direct},
                  {guid,{{2815194435,4120585403,2782523295,917766855},1}}]
    trap_exit: true
    status: running
    heap_size: 2584
    stack_size: 24
    reductions: 2122
  neighbours:

=SUPERVISOR REPORT==== 10-Jun-2014::19:19:53 ===
     Supervisor: {local,rabbit_amqqueue_sup}
     Context:    child_terminated
     Reason:     {badarith,[{rabbit_variable_queue,ram_duration,1},
                            {rabbit_mnesia,node_info,0},
                            {rabbit_amqqueue_process,handle_pre_hibernate,1},
                            {gen_server2,pre_hibernate,1},
                            {rabbit_mnesia,node_info,0}]}
     Offender:   [{pid,<0.5257.0>},
                  {name,rabbit_amqqueue},
                  {mfa,
                      {rabbit_amqqueue_process,start_link,
                          [{amqqueue,
                               {resource,<<"/unu-broker">>,queue,
                                   <<"title-server-date">>},
                               true,false,none,[],none,[],[],
                               [{vhost,<<"/unu-broker">>},
                                {name,<<"unu-ha-all">>},
                                {pattern,
                                    <<"menu-item-server.*|title-server.*|unu-server.*">>},
                                {definition,[{<<"ha-mode">>,<<"all">>}]},
                                {priority,0}],
                               []}]}},
                  {restart_type,temporary},
                  {shutdown,4294967295},
                  {child_type,worker}]


=CRASH REPORT==== 10-Jun-2014::19:21:33 ===
  crasher:
    initial call: mochiweb_acceptor:init/3
    pid: <0.4789.0>
    registered_name: []
    exception exit: {error,accept_failed}
      in function  mochiweb_acceptor:init/3
    ancestors: [rabbit_web_dispatch_sup_55672,rabbit_web_dispatch_sup,
                  <0.4613.0>]
    messages: []
    links: []
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 377
    stack_size: 24
    reductions: 2378
  neighbours:

=CRASH REPORT==== 10-Jun-2014::19:21:33 ===
  crasher:
    initial call: mochiweb_acceptor:init/3
    pid: <0.4787.0>
    registered_name: []
    exception exit: {error,accept_failed}
      in function  mochiweb_acceptor:init/3
    ancestors: [rabbit_web_dispatch_sup_55672,rabbit_web_dispatch_sup,
                  <0.4613.0>]
    messages: []
    links: []
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 377
    stack_size: 24
    reductions: 2383
  neighbours:

We can run with hipe_compile=false, but wanted to see if anyone else has seen this.

thanks

dave



_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Queue creation error

Matthias Radestock-3
On 10/06/14 20:50, David Sinclair wrote:

> I am currently running a 2 node cluster with Rabbit 3.1.5. [...]
> Operations was configuring a new cluster and put the hipe_compile flag
> to true (normally set to false). When they did this, queue creation was
> failing for queues that needed to be mirrored. The error is
> [...]
>      exception exit: {badarith,
>                          [{rabbit_variable_queue,ram_duration,1},
>                           {rabbit_mnesia,node_info,0},
>                           {rabbit_amqqueue_process,handle_pre_hibernate,1},
>                           {gen_server2,pre_hibernate,1},
>                           {rabbit_mnesia,node_info,0}]}

The code that is failing has changed in more recent versions of rabbit,
avoiding a floating point arithmetic rounding issue that could possibly
trigger the above error. Please upgrade.

Also, please make sure you are using the latest Erlang version, since
hipe compilation is an active area of development (and bug fixing).

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Queue creation error

dave sinclair
Thanks Matthias. I will get them upgraded and see if that fixes the problem.


On Wed, Jun 11, 2014 at 3:15 AM, Matthias Radestock <[hidden email]> wrote:
On 10/06/14 20:50, David Sinclair wrote:
I am currently running a 2 node cluster with Rabbit 3.1.5. [...]

Operations was configuring a new cluster and put the hipe_compile flag
to true (normally set to false). When they did this, queue creation was
failing for queues that needed to be mirrored. The error is
[...]

     exception exit: {badarith,
                         [{rabbit_variable_queue,ram_duration,1},
                          {rabbit_mnesia,node_info,0},
                          {rabbit_amqqueue_process,handle_pre_hibernate,1},
                          {gen_server2,pre_hibernate,1},
                          {rabbit_mnesia,node_info,0}]}

The code that is failing has changed in more recent versions of rabbit, avoiding a floating point arithmetic rounding issue that could possibly trigger the above error. Please upgrade.

Also, please make sure you are using the latest Erlang version, since hipe compilation is an active area of development (and bug fixing).

Matthias.


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss