RabbitMQ

Loss of durable queues on HA sync

Classic

List

Threaded

2 messages Options

Miller Andrew

Loss of durable queues on HA sync

During some server reboots at our organization over the weekend, several nodes in our non-Prod cluster had some crashes, and in the end, 7 of 84 durable queues in the cluster were gone. All nodes are running RabbitMQ 3.3.3 on Erlang R16B03-1. This is a 3-node cluster, with cluster_partition_handling = autoheal. I'm curious if this is a known issue with autoheal? I tried recreating the issue this morning by recreating the series of events, but was unable to. Our other 2-node non-Prod cluster, with no cluster_partition_handling, was also rebootted, but didn't have any problems.

Thanks,

Andrew Miller

08:12:01 - Node DIX-D-EVGAPP-02 (a disk node) saw node STR-D-EVGAPP-02 (a memory node) go down (which had been the master node for many queues), and was promoted to master for 75 queues, including those which later would disappear.

08:13:30 - Node DIX-D-EVGAPP-02 saw node STR-D-EVGAPP-02 come back, and began synchronizing back. Most queues synchronized fine, but 7 crashed with errors similar to the following - and each of these queues were gone after this. The third node (another disk node) doesn't have any crashes in its log around this time.

=ERROR REPORT==== 27-Jul-2014::08:13:30 ===

** Generic server <0.332.0> terminating

** Last message in was {add_on_right,{20,<4967.323.0>}}

** When Server state == {q,

{amqqueue,

{resource,<<"eventing">>,queue,

<<"SalesCenter2.BPClaims_ClosedClaim">>},

true,false,none,

[{<<"x-dead-letter-exchange">>,longstr,

<<"EnterpriseEventDeadLetter">>},

{<<"x-dead-letter-routing-key">>,longstr,

<<"SalesCenter2.BPClaims_ClosedClaim">>}],

<0.332.0>,

[<8174.310.0>],

[{vhost,<<"eventing">>},

{name,<<"ha-all">>},

{pattern,<<"^.*">>},

{'apply-to',<<"queues">>},

{definition,

[{<<"ha-mode">>,<<"all">>},

{<<"ha-sync-mode">>,<<"automatic">>}]},

{priority,0}],

[{<8174.311.0>,<8174.310.0>},{<0.333.0>,<0.332.0>}],

[]},

none,false,rabbit_mirror_queue_master,

{state,

{resource,<<"eventing">>,queue,

<<"SalesCenter2.BPClaims_ClosedClaim">>},

<0.333.0>,<0.4223.0>,rabbit_variable_queue,

{vqstate,

{0,{[],[]}},

{delta,undefined,0,undefined},

{0,{[],[]}},

{0,nil},

{qistate,

"d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",

{{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[]}}},

[]},

undefined,0,65536,

#Fun<rabbit_variable_queue.2.51560583>,

{0,nil}},

{{client_msstate,msg_store_persistent,

<<26,131,31,121,222,176,41,144,168,192,42,159,222,

13,179,107>>,

{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[]}}},

{state,311371,

"d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},

rabbit_msg_store_ets_index,

"d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent",

<0.266.0>,315468,307274,319565,323662},

{client_msstate,msg_store_transient,

<<32,49,130,217,86,65,129,214,32,140,24,34,254,

129,171,73>>,

{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[]}}},

{state,290886,

"d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient"},

rabbit_msg_store_ets_index,

"d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient",

<0.261.0>,294983,286789,299080,303177}},

true,0,0,0,infinity,0,0,0,0,0,

{rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},

{0,nil},

0,0},

{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[]}}},

[],

{set,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[]}}}},

{state,{queue,[],[],0},{active,1406466722448182,1.0}},

undefined,undefined,undefined,undefined,

{state,fine,5000,undefined},

{0,nil},

undefined,undefined,undefined,

{state,

{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[]}}},

delegate},

{resource,<<"eventing">>,exchange,

<<"EnterpriseEventDeadLetter">>},

<<"SalesCenter2.BPClaims_ClosedClaim">>,undefined,2,

running}

** Reason for termination ==

** {function_clause,

[{rabbit_amqqueue_process,handle_call,

[{add_on_right,{20,<4967.323.0>}},

{<4967.323.0>,#Ref<4967.0.0.2348>},

{q,{amqqueue,

{resource,<<"eventing">>,queue,

<<"SalesCenter2.BPClaims_ClosedClaim">>},

true,false,none,

[{<<"x-dead-letter-exchange">>,longstr,

<<"EnterpriseEventDeadLetter">>},

{<<"x-dead-letter-routing-key">>,longstr,

<<"SalesCenter2.BPClaims_ClosedClaim">>}],

<0.332.0>,

[<8174.310.0>],

[{vhost,<<"eventing">>},

{name,<<"ha-all">>},

{pattern,<<"^.*">>},

{'apply-to',<<"queues">>},

{definition,

[{<<"ha-mode">>,<<"all">>},

{<<"ha-sync-mode">>,<<"automatic">>}]},

{priority,0}],

[{<8174.311.0>,<8174.310.0>},{<0.333.0>,<0.332.0>}],

[]},

none,false,rabbit_mirror_queue_master,

{state,

{resource,<<"eventing">>,queue,

<<"SalesCenter2.BPClaims_ClosedClaim">>},

<0.333.0>,<0.4223.0>,rabbit_variable_queue,

{vqstate,

{0,{[],[]}},

{delta,undefined,0,undefined},

{0,{[],[]}},

{0,nil},

{qistate,

"d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",

{{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[]}}},

[]},

undefined,0,65536,

#Fun<rabbit_variable_queue.2.51560583>,

{0,nil}},

{{client_msstate,msg_store_persistent,

<<26,131,31,121,222,176,41,144,168,192,42,159,222,

13,179,107>>,

{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[]}}},

{state,311371,

"d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},

rabbit_msg_store_ets_index,

"d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent",

<0.266.0>,315468,307274,319565,323662},

{client_msstate,msg_store_transient,

<<32,49,130,217,86,65,129,214,32,140,24,34,254,

129,171,73>>,

{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[]}}},

{state,290886,

"d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient"},

rabbit_msg_store_ets_index,

"d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient",

<0.261.0>,294983,286789,299080,303177}},

true,0,0,0,infinity,0,0,0,0,0,

{rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},

{0,nil},

0,0},

{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},

[],

{set,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}},

{state,{queue,[],[],0},{active,1406466722448182,1.0}},

undefined,undefined,undefined,undefined,

{state,fine,5000,undefined},

{0,nil},

undefined,undefined,undefined,

{state,

{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},

delegate},

{resource,<<"eventing">>,exchange,

<<"EnterpriseEventDeadLetter">>},

<<"SalesCenter2.BPClaims_ClosedClaim">>,undefined,2,running}],

[]},

{gen_server2,handle_msg,2,[]},

{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]}

08:13:30 - Node STR-D-EVGAPP-02 also recorded crashes involving the same queues:

=ERROR REPORT==== 27-Jul-2014::08:13:30 ===

** Generic server <0.323.0> terminating

** Last message in was {'$gen_cast',join}

** When Server state == {state,

{20,<0.323.0>},

{{20,<0.323.0>},undefined},

{resource,<<"eventing">>,queue,

<<"Insight.BPClaims_ModifiedWorkNotAvailable">>},

rabbit_mirror_queue_slave,undefined,-1,undefined,

[<0.322.0>],

{[],[]},

[],0,undefined,

#Fun<rabbit_misc.execute_mnesia_transaction.1>}

** Reason for termination ==

** {{function_clause,

[{rabbit_amqqueue_process,handle_call,

[{add_on_right,{20,<0.323.0>}},

{<0.323.0>,#Ref<0.0.0.2348>},

{q,{amqqueue,

{resource,<<"eventing">>,queue,

<<"SalesCenter2.BPClaims_ClosedClaim">>},

true,false,none,

[{<<"x-dead-letter-exchange">>,longstr,

<<"EnterpriseEventDeadLetter">>},

{<<"x-dead-letter-routing-key">>,longstr,

<<"SalesCenter2.BPClaims_ClosedClaim">>}],

<5297.332.0>,

[<5251.310.0>],

[{vhost,<<"eventing">>},

{name,<<"ha-all">>},

{pattern,<<"^.*">>},

{'apply-to',<<"queues">>},

{definition,

[{<<"ha-mode">>,<<"all">>},

{<<"ha-sync-mode">>,<<"automatic">>}]},

{priority,0}],

[{<5251.311.0>,<5251.310.0>},{<5297.333.0>,<5297.332.0>}],

[]},

none,false,rabbit_mirror_queue_master,

{state,

{resource,<<"eventing">>,queue,

<<"SalesCenter2.BPClaims_ClosedClaim">>},

<5297.333.0>,<5297.4223.0>,rabbit_variable_queue,

{vqstate,

{0,{[],[]}},

{delta,undefined,0,undefined},

{0,{[],[]}},

{0,nil},

{qistate,

"d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",

{{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[]}}},

[]},

undefined,0,65536,

#Fun<rabbit_variable_queue.2.51560583>,

{0,nil}},

{{client_msstate,msg_store_persistent,

<<26,131,31,121,222,176,41,144,168,192,42,159,

222,13,179,107>>,

{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[]}}},

{state,311371,

"d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},

rabbit_msg_store_ets_index,

"d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent",

<5297.266.0>,315468,307274,319565,323662},

{client_msstate,msg_store_transient,

<<32,49,130,217,86,65,129,214,32,140,24,34,254,

129,171,73>>,

{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[]}}},

{state,290886,

"d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient"},

rabbit_msg_store_ets_index,

"d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient",

<5297.261.0>,294983,286789,299080,303177}},

true,0,0,0,infinity,0,0,0,0,0,

{rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},

{0,nil},

0,0},

{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},

[],

{set,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}},

{state,{queue,[],[],0},{active,1406466722448182,1.0}},

undefined,undefined,undefined,undefined,

{state,fine,5000,undefined},

{0,nil},

undefined,undefined,undefined,

{state,

{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},

delegate},

{resource,<<"eventing">>,exchange,

<<"EnterpriseEventDeadLetter">>},

<<"SalesCenter2.BPClaims_ClosedClaim">>,undefined,2,running}],

[]},

{gen_server2,handle_msg,2,[]},

{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]},

{gen_server2,call,[<5297.332.0>,{add_on_right,{20,<0.323.0>}},infinity]}}

08:13:31 - And again from node STR-D-EVGAPP-02:

=ERROR REPORT==== 27-Jul-2014::08:13:31 ===

** Generic server <0.322.0> terminating

** Last message in was {'EXIT',<0.323.0>,

{{function_clause,

[{rabbit_amqqueue_process,handle_call,

[{add_on_right,{20,<0.323.0>}},

{<0.323.0>,#Ref<0.0.0.2348>},

{q,

{amqqueue,

{resource,<<"eventing">>,queue,

<<"SalesCenter2.BPClaims_ClosedClaim">>},

true,false,none,

[{<<"x-dead-letter-exchange">>,longstr,

<<"EnterpriseEventDeadLetter">>},

{<<"x-dead-letter-routing-key">>,longstr,

<<"SalesCenter2.BPClaims_ClosedClaim">>}],

<5297.332.0>,

[<5251.310.0>],

[{vhost,<<"eventing">>},

{name,<<"ha-all">>},

{pattern,<<"^.*">>},

{'apply-to',<<"queues">>},

{definition,

[{<<"ha-mode">>,<<"all">>},

{<<"ha-sync-mode">>,<<"automatic">>}]},

{priority,0}],

[{<5251.311.0>,<5251.310.0>},

{<5297.333.0>,<5297.332.0>}],

[]},

none,false,rabbit_mirror_queue_master,

{state,

{resource,<<"eventing">>,queue,

<<"SalesCenter2.BPClaims_ClosedClaim">>},

<5297.333.0>,<5297.4223.0>,

rabbit_variable_queue,

{vqstate,

{0,{[],[]}},

{delta,undefined,0,undefined},

{0,{[],[]}},

{0,nil},

{qistate,

"d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",

{{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[],[]}}},

[]},

undefined,0,65536,

#Fun<rabbit_variable_queue.2.51560583>,

{0,nil}},

{{client_msstate,msg_store_persistent,

<<26,131,31,121,222,176,41,144,168,192,42,

159,222,13,179,107>>,

{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[],[]}}},

{state,311371,

"d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},

rabbit_msg_store_ets_index,

"d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent",

<5297.266.0>,315468,307274,319565,323662},

{client_msstate,msg_store_transient,

<<32,49,130,217,86,65,129,214,32,140,24,34,

254,129,171,73>>,

{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[],[]}}},

{state,290886,

"d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient"},

rabbit_msg_store_ets_index,

"d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient",

<5297.261.0>,294983,286789,299080,303177}},

true,0,0,0,infinity,0,0,0,0,0,

{rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},

{0,nil},

0,0},

{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[]}}},

[],

{set,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[]}}}},

{state,

{queue,[],[],0},

{active,1406466722448182,1.0}},

undefined,undefined,undefined,undefined,

{state,fine,5000,undefined},

{0,nil},

undefined,undefined,undefined,

{state,

{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[]}}},

delegate},

{resource,<<"eventing">>,exchange,

<<"EnterpriseEventDeadLetter">>},

<<"SalesCenter2.BPClaims_ClosedClaim">>,

undefined,2,running}],

[]},

{gen_server2,handle_msg,2,[]},

{proc_lib,wake_up,3,

[{file,"proc_lib.erl"},{line,249}]}]},

{gen_server2,call,

[<5297.332.0>,

{add_on_right,{20,<0.323.0>}},

infinity]}}}

** When Server state == {state,

{amqqueue,

{resource,<<"eventing">>,queue,

<<"Insight.BPClaims_ModifiedWorkNotAvailable">>},

true,false,none,

[{<<"x-dead-letter-exchange">>,longstr,

<<"EnterpriseEventDeadLetter">>},

{<<"x-dead-letter-routing-key">>,longstr,

<<"Insight.BPClaims_ModifiedWorkNotAvailable">>}],

<5251.335.0>,

[<5297.331.0>],

[{vhost,<<"eventing">>},

{name,<<"ha-all">>},

{pattern,<<"^.*">>},

{'apply-to',<<"queues">>},

{definition,

[{<<"ha-mode">>,<<"all">>},

{<<"ha-sync-mode">>,<<"automatic">>}]},

{priority,0}],

[{<5297.332.0>,<5297.331.0>},

{<5251.336.0>,<5251.335.0>}],

[]},

<0.323.0>,rabbit_variable_queue,

{vqstate,

{0,{[],[]}},

{delta,undefined,0,undefined},

{0,{[],[]}},

{0,nil},

{qistate,

"d:/RabbitMQ/db/rabbit@STR-D-EVGAPP-02-mnesia/queues/11YNGR2UMDIGSF4OWRUUJ2HUX",

{{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[]}}},

[]},

undefined,0,65536,

#Fun<rabbit_variable_queue.2.51560583>,

{0,nil}},

{{client_msstate,msg_store_persistent,

<<142,49,10,175,39,218,164,74,2,215,253,147,201,

168,141,2>>,

{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[]}}},

{state,303177,

"d:/RabbitMQ/db/rabbit@STR-D-EVGAPP-02-mnesia/msg_store_persistent"},

rabbit_msg_store_ets_index,

"d:/RabbitMQ/db/rabbit@STR-D-EVGAPP-02-mnesia/msg_store_persistent",

<0.251.0>,307274,299080,311371,315468},

{client_msstate,msg_store_transient,

<<18,152,170,252,55,149,243,37,252,234,131,180,53,

66,14,182>>,

{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[]}}},

{state,282692,

"d:/RabbitMQ/db/rabbit@STR-D-EVGAPP-02-mnesia/msg_store_transient"},

rabbit_msg_store_ets_index,

"d:/RabbitMQ/db/rabbit@STR-D-EVGAPP-02-mnesia/msg_store_transient",

<0.246.0>,286789,278595,290886,294983}},

true,0,0,0,infinity,0,0,0,0,0,

{rates,0.0,0.0,0.0,0.0,{1406,466811,43211}},

{0,nil},

0,0},

undefined,undefined,

{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},

{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},

{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},

{state,

{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[]}}},

delegate},

undefined}

** Reason for termination ==

** {{function_clause,

[{rabbit_amqqueue_process,handle_call,

[{add_on_right,{20,<0.323.0>}},

{<0.323.0>,#Ref<0.0.0.2348>},

{q,{amqqueue,

{resource,<<"eventing">>,queue,

<<"SalesCenter2.BPClaims_ClosedClaim">>},

true,false,none,

[{<<"x-dead-letter-exchange">>,longstr,

<<"EnterpriseEventDeadLetter">>},

{<<"x-dead-letter-routing-key">>,longstr,

<<"SalesCenter2.BPClaims_ClosedClaim">>}],

<5297.332.0>,

[<5251.310.0>],

[{vhost,<<"eventing">>},

{name,<<"ha-all">>},

{pattern,<<"^.*">>},

{'apply-to',<<"queues">>},

{definition,

[{<<"ha-mode">>,<<"all">>},

{<<"ha-sync-mode">>,<<"automatic">>}]},

{priority,0}],

[{<5251.311.0>,<5251.310.0>},{<5297.333.0>,<5297.332.0>}],

[]},

none,false,rabbit_mirror_queue_master,

{state,

{resource,<<"eventing">>,queue,

<<"SalesCenter2.BPClaims_ClosedClaim">>},

<5297.333.0>,<5297.4223.0>,rabbit_variable_queue,

{vqstate,

{0,{[],[]}},

{delta,undefined,0,undefined},

{0,{[],[]}},

{0,nil},

{qistate,

"d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",

{{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[]}}},

[]},

undefined,0,65536,

#Fun<rabbit_variable_queue.2.51560583>,

{0,nil}},

{{client_msstate,msg_store_persistent,

<<26,131,31,121,222,176,41,144,168,192,42,159,

222,13,179,107>>,

{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[]}}},

{state,311371,

"d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},

rabbit_msg_store_ets_index,

"d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent",

<5297.266.0>,315468,307274,319565,323662},

{client_msstate,msg_store_transient,

<<32,49,130,217,86,65,129,214,32,140,24,34,254,

129,171,73>>,

{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[]}}},

{state,290886,

"d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient"},

rabbit_msg_store_ets_index,

"d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient",

<5297.261.0>,294983,286789,299080,303177}},

true,0,0,0,infinity,0,0,0,0,0,

{rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},

{0,nil},

0,0},

{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},

[],

{set,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}},

{state,{queue,[],[],0},{active,1406466722448182,1.0}},

undefined,undefined,undefined,undefined,

{state,fine,5000,undefined},

{0,nil},

undefined,undefined,undefined,

{state,

{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},

delegate},

{resource,<<"eventing">>,exchange,

<<"EnterpriseEventDeadLetter">>},

<<"SalesCenter2.BPClaims_ClosedClaim">>,undefined,2,running}],

[]},

{gen_server2,handle_msg,2,[]},

{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]},

{gen_server2,call,[<5297.332.0>,{add_on_right,{20,<0.323.0>}},infinity]}}

This e-mail is confidential. If you are not the intended recipient, you must not disclose or use the information contained in it. If you have received this e-mail in error, please tell us immediately by return e-mail and delete the document. No recipient may use the information in this e-mail in violation of any civil or criminal statute. Sentry disclaims all liability for any unauthorized uses of this e-mail or its contents. Sentry accepts no liability or responsibility for any damage caused by any virus transmitted with this e-mail.

_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

Simon MacMullen-2

Re: Loss of durable queues on HA sync

Thanks for this. This is not a known issue, and not something we've seen
before.

In all cases it looks like a queue process crashed because it received a
message intended for a GM process. That should never be able to happen.

Could you send me full logs from all nodes around the time of the crash?
It would be useful to see in what order things happened, if anything led
up to this, when nodes restarted for autoheal (if any autoheal
happened), etc.

Cheers, Simon

On 28/07/14 17:40, Miller Andrew wrote:

> During some server reboots at our organization over the weekend, several
> nodes in our non-Prod cluster had some crashes, and in the end, 7 of 84
> durable queues in the cluster were gone. All nodes are running RabbitMQ
> 3.3.3 on Erlang R16B03-1. This is a 3-node cluster, with
> cluster_partition_handling = autoheal. I'm curious if this is a known
> issue with autoheal? I tried recreating the issue this morning by
> recreating the series of events, but was unable to. Our other 2-node
> non-Prod cluster, with no cluster_partition_handling, was also
> rebootted, but didn't have any problems.
>
> Thanks,
>
> Andrew Miller
>
> 08:12:01 - Node DIX-D-EVGAPP-02 (a disk node) saw node STR-D-EVGAPP-02
> (a memory node) go down (which had been the master node for many
> queues), and was promoted to master for 75 queues, including those which
> later would disappear.
>
> 08:13:30 - Node DIX-D-EVGAPP-02 saw node STR-D-EVGAPP-02 come back, and
> began synchronizing back. Most queues synchronized fine, but 7 crashed
> with errors similar to the following - and each of these queues were
> gone after this. The third node (another disk node) doesn't have any
> crashes in its log around this time.
>
> =ERROR REPORT==== 27-Jul-2014::08:13:30 ===
>
> ** Generic server <0.332.0> terminating
>
> ** Last message in was {add_on_right,{20,<4967.323.0>}}
>
> ** When Server state == {q,
>
> {amqqueue,
>
> {resource,<<"eventing">>,queue,
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
> true,false,none,
>
> [{<<"x-dead-letter-exchange">>,longstr,
>
> <<"EnterpriseEventDeadLetter">>},
>
> {<<"x-dead-letter-routing-key">>,longstr,
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>}],
>
> <0.332.0>,
>
> [<8174.310.0>],
>
> [<8174.310.0>],
>
> [{vhost,<<"eventing">>},
>
> {name,<<"ha-all">>},
>
> {pattern,<<"^.*">>},
>
> {'apply-to',<<"queues">>},
>
> {definition,
>
> [{<<"ha-mode">>,<<"all">>},
>
> {<<"ha-sync-mode">>,<<"automatic">>}]},
>
> {priority,0}],
>
>
> [{<8174.311.0>,<8174.310.0>},{<0.333.0>,<0.332.0>}],
>
> []},
>
> none,false,rabbit_mirror_queue_master,
>
> {state,
>
> {resource,<<"eventing">>,queue,
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
> <0.333.0>,<0.4223.0>,rabbit_variable_queue,
>
> {vqstate,
>
> {0,{[],[]}},
>
> {0,{[],[]}},
>
> {delta,undefined,0,undefined},
>
> {0,{[],[]}},
>
> {0,{[],[]}},
>
> 0,
>
> {0,nil},
>
> {0,nil},
>
> {qistate,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",
>
> {{dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> []},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> []}}},
>
> []},
>
> undefined,0,65536,
>
> #Fun<rabbit_variable_queue.2.51560583>,
>
> {0,nil}},
>
> {{client_msstate,msg_store_persistent,
>
>
> <<26,131,31,121,222,176,41,144,168,192,42,159,222,
>
> 13,179,107>>,
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> []},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> []}}},
>
> {state,311371,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},
>
> rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent",
>
> <0.266.0>,315468,307274,319565,323662},
>
> {client_msstate,msg_store_transient,
>
>
> <<32,49,130,217,86,65,129,214,32,140,24,34,254,
>
> 129,171,73>>,
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> []},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> []}}},
>
> {state,290886,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient"},
>
> rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient",
>
> <0.261.0>,294983,286789,299080,303177}},
>
> true,0,0,0,infinity,0,0,0,0,0,
>
> {rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},
>
> {0,nil},
>
> {0,nil},
>
> {0,nil},
>
> {0,nil},
>
> 0,0},
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> []}}},
>
> [],
>
> {set,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> []}}}},
>
>
> {state,{queue,[],[],0},{active,1406466722448182,1.0}},
>
> undefined,undefined,undefined,undefined,
>
> {state,fine,5000,undefined},
>
> {0,nil},
>
> undefined,undefined,undefined,
>
> {state,
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> []}}},
>
> delegate},
>
> {resource,<<"eventing">>,exchange,
>
> <<"EnterpriseEventDeadLetter">>},
>
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>,undefined,2,
>
> running}
>
> ** Reason for termination ==
>
> ** {function_clause,
>
> [{rabbit_amqqueue_process,handle_call,
>
> [{add_on_right,{20,<4967.323.0>}},
>
> {<4967.323.0>,#Ref<4967.0.0.2348>},
>
> {q,{amqqueue,
>
> {resource,<<"eventing">>,queue,
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
> true,false,none,
>
> [{<<"x-dead-letter-exchange">>,longstr,
>
> <<"EnterpriseEventDeadLetter">>},
>
> {<<"x-dead-letter-routing-key">>,longstr,
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>}],
>
> <0.332.0>,
>
> [<8174.310.0>],
>
> [<8174.310.0>],
>
> [{vhost,<<"eventing">>},
>
> {name,<<"ha-all">>},
>
> {pattern,<<"^.*">>},
>
> {'apply-to',<<"queues">>},
>
> {definition,
>
> [{<<"ha-mode">>,<<"all">>},
>
> {<<"ha-sync-mode">>,<<"automatic">>}]},
>
> {priority,0}],
>
> [{<8174.311.0>,<8174.310.0>},{<0.333.0>,<0.332.0>}],
>
> []},
>
> none,false,rabbit_mirror_queue_master,
>
> {state,
>
> {resource,<<"eventing">>,queue,
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
> <0.333.0>,<0.4223.0>,rabbit_variable_queue,
>
> {vqstate,
>
> {0,{[],[]}},
>
> {0,{[],[]}},
>
> {delta,undefined,0,undefined},
>
> {0,{[],[]}},
>
> {0,{[],[]}},
>
> 0,
>
> {0,nil},
>
> {0,nil},
>
> {qistate,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",
>
> {{dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]}}},
>
> []},
>
> undefined,0,65536,
>
> #Fun<rabbit_variable_queue.2.51560583>,
>
> {0,nil}},
>
> {{client_msstate,msg_store_persistent,
>
>
> <<26,131,31,121,222,176,41,144,168,192,42,159,222,
>
> 13,179,107>>,
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]}}},
>
> {state,311371,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},
>
> rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent",
>
> <0.266.0>,315468,307274,319565,323662},
>
> {client_msstate,msg_store_transient,
>
>
> <<32,49,130,217,86,65,129,214,32,140,24,34,254,
>
> 129,171,73>>,
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]}}},
>
> {state,290886,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient"},
>
> rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient",
>
> <0.261.0>,294983,286789,299080,303177}},
>
> true,0,0,0,infinity,0,0,0,0,0,
>
> {rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},
>
> {0,nil},
>
> {0,nil},
>
> {0,nil},
>
> {0,nil},
>
> 0,0},
>
> {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
> [],
>
> {set,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}},
>
> {state,{queue,[],[],0},{active,1406466722448182,1.0}},
>
> undefined,undefined,undefined,undefined,
>
> {state,fine,5000,undefined},
>
> {0,nil},
>
> undefined,undefined,undefined,
>
> {state,
>
> {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
> delegate},
>
> {resource,<<"eventing">>,exchange,
>
> <<"EnterpriseEventDeadLetter">>},
>
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>,undefined,2,running}],
>
> []},
>
> {gen_server2,handle_msg,2,[]},
>
> {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]}
>
> 08:13:30 - Node STR-D-EVGAPP-02 also recorded crashes involving the same
> queues:
>
> =ERROR REPORT==== 27-Jul-2014::08:13:30 ===
>
> ** Generic server <0.323.0> terminating
>
> ** Last message in was {'$gen_cast',join}
>
> ** When Server state == {state,
>
> {20,<0.323.0>},
>
> {{20,<0.323.0>},undefined},
>
> {{20,<0.323.0>},undefined},
>
> {resource,<<"eventing">>,queue,
>
>
> <<"Insight.BPClaims_ModifiedWorkNotAvailable">>},
>
>
> rabbit_mirror_queue_slave,undefined,-1,undefined,
>
> [<0.322.0>],
>
> {[],[]},
>
> [],0,undefined,
>
> #Fun<rabbit_misc.execute_mnesia_transaction.1>}
>
> ** Reason for termination ==
>
> ** {{function_clause,
>
> [{rabbit_amqqueue_process,handle_call,
>
> [{add_on_right,{20,<0.323.0>}},
>
> {<0.323.0>,#Ref<0.0.0.2348>},
>
> {q,{amqqueue,
>
> {resource,<<"eventing">>,queue,
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
> true,false,none,
>
> [{<<"x-dead-letter-exchange">>,longstr,
>
> <<"EnterpriseEventDeadLetter">>},
>
> {<<"x-dead-letter-routing-key">>,longstr,
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>}],
>
> <5297.332.0>,
>
> [<5251.310.0>],
>
> [<5251.310.0>],
>
> [{vhost,<<"eventing">>},
>
> {name,<<"ha-all">>},
>
> {pattern,<<"^.*">>},
>
> {'apply-to',<<"queues">>},
>
> {definition,
>
> [{<<"ha-mode">>,<<"all">>},
>
> {<<"ha-sync-mode">>,<<"automatic">>}]},
>
> {priority,0}],
>
>
> [{<5251.311.0>,<5251.310.0>},{<5297.333.0>,<5297.332.0>}],
>
> []},
>
> none,false,rabbit_mirror_queue_master,
>
> {state,
>
> {resource,<<"eventing">>,queue,
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
> <5297.333.0>,<5297.4223.0>,rabbit_variable_queue,
>
> {vqstate,
>
> {0,{[],[]}},
>
> {0,{[],[]}},
>
> {delta,undefined,0,undefined},
>
> {0,{[],[]}},
>
> {0,{[],[]}},
>
> 0,
>
> {0,nil},
>
> {0,nil},
>
> {qistate,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",
>
> {{dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]}}},
>
> []},
>
> undefined,0,65536,
>
> #Fun<rabbit_variable_queue.2.51560583>,
>
> {0,nil}},
>
> {{client_msstate,msg_store_persistent,
>
>
> <<26,131,31,121,222,176,41,144,168,192,42,159,
>
> 222,13,179,107>>,
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]}}},
>
> {state,311371,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},
>
> rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent",
>
> <5297.266.0>,315468,307274,319565,323662},
>
> {client_msstate,msg_store_transient,
>
>
> <<32,49,130,217,86,65,129,214,32,140,24,34,254,
>
> 129,171,73>>,
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]}}},
>
> {state,290886,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient"},
>
> rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient",
>
> <5297.261.0>,294983,286789,299080,303177}},
>
> true,0,0,0,infinity,0,0,0,0,0,
>
> {rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},
>
> {0,nil},
>
> {0,nil},
>
> {0,nil},
>
> {0,nil},
>
> 0,0},
>
> {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
> [],
>
> {set,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}},
>
> {state,{queue,[],[],0},{active,1406466722448182,1.0}},
>
> undefined,undefined,undefined,undefined,
>
> {state,fine,5000,undefined},
>
> {0,nil},
>
> undefined,undefined,undefined,
>
> {state,
>
> {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
> delegate},
>
> {resource,<<"eventing">>,exchange,
>
> <<"EnterpriseEventDeadLetter">>},
>
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>,undefined,2,running}],
>
> []},
>
> {gen_server2,handle_msg,2,[]},
>
> {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]},
>
>
> {gen_server2,call,[<5297.332.0>,{add_on_right,{20,<0.323.0>}},infinity]}}
>
> 08:13:31 - And again from node STR-D-EVGAPP-02:
>
> =ERROR REPORT==== 27-Jul-2014::08:13:31 ===
>
> ** Generic server <0.322.0> terminating
>
> ** Last message in was {'EXIT',<0.323.0>,
>
> {{function_clause,
>
> [{rabbit_amqqueue_process,handle_call,
>
> [{add_on_right,{20,<0.323.0>}},
>
> {<0.323.0>,#Ref<0.0.0.2348>},
>
> {q,
>
> {amqqueue,
>
> {resource,<<"eventing">>,queue,
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
> true,false,none,
>
> [{<<"x-dead-letter-exchange">>,longstr,
>
> <<"EnterpriseEventDeadLetter">>},
>
> {<<"x-dead-letter-routing-key">>,longstr,
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>}],
>
> <5297.332.0>,
>
> [<5251.310.0>],
>
> [<5251.310.0>],
>
> [{vhost,<<"eventing">>},
>
> {name,<<"ha-all">>},
>
> {pattern,<<"^.*">>},
>
> {'apply-to',<<"queues">>},
>
> {definition,
>
> [{<<"ha-mode">>,<<"all">>},
>
> {<<"ha-sync-mode">>,<<"automatic">>}]},
>
> {priority,0}],
>
> [{<5251.311.0>,<5251.310.0>},
>
> {<5297.333.0>,<5297.332.0>}],
>
> []},
>
> none,false,rabbit_mirror_queue_master,
>
> {state,
>
> {resource,<<"eventing">>,queue,
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
> <5297.333.0>,<5297.4223.0>,
>
> rabbit_variable_queue,
>
> {vqstate,
>
> {0,{[],[]}},
>
> {0,{[],[]}},
>
> {delta,undefined,0,undefined},
>
> {0,{[],[]}},
>
> {0,{[],[]}},
>
> 0,
>
> {0,nil},
>
> {0,nil},
>
> {qistate,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",
>
> {{dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[],[]}}},
>
> []},
>
> undefined,0,65536,
>
> #Fun<rabbit_variable_queue.2.51560583>,
>
> {0,nil}},
>
> {{client_msstate,msg_store_persistent,
>
>
> <<26,131,31,121,222,176,41,144,168,192,42,
>
> 159,222,13,179,107>>,
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[],[]}}},
>
> {state,311371,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},
>
> rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent",
>
>
> <5297.266.0>,315468,307274,319565,323662},
>
> {client_msstate,msg_store_transient,
>
>
> <<32,49,130,217,86,65,129,214,32,140,24,34,
>
> 254,129,171,73>>,
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[],[]}}},
>
> {state,290886,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient"},
>
> rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient",
>
>
> <5297.261.0>,294983,286789,299080,303177}},
>
> true,0,0,0,infinity,0,0,0,0,0,
>
>
> {rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},
>
> {0,nil},
>
> {0,nil},
>
> {0,nil},
>
> {0,nil},
>
> 0,0},
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> []},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]}}},
>
> [],
>
> {set,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> []},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]}}}},
>
> {state,
>
> {queue,[],[],0},
>
> {active,1406466722448182,1.0}},
>
> undefined,undefined,undefined,undefined,
>
> {state,fine,5000,undefined},
>
> {0,nil},
>
> undefined,undefined,undefined,
>
> {state,
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> []},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]}}},
>
> delegate},
>
> {resource,<<"eventing">>,exchange,
>
> <<"EnterpriseEventDeadLetter">>},
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>,
>
> undefined,2,running}],
>
> []},
>
> {gen_server2,handle_msg,2,[]},
>
> {proc_lib,wake_up,3,
>
> [{file,"proc_lib.erl"},{line,249}]}]},
>
> {gen_server2,call,
>
> [<5297.332.0>,
>
> {add_on_right,{20,<0.323.0>}},
>
> infinity]}}}
>
> ** When Server state == {state,
>
> {amqqueue,
>
> {resource,<<"eventing">>,queue,
>
>
> <<"Insight.BPClaims_ModifiedWorkNotAvailable">>},
>
> true,false,none,
>
> [{<<"x-dead-letter-exchange">>,longstr,
>
> <<"EnterpriseEventDeadLetter">>},
>
> {<<"x-dead-letter-routing-key">>,longstr,
>
>
> <<"Insight.BPClaims_ModifiedWorkNotAvailable">>}],
>
> <5251.335.0>,
>
> [<5297.331.0>],
>
> [<5297.331.0>],
>
> [{vhost,<<"eventing">>},
>
> {name,<<"ha-all">>},
>
> {pattern,<<"^.*">>},
>
> {'apply-to',<<"queues">>},
>
> {definition,
>
> [{<<"ha-mode">>,<<"all">>},
>
> {<<"ha-sync-mode">>,<<"automatic">>}]},
>
> {priority,0}],
>
> [{<5297.332.0>,<5297.331.0>},
>
> {<5251.336.0>,<5251.335.0>}],
>
> []},
>
> <0.323.0>,rabbit_variable_queue,
>
> {vqstate,
>
> {0,{[],[]}},
>
> {0,{[],[]}},
>
> {delta,undefined,0,undefined},
>
> {0,{[],[]}},
>
> {0,{[],[]}},
>
> 0,
>
> {0,nil},
>
> {0,nil},
>
> {qistate,
>
>
> "d:/RabbitMQ/db/rabbit@STR-D-EVGAPP-02-mnesia/queues/11YNGR2UMDIGSF4OWRUUJ2HUX",
>
> {{dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> []}}},
>
> []},
>
> undefined,0,65536,
>
> #Fun<rabbit_variable_queue.2.51560583>,
>
> {0,nil}},
>
> {{client_msstate,msg_store_persistent,
>
>
> <<142,49,10,175,39,218,164,74,2,215,253,147,201,
>
> 168,141,2>>,
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> []}}},
>
> {state,303177,
>
>
> "d:/RabbitMQ/db/rabbit@STR-D-EVGAPP-02-mnesia/msg_store_persistent"},
>
> rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit@STR-D-EVGAPP-02-mnesia/msg_store_persistent",
>
> <0.251.0>,307274,299080,311371,315468},
>
> {client_msstate,msg_store_transient,
>
>
> <<18,152,170,252,55,149,243,37,252,234,131,180,53,
>
> 66,14,182>>,
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> []}}},
>
> {state,282692,
>
>
> "d:/RabbitMQ/db/rabbit@STR-D-EVGAPP-02-mnesia/msg_store_transient"},
>
> rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit@STR-D-EVGAPP-02-mnesia/msg_store_transient",
>
> <0.246.0>,286789,278595,290886,294983}},
>
> true,0,0,0,infinity,0,0,0,0,0,
>
> {rates,0.0,0.0,0.0,0.0,{1406,466811,43211}},
>
> {0,nil},
>
> {0,nil},
>
> {0,nil},
>
> {0,nil},
>
> 0,0},
>
> undefined,undefined,
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
> {state,
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> []}}},
>
> delegate},
>
> undefined}
>
> ** Reason for termination ==
>
> ** {{function_clause,
>
> [{rabbit_amqqueue_process,handle_call,
>
> [{add_on_right,{20,<0.323.0>}},
>
> {<0.323.0>,#Ref<0.0.0.2348>},
>
> {q,{amqqueue,
>
> {resource,<<"eventing">>,queue,
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
> true,false,none,
>
> [{<<"x-dead-letter-exchange">>,longstr,
>
> <<"EnterpriseEventDeadLetter">>},
>
> {<<"x-dead-letter-routing-key">>,longstr,
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>}],
>
> <5297.332.0>,
>
> [<5251.310.0>],
>
> [<5251.310.0>],
>
> [{vhost,<<"eventing">>},
>
> {name,<<"ha-all">>},
>
> {pattern,<<"^.*">>},
>
> {'apply-to',<<"queues">>},
>
> {definition,
>
> [{<<"ha-mode">>,<<"all">>},
>
> {<<"ha-sync-mode">>,<<"automatic">>}]},
>
> {priority,0}],
>
>
> [{<5251.311.0>,<5251.310.0>},{<5297.333.0>,<5297.332.0>}],
>
> []},
>
> none,false,rabbit_mirror_queue_master,
>
> {state,
>
> {resource,<<"eventing">>,queue,
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
> <5297.333.0>,<5297.4223.0>,rabbit_variable_queue,
>
> {vqstate,
>
> {0,{[],[]}},
>
> {0,{[],[]}},
>
> {delta,undefined,0,undefined},
>
> {0,{[],[]}},
>
> {0,{[],[]}},
>
> 0,
>
> {0,nil},
>
> {0,nil},
>
> {qistate,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",
>
> {{dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]}}},
>
> []},
>
> undefined,0,65536,
>
> #Fun<rabbit_variable_queue.2.51560583>,
>
> {0,nil}},
>
> {{client_msstate,msg_store_persistent,
>
>
> <<26,131,31,121,222,176,41,144,168,192,42,159,
>
> 222,13,179,107>>,
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]}}},
>
> {state,311371,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},
>
> rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent",
>
> <5297.266.0>,315468,307274,319565,323662},
>
> {client_msstate,msg_store_transient,
>
>
> <<32,49,130,217,86,65,129,214,32,140,24,34,254,
>
> 129,171,73>>,
>
> {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]}}},
>
> {state,290886,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient"},
>
> rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient",
>
> <5297.261.0>,294983,286789,299080,303177}},
>
> true,0,0,0,infinity,0,0,0,0,0,
>
> {rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},
>
> {0,nil},
>
> {0,nil},
>
> {0,nil},
>
> {0,nil},
>
> 0,0},
>
> {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
> [],
>
> {set,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}},
>
> {state,{queue,[],[],0},{active,1406466722448182,1.0}},
>
> undefined,undefined,undefined,undefined,
>
> {state,fine,5000,undefined},
>
> {0,nil},
>
> undefined,undefined,undefined,
>
> {state,
>
> {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
> delegate},
>
> {resource,<<"eventing">>,exchange,
>
> <<"EnterpriseEventDeadLetter">>},
>
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>,undefined,2,running}],
>
> []},
>
> {gen_server2,handle_msg,2,[]},
>
> {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]},
>
>
> {gen_server2,call,[<5297.332.0>,{add_on_right,{20,<0.323.0>}},infinity]}}
>
> This e-mail is confidential. If you are not the intended recipient, you
> must not disclose or use the information contained in it. If you have
> received this e-mail in error, please tell us immediately by return
> e-mail and delete the document. No recipient may use the information in
> this e-mail in violation of any civil or criminal statute. Sentry
> disclaims all liability for any unauthorized uses of this e-mail or its
> contents. Sentry accepts no liability or responsibility for any damage
> caused by any virus transmitted with this e-mail.
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
> please subscribe to the new list!
>
> [hidden email]
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>