Quantcast

Loss of durable queues on HA sync

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Loss of durable queues on HA sync

Miller Andrew

During some server reboots at our organization over the weekend, several nodes in our non-Prod cluster had some crashes, and in the end, 7 of 84 durable queues in the cluster were gone.  All nodes are running RabbitMQ 3.3.3 on Erlang R16B03-1.  This is a 3-node cluster, with cluster_partition_handling = autoheal.  I'm curious if this is a known issue with autoheal?  I tried recreating the issue this morning by recreating the series of events, but was unable to.  Our other 2-node non-Prod cluster, with no cluster_partition_handling, was also rebootted, but didn't have any problems.  

 

Thanks,

Andrew Miller

 

08:12:01 - Node DIX-D-EVGAPP-02 (a disk node) saw node STR-D-EVGAPP-02 (a memory node) go down (which had been the master node for many queues), and was promoted to master for 75 queues, including those which later would disappear.

 

08:13:30 - Node DIX-D-EVGAPP-02 saw node STR-D-EVGAPP-02 come back, and began synchronizing back.  Most queues synchronized fine, but 7 crashed with errors similar to the following - and each of these queues were gone after this.  The third node (another disk node) doesn't have any crashes in its log around this time.

 

=ERROR REPORT==== 27-Jul-2014::08:13:30 ===

** Generic server <0.332.0> terminating

** Last message in was {add_on_right,{20,<4967.323.0>}}

** When Server state == {q,

                         {amqqueue,

                          {resource,<<"eventing">>,queue,

                           <<"SalesCenter2.BPClaims_ClosedClaim">>},

                          true,false,none,

                          [{<<"x-dead-letter-exchange">>,longstr,

                            <<"EnterpriseEventDeadLetter">>},

                           {<<"x-dead-letter-routing-key">>,longstr,

                            <<"SalesCenter2.BPClaims_ClosedClaim">>}],

                          <0.332.0>,

                          [<8174.310.0>],

                          [<8174.310.0>],

                          [{vhost,<<"eventing">>},

                           {name,<<"ha-all">>},

                           {pattern,<<"^.*">>},

                           {'apply-to',<<"queues">>},

                           {definition,

                            [{<<"ha-mode">>,<<"all">>},

                             {<<"ha-sync-mode">>,<<"automatic">>}]},

                           {priority,0}],

                          [{<8174.311.0>,<8174.310.0>},{<0.333.0>,<0.332.0>}],

                          []},

                         none,false,rabbit_mirror_queue_master,

                         {state,

                          {resource,<<"eventing">>,queue,

                           <<"SalesCenter2.BPClaims_ClosedClaim">>},

                          <0.333.0>,<0.4223.0>,rabbit_variable_queue,

                          {vqstate,

                           {0,{[],[]}},

                           {0,{[],[]}},

                           {delta,undefined,0,undefined},

                           {0,{[],[]}},

                           {0,{[],[]}},

                           0,

                           {0,nil},

                           {0,nil},

                           {qistate,

                            "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",

                            {{dict,0,16,16,8,80,48,

                              {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                               []},

                              {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                []}}},

                             []},

                            undefined,0,65536,

                            #Fun<rabbit_variable_queue.2.51560583>,

                            {0,nil}},

                           {{client_msstate,msg_store_persistent,

                             <<26,131,31,121,222,176,41,144,168,192,42,159,222,

                               13,179,107>>,

                             {dict,0,16,16,8,80,48,

                              {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                               []},

                              {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                []}}},

                             {state,311371,

                              "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},

                             rabbit_msg_store_ets_index,

                             "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent",

                             <0.266.0>,315468,307274,319565,323662},

                            {client_msstate,msg_store_transient,

                             <<32,49,130,217,86,65,129,214,32,140,24,34,254,

                               129,171,73>>,

                             {dict,0,16,16,8,80,48,

                              {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                               []},

                              {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                []}}},

                             {state,290886,

                              "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient"},

                             rabbit_msg_store_ets_index,

                             "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient",

                             <0.261.0>,294983,286789,299080,303177}},

                           true,0,0,0,infinity,0,0,0,0,0,

                           {rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},

                           {0,nil},

                           {0,nil},

                           {0,nil},

                           {0,nil},

                           0,0},

                          {dict,0,16,16,8,80,48,

                           {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

                           {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                             []}}},

                          [],

                          {set,0,16,16,8,80,48,

                           {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

                           {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                             []}}}},

                         {state,{queue,[],[],0},{active,1406466722448182,1.0}},

                         undefined,undefined,undefined,undefined,

                         {state,fine,5000,undefined},

                         {0,nil},

                         undefined,undefined,undefined,

                         {state,

                          {dict,0,16,16,8,80,48,

                           {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

                           {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                             []}}},

                          delegate},

                         {resource,<<"eventing">>,exchange,

                          <<"EnterpriseEventDeadLetter">>},

                         <<"SalesCenter2.BPClaims_ClosedClaim">>,undefined,2,

                         running}

** Reason for termination ==

** {function_clause,

       [{rabbit_amqqueue_process,handle_call,

            [{add_on_right,{20,<4967.323.0>}},

             {<4967.323.0>,#Ref<4967.0.0.2348>},

             {q,{amqqueue,

                    {resource,<<"eventing">>,queue,

                        <<"SalesCenter2.BPClaims_ClosedClaim">>},

                    true,false,none,

                    [{<<"x-dead-letter-exchange">>,longstr,

                      <<"EnterpriseEventDeadLetter">>},

                     {<<"x-dead-letter-routing-key">>,longstr,

                      <<"SalesCenter2.BPClaims_ClosedClaim">>}],

                    <0.332.0>,

                    [<8174.310.0>],

                    [<8174.310.0>],

                    [{vhost,<<"eventing">>},

                     {name,<<"ha-all">>},

                     {pattern,<<"^.*">>},

                     {'apply-to',<<"queues">>},

                     {definition,

                         [{<<"ha-mode">>,<<"all">>},

                          {<<"ha-sync-mode">>,<<"automatic">>}]},

                     {priority,0}],

                    [{<8174.311.0>,<8174.310.0>},{<0.333.0>,<0.332.0>}],

                    []},

                none,false,rabbit_mirror_queue_master,

                {state,

                    {resource,<<"eventing">>,queue,

                        <<"SalesCenter2.BPClaims_ClosedClaim">>},

                    <0.333.0>,<0.4223.0>,rabbit_variable_queue,

                    {vqstate,

                        {0,{[],[]}},

                        {0,{[],[]}},

                        {delta,undefined,0,undefined},

                        {0,{[],[]}},

                        {0,{[],[]}},

                        0,

                        {0,nil},

                        {0,nil},

                        {qistate,

                            "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",

                            {{dict,0,16,16,8,80,48,

                                 {[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                  [],[]},

                                 {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                   [],[]}}},

                             []},

                            undefined,0,65536,

                            #Fun<rabbit_variable_queue.2.51560583>,

                            {0,nil}},

                        {{client_msstate,msg_store_persistent,

                             <<26,131,31,121,222,176,41,144,168,192,42,159,222,

                               13,179,107>>,

                             {dict,0,16,16,8,80,48,

                                 {[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                  [],[]},

                                 {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                   [],[]}}},

                             {state,311371,

                                 "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},

                             rabbit_msg_store_ets_index,

                             "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent",

                             <0.266.0>,315468,307274,319565,323662},

                         {client_msstate,msg_store_transient,

                             <<32,49,130,217,86,65,129,214,32,140,24,34,254,

                               129,171,73>>,

                             {dict,0,16,16,8,80,48,

                                 {[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                  [],[]},

                                 {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                   [],[]}}},

                             {state,290886,

                                 "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient"},

                             rabbit_msg_store_ets_index,

                             "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient",

                             <0.261.0>,294983,286789,299080,303177}},

                        true,0,0,0,infinity,0,0,0,0,0,

                        {rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},

                        {0,nil},

                        {0,nil},

                        {0,nil},

                        {0,nil},

                        0,0},

                    {dict,0,16,16,8,80,48,

                        {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

                        {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},

                    [],

                    {set,0,16,16,8,80,48,

                        {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

                        {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}},

                {state,{queue,[],[],0},{active,1406466722448182,1.0}},

                undefined,undefined,undefined,undefined,

                {state,fine,5000,undefined},

                {0,nil},

                undefined,undefined,undefined,

                {state,

                    {dict,0,16,16,8,80,48,

                        {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

                        {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},

                    delegate},

                {resource,<<"eventing">>,exchange,

                    <<"EnterpriseEventDeadLetter">>},

                <<"SalesCenter2.BPClaims_ClosedClaim">>,undefined,2,running}],

            []},

        {gen_server2,handle_msg,2,[]},

        {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]}

 

 

08:13:30 - Node STR-D-EVGAPP-02 also recorded crashes involving the same queues:

 

=ERROR REPORT==== 27-Jul-2014::08:13:30 ===

** Generic server <0.323.0> terminating

** Last message in was {'$gen_cast',join}

** When Server state == {state,

                            {20,<0.323.0>},

                            {{20,<0.323.0>},undefined},

                            {{20,<0.323.0>},undefined},

                            {resource,<<"eventing">>,queue,

                                <<"Insight.BPClaims_ModifiedWorkNotAvailable">>},

                            rabbit_mirror_queue_slave,undefined,-1,undefined,

                            [<0.322.0>],

                            {[],[]},

                            [],0,undefined,

                            #Fun<rabbit_misc.execute_mnesia_transaction.1>}

** Reason for termination ==

** {{function_clause,

        [{rabbit_amqqueue_process,handle_call,

             [{add_on_right,{20,<0.323.0>}},

              {<0.323.0>,#Ref<0.0.0.2348>},

              {q,{amqqueue,

                     {resource,<<"eventing">>,queue,

                         <<"SalesCenter2.BPClaims_ClosedClaim">>},

                     true,false,none,

                     [{<<"x-dead-letter-exchange">>,longstr,

                       <<"EnterpriseEventDeadLetter">>},

                      {<<"x-dead-letter-routing-key">>,longstr,

                       <<"SalesCenter2.BPClaims_ClosedClaim">>}],

                     <5297.332.0>,

                     [<5251.310.0>],

                     [<5251.310.0>],

                     [{vhost,<<"eventing">>},

                      {name,<<"ha-all">>},

                      {pattern,<<"^.*">>},

                      {'apply-to',<<"queues">>},

                      {definition,

                          [{<<"ha-mode">>,<<"all">>},

                           {<<"ha-sync-mode">>,<<"automatic">>}]},

                      {priority,0}],

                     [{<5251.311.0>,<5251.310.0>},{<5297.333.0>,<5297.332.0>}],

                     []},

                 none,false,rabbit_mirror_queue_master,

                 {state,

                     {resource,<<"eventing">>,queue,

                         <<"SalesCenter2.BPClaims_ClosedClaim">>},

                     <5297.333.0>,<5297.4223.0>,rabbit_variable_queue,

                     {vqstate,

                         {0,{[],[]}},

                         {0,{[],[]}},

                         {delta,undefined,0,undefined},

                         {0,{[],[]}},

                         {0,{[],[]}},

                         0,

                         {0,nil},

                         {0,nil},

                         {qistate,

                             "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",

                             {{dict,0,16,16,8,80,48,

                                  {[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                   [],[]},

                                  {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                    [],[]}}},

                              []},

                             undefined,0,65536,

                             #Fun<rabbit_variable_queue.2.51560583>,

                             {0,nil}},

                         {{client_msstate,msg_store_persistent,

                              <<26,131,31,121,222,176,41,144,168,192,42,159,

                                222,13,179,107>>,

                              {dict,0,16,16,8,80,48,

                                  {[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                   [],[]},

                                  {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                    [],[]}}},

                              {state,311371,

                                  "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},

                              rabbit_msg_store_ets_index,

                              "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent",

                              <5297.266.0>,315468,307274,319565,323662},

                          {client_msstate,msg_store_transient,

                              <<32,49,130,217,86,65,129,214,32,140,24,34,254,

                                129,171,73>>,

                              {dict,0,16,16,8,80,48,

                                  {[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                   [],[]},

                                  {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                    [],[]}}},

                              {state,290886,

                                  "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient"},

                              rabbit_msg_store_ets_index,

                              "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient",

                              <5297.261.0>,294983,286789,299080,303177}},

                         true,0,0,0,infinity,0,0,0,0,0,

                         {rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},

                         {0,nil},

                         {0,nil},

                         {0,nil},

                         {0,nil},

                         0,0},

                     {dict,0,16,16,8,80,48,

                         {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

                         {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},

                     [],

                     {set,0,16,16,8,80,48,

                         {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

                         {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}},

                 {state,{queue,[],[],0},{active,1406466722448182,1.0}},

                 undefined,undefined,undefined,undefined,

                 {state,fine,5000,undefined},

                 {0,nil},

                 undefined,undefined,undefined,

                 {state,

                     {dict,0,16,16,8,80,48,

                         {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

                         {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},

                     delegate},

                 {resource,<<"eventing">>,exchange,

                     <<"EnterpriseEventDeadLetter">>},

                 <<"SalesCenter2.BPClaims_ClosedClaim">>,undefined,2,running}],

             []},

         {gen_server2,handle_msg,2,[]},

         {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]},

    {gen_server2,call,[<5297.332.0>,{add_on_right,{20,<0.323.0>}},infinity]}}

 

08:13:31 - And again from node STR-D-EVGAPP-02:

 

=ERROR REPORT==== 27-Jul-2014::08:13:31 ===

** Generic server <0.322.0> terminating

** Last message in was {'EXIT',<0.323.0>,

                        {{function_clause,

                          [{rabbit_amqqueue_process,handle_call,

                            [{add_on_right,{20,<0.323.0>}},

                             {<0.323.0>,#Ref<0.0.0.2348>},

                             {q,

                              {amqqueue,

                               {resource,<<"eventing">>,queue,

                                <<"SalesCenter2.BPClaims_ClosedClaim">>},

                               true,false,none,

                               [{<<"x-dead-letter-exchange">>,longstr,

                                 <<"EnterpriseEventDeadLetter">>},

                                {<<"x-dead-letter-routing-key">>,longstr,

                                 <<"SalesCenter2.BPClaims_ClosedClaim">>}],

                               <5297.332.0>,

                               [<5251.310.0>],

                               [<5251.310.0>],

                               [{vhost,<<"eventing">>},

                                {name,<<"ha-all">>},

                                {pattern,<<"^.*">>},

                                {'apply-to',<<"queues">>},

                                {definition,

                                 [{<<"ha-mode">>,<<"all">>},

                                  {<<"ha-sync-mode">>,<<"automatic">>}]},

                                {priority,0}],

                               [{<5251.311.0>,<5251.310.0>},

                                {<5297.333.0>,<5297.332.0>}],

                               []},

                              none,false,rabbit_mirror_queue_master,

                              {state,

                               {resource,<<"eventing">>,queue,

                                <<"SalesCenter2.BPClaims_ClosedClaim">>},

                               <5297.333.0>,<5297.4223.0>,

                               rabbit_variable_queue,

                               {vqstate,

                                {0,{[],[]}},

                                {0,{[],[]}},

                                {delta,undefined,0,undefined},

                                {0,{[],[]}},

                                {0,{[],[]}},

                                0,

                                {0,nil},

                                {0,nil},

                                {qistate,

                                 "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",

                                 {{dict,0,16,16,8,80,48,

                                   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                    [],[]},

                                   {{[],[],[],[],[],[],[],[],[],[],[],[],[],

                                     [],[],[]}}},

                                  []},

                                 undefined,0,65536,

                                 #Fun<rabbit_variable_queue.2.51560583>,

                                 {0,nil}},

                                {{client_msstate,msg_store_persistent,

                                  <<26,131,31,121,222,176,41,144,168,192,42,

                                    159,222,13,179,107>>,

                                  {dict,0,16,16,8,80,48,

                                   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                    [],[]},

                                   {{[],[],[],[],[],[],[],[],[],[],[],[],[],

                                     [],[],[]}}},

                                  {state,311371,

                                   "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},

                                  rabbit_msg_store_ets_index,

                                  "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent",

                                  <5297.266.0>,315468,307274,319565,323662},

                                 {client_msstate,msg_store_transient,

                                  <<32,49,130,217,86,65,129,214,32,140,24,34,

                                    254,129,171,73>>,

                                  {dict,0,16,16,8,80,48,

                                   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                    [],[]},

                                   {{[],[],[],[],[],[],[],[],[],[],[],[],[],

                                     [],[],[]}}},

                                  {state,290886,

                                   "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient"},

                                  rabbit_msg_store_ets_index,

                                  "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient",

                                  <5297.261.0>,294983,286789,299080,303177}},

                                true,0,0,0,infinity,0,0,0,0,0,

                                {rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},

                                {0,nil},

                                {0,nil},

                                {0,nil},

                                {0,nil},

                                0,0},

                               {dict,0,16,16,8,80,48,

                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                 []},

                                {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                  [],[]}}},

                               [],

                               {set,0,16,16,8,80,48,

                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                 []},

                                {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                  [],[]}}}},

                              {state,

                               {queue,[],[],0},

                               {active,1406466722448182,1.0}},

                              undefined,undefined,undefined,undefined,

                              {state,fine,5000,undefined},

                              {0,nil},

                              undefined,undefined,undefined,

                              {state,

                               {dict,0,16,16,8,80,48,

                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                 []},

                                {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                  [],[]}}},

                               delegate},

                              {resource,<<"eventing">>,exchange,

                               <<"EnterpriseEventDeadLetter">>},

                              <<"SalesCenter2.BPClaims_ClosedClaim">>,

                              undefined,2,running}],

                            []},

                           {gen_server2,handle_msg,2,[]},

                           {proc_lib,wake_up,3,

                            [{file,"proc_lib.erl"},{line,249}]}]},

                         {gen_server2,call,

                          [<5297.332.0>,

                           {add_on_right,{20,<0.323.0>}},

                           infinity]}}}

** When Server state == {state,

                         {amqqueue,

                          {resource,<<"eventing">>,queue,

                           <<"Insight.BPClaims_ModifiedWorkNotAvailable">>},

                          true,false,none,

                          [{<<"x-dead-letter-exchange">>,longstr,

                            <<"EnterpriseEventDeadLetter">>},

                           {<<"x-dead-letter-routing-key">>,longstr,

                            <<"Insight.BPClaims_ModifiedWorkNotAvailable">>}],

                          <5251.335.0>,

                          [<5297.331.0>],

                          [<5297.331.0>],

                          [{vhost,<<"eventing">>},

                           {name,<<"ha-all">>},

                           {pattern,<<"^.*">>},

                           {'apply-to',<<"queues">>},

                           {definition,

                            [{<<"ha-mode">>,<<"all">>},

                             {<<"ha-sync-mode">>,<<"automatic">>}]},

                           {priority,0}],

                          [{<5297.332.0>,<5297.331.0>},

                           {<5251.336.0>,<5251.335.0>}],

                          []},

                         <0.323.0>,rabbit_variable_queue,

                         {vqstate,

                          {0,{[],[]}},

                          {0,{[],[]}},

                          {delta,undefined,0,undefined},

                          {0,{[],[]}},

                          {0,{[],[]}},

                          0,

                          {0,nil},

                          {0,nil},

                          {qistate,

                           "d:/RabbitMQ/db/rabbit@STR-D-EVGAPP-02-mnesia/queues/11YNGR2UMDIGSF4OWRUUJ2HUX",

                           {{dict,0,16,16,8,80,48,

                             {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                               []}}},

                            []},

                           undefined,0,65536,

                           #Fun<rabbit_variable_queue.2.51560583>,

                           {0,nil}},

                          {{client_msstate,msg_store_persistent,

                            <<142,49,10,175,39,218,164,74,2,215,253,147,201,

                              168,141,2>>,

                            {dict,0,16,16,8,80,48,

                             {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                               []}}},

                            {state,303177,

                             "d:/RabbitMQ/db/rabbit@STR-D-EVGAPP-02-mnesia/msg_store_persistent"},

                            rabbit_msg_store_ets_index,

                            "d:/RabbitMQ/db/rabbit@STR-D-EVGAPP-02-mnesia/msg_store_persistent",

                            <0.251.0>,307274,299080,311371,315468},

                           {client_msstate,msg_store_transient,

                            <<18,152,170,252,55,149,243,37,252,234,131,180,53,

                              66,14,182>>,

                            {dict,0,16,16,8,80,48,

                             {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                               []}}},

                            {state,282692,

                             "d:/RabbitMQ/db/rabbit@STR-D-EVGAPP-02-mnesia/msg_store_transient"},

                            rabbit_msg_store_ets_index,

                            "d:/RabbitMQ/db/rabbit@STR-D-EVGAPP-02-mnesia/msg_store_transient",

                            <0.246.0>,286789,278595,290886,294983}},

                          true,0,0,0,infinity,0,0,0,0,0,

                          {rates,0.0,0.0,0.0,0.0,{1406,466811,43211}},

                          {0,nil},

                          {0,nil},

                          {0,nil},

                          {0,nil},

                          0,0},

                         undefined,undefined,

                         {dict,0,16,16,8,80,48,

                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},

                         {dict,0,16,16,8,80,48,

                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},

                         {dict,0,16,16,8,80,48,

                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},

                         {state,

                          {dict,0,16,16,8,80,48,

                           {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

                           {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                             []}}},

                          delegate},

                         undefined}

** Reason for termination ==

** {{function_clause,

        [{rabbit_amqqueue_process,handle_call,

             [{add_on_right,{20,<0.323.0>}},

              {<0.323.0>,#Ref<0.0.0.2348>},

              {q,{amqqueue,

                     {resource,<<"eventing">>,queue,

                         <<"SalesCenter2.BPClaims_ClosedClaim">>},

                     true,false,none,

                     [{<<"x-dead-letter-exchange">>,longstr,

                       <<"EnterpriseEventDeadLetter">>},

                      {<<"x-dead-letter-routing-key">>,longstr,

                       <<"SalesCenter2.BPClaims_ClosedClaim">>}],

                     <5297.332.0>,

                     [<5251.310.0>],

                     [<5251.310.0>],

                     [{vhost,<<"eventing">>},

                      {name,<<"ha-all">>},

                      {pattern,<<"^.*">>},

                      {'apply-to',<<"queues">>},

                      {definition,

                          [{<<"ha-mode">>,<<"all">>},

                           {<<"ha-sync-mode">>,<<"automatic">>}]},

                      {priority,0}],

                     [{<5251.311.0>,<5251.310.0>},{<5297.333.0>,<5297.332.0>}],

                     []},

                 none,false,rabbit_mirror_queue_master,

                 {state,

                     {resource,<<"eventing">>,queue,

                         <<"SalesCenter2.BPClaims_ClosedClaim">>},

                     <5297.333.0>,<5297.4223.0>,rabbit_variable_queue,

                     {vqstate,

                         {0,{[],[]}},

                         {0,{[],[]}},

                         {delta,undefined,0,undefined},

                         {0,{[],[]}},

                         {0,{[],[]}},

                         0,

                         {0,nil},

                         {0,nil},

                         {qistate,

                             "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",

                             {{dict,0,16,16,8,80,48,

                                  {[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                   [],[]},

                                  {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                    [],[]}}},

                              []},

                             undefined,0,65536,

                             #Fun<rabbit_variable_queue.2.51560583>,

                             {0,nil}},

                         {{client_msstate,msg_store_persistent,

                              <<26,131,31,121,222,176,41,144,168,192,42,159,

                                222,13,179,107>>,

                              {dict,0,16,16,8,80,48,

                                  {[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                   [],[]},

                                  {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                    [],[]}}},

                              {state,311371,

                                  "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},

                              rabbit_msg_store_ets_index,

                              "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent",

                              <5297.266.0>,315468,307274,319565,323662},

                          {client_msstate,msg_store_transient,

                              <<32,49,130,217,86,65,129,214,32,140,24,34,254,

                                129,171,73>>,

                              {dict,0,16,16,8,80,48,

                                  {[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                   [],[]},

                                  {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                    [],[]}}},

                              {state,290886,

                                  "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient"},

                              rabbit_msg_store_ets_index,

                              "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient",

                              <5297.261.0>,294983,286789,299080,303177}},

                         true,0,0,0,infinity,0,0,0,0,0,

                         {rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},

                         {0,nil},

                         {0,nil},

                         {0,nil},

                         {0,nil},

                         0,0},

                     {dict,0,16,16,8,80,48,

                         {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

                         {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},

                     [],

                     {set,0,16,16,8,80,48,

                         {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

                         {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}},

                 {state,{queue,[],[],0},{active,1406466722448182,1.0}},

                 undefined,undefined,undefined,undefined,

                 {state,fine,5000,undefined},

                 {0,nil},

                 undefined,undefined,undefined,

                 {state,

                     {dict,0,16,16,8,80,48,

                         {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

                         {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},

                     delegate},

                 {resource,<<"eventing">>,exchange,

                     <<"EnterpriseEventDeadLetter">>},

                 <<"SalesCenter2.BPClaims_ClosedClaim">>,undefined,2,running}],

             []},

         {gen_server2,handle_msg,2,[]},

         {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]},

    {gen_server2,call,[<5297.332.0>,{add_on_right,{20,<0.323.0>}},infinity]}}

This e-mail is confidential.  If you are not the intended recipient, you must not disclose or use the information contained in it.  If you have received this e-mail in error, please tell us immediately by return e-mail and delete the document. No recipient may use the information in this e-mail in violation of any civil or criminal statute. Sentry disclaims all liability for any unauthorized uses of this e-mail or its contents. Sentry accepts no liability or responsibility for any damage caused by any virus transmitted with this e-mail.


_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Loss of durable queues on HA sync

Simon MacMullen-2
Thanks for this. This is not a known issue, and not something we've seen
before.

In all cases it looks like a queue process crashed because it received a
message intended for a GM process. That should never be able to happen.

Could you send me full logs from all nodes around the time of the crash?
It would be useful to see in what order things happened, if anything led
up to this, when nodes restarted for autoheal (if any autoheal
happened), etc.

Cheers, Simon

On 28/07/14 17:40, Miller Andrew wrote:

> During some server reboots at our organization over the weekend, several
> nodes in our non-Prod cluster had some crashes, and in the end, 7 of 84
> durable queues in the cluster were gone.  All nodes are running RabbitMQ
> 3.3.3 on Erlang R16B03-1.  This is a 3-node cluster, with
> cluster_partition_handling = autoheal.  I'm curious if this is a known
> issue with autoheal?  I tried recreating the issue this morning by
> recreating the series of events, but was unable to.  Our other 2-node
> non-Prod cluster, with no cluster_partition_handling, was also
> rebootted, but didn't have any problems.
>
> Thanks,
>
> Andrew Miller
>
> 08:12:01 - Node DIX-D-EVGAPP-02 (a disk node) saw node STR-D-EVGAPP-02
> (a memory node) go down (which had been the master node for many
> queues), and was promoted to master for 75 queues, including those which
> later would disappear.
>
> 08:13:30 - Node DIX-D-EVGAPP-02 saw node STR-D-EVGAPP-02 come back, and
> began synchronizing back.  Most queues synchronized fine, but 7 crashed
> with errors similar to the following - and each of these queues were
> gone after this.  The third node (another disk node) doesn't have any
> crashes in its log around this time.
>
> =ERROR REPORT==== 27-Jul-2014::08:13:30 ===
>
> ** Generic server <0.332.0> terminating
>
> ** Last message in was {add_on_right,{20,<4967.323.0>}}
>
> ** When Server state == {q,
>
>                           {amqqueue,
>
>                            {resource,<<"eventing">>,queue,
>
>                             <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
>                            true,false,none,
>
>                            [{<<"x-dead-letter-exchange">>,longstr,
>
>                              <<"EnterpriseEventDeadLetter">>},
>
>                             {<<"x-dead-letter-routing-key">>,longstr,
>
>                              <<"SalesCenter2.BPClaims_ClosedClaim">>}],
>
>                            <0.332.0>,
>
>                            [<8174.310.0>],
>
>                            [<8174.310.0>],
>
>                            [{vhost,<<"eventing">>},
>
>                             {name,<<"ha-all">>},
>
>                             {pattern,<<"^.*">>},
>
>                             {'apply-to',<<"queues">>},
>
>                             {definition,
>
>                              [{<<"ha-mode">>,<<"all">>},
>
>                               {<<"ha-sync-mode">>,<<"automatic">>}]},
>
>                             {priority,0}],
>
>
> [{<8174.311.0>,<8174.310.0>},{<0.333.0>,<0.332.0>}],
>
>                            []},
>
>                           none,false,rabbit_mirror_queue_master,
>
>                           {state,
>
>                            {resource,<<"eventing">>,queue,
>
>                             <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
>                            <0.333.0>,<0.4223.0>,rabbit_variable_queue,
>
>                            {vqstate,
>
>                             {0,{[],[]}},
>
>                             {0,{[],[]}},
>
>                             {delta,undefined,0,undefined},
>
>                             {0,{[],[]}},
>
>                             {0,{[],[]}},
>
>                             0,
>
>                             {0,nil},
>
>                             {0,nil},
>
>                             {qistate,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",
>
>                              {{dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                 []},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                  []}}},
>
>                               []},
>
>                              undefined,0,65536,
>
>                              #Fun<rabbit_variable_queue.2.51560583>,
>
>                              {0,nil}},
>
>                             {{client_msstate,msg_store_persistent,
>
>
> <<26,131,31,121,222,176,41,144,168,192,42,159,222,
>
>                                 13,179,107>>,
>
>                               {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                 []},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                  []}}},
>
>                               {state,311371,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},
>
>                               rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent",
>
>                               <0.266.0>,315468,307274,319565,323662},
>
>                              {client_msstate,msg_store_transient,
>
>
>                   <<32,49,130,217,86,65,129,214,32,140,24,34,254,
>
>                                 129,171,73>>,
>
>                               {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                 []},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                  []}}},
>
>                               {state,290886,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient"},
>
>                               rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient",
>
>                               <0.261.0>,294983,286789,299080,303177}},
>
>                             true,0,0,0,infinity,0,0,0,0,0,
>
>                             {rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},
>
>                             {0,nil},
>
>                             {0,nil},
>
>                             {0,nil},
>
>                             {0,nil},
>
>                             0,0},
>
>                            {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                               []}}},
>
>                            [],
>
>                            {set,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                               []}}}},
>
>
> {state,{queue,[],[],0},{active,1406466722448182,1.0}},
>
>                           undefined,undefined,undefined,undefined,
>
>                           {state,fine,5000,undefined},
>
>                           {0,nil},
>
>                           undefined,undefined,undefined,
>
>                           {state,
>
>                            {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                               []}}},
>
>                            delegate},
>
>                           {resource,<<"eventing">>,exchange,
>
>                            <<"EnterpriseEventDeadLetter">>},
>
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>,undefined,2,
>
>                           running}
>
> ** Reason for termination ==
>
> ** {function_clause,
>
>         [{rabbit_amqqueue_process,handle_call,
>
>              [{add_on_right,{20,<4967.323.0>}},
>
>               {<4967.323.0>,#Ref<4967.0.0.2348>},
>
>               {q,{amqqueue,
>
>                      {resource,<<"eventing">>,queue,
>
>                          <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
>                      true,false,none,
>
>                      [{<<"x-dead-letter-exchange">>,longstr,
>
>                        <<"EnterpriseEventDeadLetter">>},
>
>                       {<<"x-dead-letter-routing-key">>,longstr,
>
>                        <<"SalesCenter2.BPClaims_ClosedClaim">>}],
>
>                      <0.332.0>,
>
>                      [<8174.310.0>],
>
>                      [<8174.310.0>],
>
>                      [{vhost,<<"eventing">>},
>
>                       {name,<<"ha-all">>},
>
>                       {pattern,<<"^.*">>},
>
>                       {'apply-to',<<"queues">>},
>
>                       {definition,
>
>                           [{<<"ha-mode">>,<<"all">>},
>
>                            {<<"ha-sync-mode">>,<<"automatic">>}]},
>
>                       {priority,0}],
>
>                      [{<8174.311.0>,<8174.310.0>},{<0.333.0>,<0.332.0>}],
>
>                      []},
>
>                  none,false,rabbit_mirror_queue_master,
>
>                  {state,
>
>                      {resource,<<"eventing">>,queue,
>
>                          <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
>                      <0.333.0>,<0.4223.0>,rabbit_variable_queue,
>
>                      {vqstate,
>
>                          {0,{[],[]}},
>
>                          {0,{[],[]}},
>
>                          {delta,undefined,0,undefined},
>
>                          {0,{[],[]}},
>
>                          {0,{[],[]}},
>
>                          0,
>
>                          {0,nil},
>
>                          {0,nil},
>
>                          {qistate,
>
>
>        "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",
>
>                              {{dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                    [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                     [],[]}}},
>
>                               []},
>
>                              undefined,0,65536,
>
>                              #Fun<rabbit_variable_queue.2.51560583>,
>
>                              {0,nil}},
>
>                          {{client_msstate,msg_store_persistent,
>
>
> <<26,131,31,121,222,176,41,144,168,192,42,159,222,
>
>                                 13,179,107>>,
>
>                               {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                    [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                     [],[]}}},
>
>                               {state,311371,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},
>
>                               rabbit_msg_store_ets_index,
>
>
>                        "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent",
>
>                               <0.266.0>,315468,307274,319565,323662},
>
>                           {client_msstate,msg_store_transient,
>
>
> <<32,49,130,217,86,65,129,214,32,140,24,34,254,
>
>                                 129,171,73>>,
>
>                               {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                    [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                     [],[]}}},
>
>                               {state,290886,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient"},
>
>                               rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient",
>
>                               <0.261.0>,294983,286789,299080,303177}},
>
>                          true,0,0,0,infinity,0,0,0,0,0,
>
>                          {rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},
>
>                          {0,nil},
>
>                          {0,nil},
>
>                          {0,nil},
>
>                          {0,nil},
>
>                          0,0},
>
>                      {dict,0,16,16,8,80,48,
>
>                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
>                      [],
>
>                      {set,0,16,16,8,80,48,
>
>                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}},
>
>                  {state,{queue,[],[],0},{active,1406466722448182,1.0}},
>
>                  undefined,undefined,undefined,undefined,
>
>                  {state,fine,5000,undefined},
>
>                  {0,nil},
>
>                  undefined,undefined,undefined,
>
>                  {state,
>
>                      {dict,0,16,16,8,80,48,
>
>                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
>                      delegate},
>
>                  {resource,<<"eventing">>,exchange,
>
>                      <<"EnterpriseEventDeadLetter">>},
>
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>,undefined,2,running}],
>
>              []},
>
>          {gen_server2,handle_msg,2,[]},
>
>          {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]}
>
> 08:13:30 - Node STR-D-EVGAPP-02 also recorded crashes involving the same
> queues:
>
> =ERROR REPORT==== 27-Jul-2014::08:13:30 ===
>
> ** Generic server <0.323.0> terminating
>
> ** Last message in was {'$gen_cast',join}
>
> ** When Server state == {state,
>
>                              {20,<0.323.0>},
>
>                              {{20,<0.323.0>},undefined},
>
>                              {{20,<0.323.0>},undefined},
>
>                              {resource,<<"eventing">>,queue,
>
>
> <<"Insight.BPClaims_ModifiedWorkNotAvailable">>},
>
>
> rabbit_mirror_queue_slave,undefined,-1,undefined,
>
>                              [<0.322.0>],
>
>                              {[],[]},
>
>                              [],0,undefined,
>
>                              #Fun<rabbit_misc.execute_mnesia_transaction.1>}
>
> ** Reason for termination ==
>
> ** {{function_clause,
>
>          [{rabbit_amqqueue_process,handle_call,
>
>               [{add_on_right,{20,<0.323.0>}},
>
>                {<0.323.0>,#Ref<0.0.0.2348>},
>
>                {q,{amqqueue,
>
>                       {resource,<<"eventing">>,queue,
>
>                           <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
>                       true,false,none,
>
>                       [{<<"x-dead-letter-exchange">>,longstr,
>
>                         <<"EnterpriseEventDeadLetter">>},
>
>                        {<<"x-dead-letter-routing-key">>,longstr,
>
>                         <<"SalesCenter2.BPClaims_ClosedClaim">>}],
>
>                       <5297.332.0>,
>
>                       [<5251.310.0>],
>
>                       [<5251.310.0>],
>
>                       [{vhost,<<"eventing">>},
>
>                        {name,<<"ha-all">>},
>
>                        {pattern,<<"^.*">>},
>
>                        {'apply-to',<<"queues">>},
>
>                        {definition,
>
>                            [{<<"ha-mode">>,<<"all">>},
>
>                             {<<"ha-sync-mode">>,<<"automatic">>}]},
>
>                        {priority,0}],
>
>
> [{<5251.311.0>,<5251.310.0>},{<5297.333.0>,<5297.332.0>}],
>
>                       []},
>
>                   none,false,rabbit_mirror_queue_master,
>
>                   {state,
>
>                       {resource,<<"eventing">>,queue,
>
>                           <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
>                       <5297.333.0>,<5297.4223.0>,rabbit_variable_queue,
>
>                       {vqstate,
>
>                           {0,{[],[]}},
>
>                           {0,{[],[]}},
>
>                           {delta,undefined,0,undefined},
>
>                           {0,{[],[]}},
>
>                           {0,{[],[]}},
>
>                           0,
>
>                           {0,nil},
>
>                           {0,nil},
>
>                           {qistate,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",
>
>                               {{dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                     [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                      [],[]}}},
>
>                                []},
>
>                               undefined,0,65536,
>
>                               #Fun<rabbit_variable_queue.2.51560583>,
>
>                               {0,nil}},
>
>                           {{client_msstate,msg_store_persistent,
>
>
> <<26,131,31,121,222,176,41,144,168,192,42,159,
>
>                                  222,13,179,107>>,
>
>                                {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                     [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                      [],[]}}},
>
>                                {state,311371,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},
>
>                                rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent",
>
>                                <5297.266.0>,315468,307274,319565,323662},
>
>                            {client_msstate,msg_store_transient,
>
>
> <<32,49,130,217,86,65,129,214,32,140,24,34,254,
>
>                                  129,171,73>>,
>
>                                {dict,0,16,16,8,80,48,
>
>
>            {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                     [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                      [],[]}}},
>
>                                {state,290886,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient"},
>
>                                rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient",
>
>                                <5297.261.0>,294983,286789,299080,303177}},
>
>                           true,0,0,0,infinity,0,0,0,0,0,
>
>                           {rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},
>
>                           {0,nil},
>
>                           {0,nil},
>
>                           {0,nil},
>
>                           {0,nil},
>
>                           0,0},
>
>                       {dict,0,16,16,8,80,48,
>
>                           {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
>                       [],
>
>                       {set,0,16,16,8,80,48,
>
>                           {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}},
>
>                   {state,{queue,[],[],0},{active,1406466722448182,1.0}},
>
>                   undefined,undefined,undefined,undefined,
>
>                   {state,fine,5000,undefined},
>
>                   {0,nil},
>
>                   undefined,undefined,undefined,
>
>                   {state,
>
>                       {dict,0,16,16,8,80,48,
>
>                           {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
>                       delegate},
>
>                   {resource,<<"eventing">>,exchange,
>
>                       <<"EnterpriseEventDeadLetter">>},
>
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>,undefined,2,running}],
>
>               []},
>
>           {gen_server2,handle_msg,2,[]},
>
>           {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]},
>
>
> {gen_server2,call,[<5297.332.0>,{add_on_right,{20,<0.323.0>}},infinity]}}
>
> 08:13:31 - And again from node STR-D-EVGAPP-02:
>
> =ERROR REPORT==== 27-Jul-2014::08:13:31 ===
>
> ** Generic server <0.322.0> terminating
>
> ** Last message in was {'EXIT',<0.323.0>,
>
>                          {{function_clause,
>
>                            [{rabbit_amqqueue_process,handle_call,
>
>                              [{add_on_right,{20,<0.323.0>}},
>
>                               {<0.323.0>,#Ref<0.0.0.2348>},
>
>                               {q,
>
>                                {amqqueue,
>
>                                 {resource,<<"eventing">>,queue,
>
>                                  <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
>                                 true,false,none,
>
>                                 [{<<"x-dead-letter-exchange">>,longstr,
>
>                                   <<"EnterpriseEventDeadLetter">>},
>
>                                  {<<"x-dead-letter-routing-key">>,longstr,
>
>                                   <<"SalesCenter2.BPClaims_ClosedClaim">>}],
>
>                                 <5297.332.0>,
>
>                                 [<5251.310.0>],
>
>                                 [<5251.310.0>],
>
>                                 [{vhost,<<"eventing">>},
>
>                                  {name,<<"ha-all">>},
>
>                                  {pattern,<<"^.*">>},
>
>                                  {'apply-to',<<"queues">>},
>
>                                  {definition,
>
>                                   [{<<"ha-mode">>,<<"all">>},
>
>                                    {<<"ha-sync-mode">>,<<"automatic">>}]},
>
>                                  {priority,0}],
>
>                                 [{<5251.311.0>,<5251.310.0>},
>
>                                  {<5297.333.0>,<5297.332.0>}],
>
>                                 []},
>
>                                none,false,rabbit_mirror_queue_master,
>
>                                {state,
>
>                                 {resource,<<"eventing">>,queue,
>
>                                  <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
>                                 <5297.333.0>,<5297.4223.0>,
>
>                                 rabbit_variable_queue,
>
>                                 {vqstate,
>
>                                  {0,{[],[]}},
>
>                                  {0,{[],[]}},
>
>                                  {delta,undefined,0,undefined},
>
>                                  {0,{[],[]}},
>
>                                  {0,{[],[]}},
>
>                                  0,
>
>                                  {0,nil},
>
>                                  {0,nil},
>
>                                  {qistate,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",
>
>                                   {{dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                      [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                       [],[],[]}}},
>
>                                    []},
>
>                                   undefined,0,65536,
>
>                                   #Fun<rabbit_variable_queue.2.51560583>,
>
>                                   {0,nil}},
>
>                                  {{client_msstate,msg_store_persistent,
>
>
> <<26,131,31,121,222,176,41,144,168,192,42,
>
>                                      159,222,13,179,107>>,
>
>                                    {dict,0,16,16,8,80,48,
>
>
>                      {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                      [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                       [],[],[]}}},
>
>                                    {state,311371,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},
>
>                                    rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent",
>
>
> <5297.266.0>,315468,307274,319565,323662},
>
>                                   {client_msstate,msg_store_transient,
>
>
> <<32,49,130,217,86,65,129,214,32,140,24,34,
>
>                                      254,129,171,73>>,
>
>                                    {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                      [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                       [],[],[]}}},
>
>                                    {state,290886,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient"},
>
>                                    rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient",
>
>
> <5297.261.0>,294983,286789,299080,303177}},
>
>                                  true,0,0,0,infinity,0,0,0,0,0,
>
>
> {rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},
>
>                                  {0,nil},
>
>                                  {0,nil},
>
>                                  {0,nil},
>
>                                  {0,nil},
>
>                                  0,0},
>
>                                 {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                   []},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                    [],[]}}},
>
>                                 [],
>
>                                 {set,0,16,16,8,80,48,
>
>
>                         {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                   []},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                    [],[]}}}},
>
>                                {state,
>
>                                 {queue,[],[],0},
>
>                                 {active,1406466722448182,1.0}},
>
>                                undefined,undefined,undefined,undefined,
>
>                                {state,fine,5000,undefined},
>
>                                {0,nil},
>
>                                undefined,undefined,undefined,
>
>                                {state,
>
>                                 {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                   []},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                    [],[]}}},
>
>                                 delegate},
>
>                                {resource,<<"eventing">>,exchange,
>
>                                 <<"EnterpriseEventDeadLetter">>},
>
>                                <<"SalesCenter2.BPClaims_ClosedClaim">>,
>
>                                undefined,2,running}],
>
>                              []},
>
>                             {gen_server2,handle_msg,2,[]},
>
>                             {proc_lib,wake_up,3,
>
>                              [{file,"proc_lib.erl"},{line,249}]}]},
>
>                           {gen_server2,call,
>
>                            [<5297.332.0>,
>
>                             {add_on_right,{20,<0.323.0>}},
>
>                             infinity]}}}
>
> ** When Server state == {state,
>
>                           {amqqueue,
>
>                            {resource,<<"eventing">>,queue,
>
>
> <<"Insight.BPClaims_ModifiedWorkNotAvailable">>},
>
>                            true,false,none,
>
>                            [{<<"x-dead-letter-exchange">>,longstr,
>
>                              <<"EnterpriseEventDeadLetter">>},
>
>                             {<<"x-dead-letter-routing-key">>,longstr,
>
>
> <<"Insight.BPClaims_ModifiedWorkNotAvailable">>}],
>
>                            <5251.335.0>,
>
>                            [<5297.331.0>],
>
>                            [<5297.331.0>],
>
>                            [{vhost,<<"eventing">>},
>
>                             {name,<<"ha-all">>},
>
>                             {pattern,<<"^.*">>},
>
>                             {'apply-to',<<"queues">>},
>
>                             {definition,
>
>                              [{<<"ha-mode">>,<<"all">>},
>
>                               {<<"ha-sync-mode">>,<<"automatic">>}]},
>
>                             {priority,0}],
>
>                            [{<5297.332.0>,<5297.331.0>},
>
>                             {<5251.336.0>,<5251.335.0>}],
>
>                            []},
>
>                           <0.323.0>,rabbit_variable_queue,
>
>                           {vqstate,
>
>                            {0,{[],[]}},
>
>                            {0,{[],[]}},
>
>                            {delta,undefined,0,undefined},
>
>                            {0,{[],[]}},
>
>                            {0,{[],[]}},
>
>                            0,
>
>                            {0,nil},
>
>                            {0,nil},
>
>                            {qistate,
>
>
> "d:/RabbitMQ/db/rabbit@STR-D-EVGAPP-02-mnesia/queues/11YNGR2UMDIGSF4OWRUUJ2HUX",
>
>                             {{dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                 []}}},
>
>                              []},
>
>                             undefined,0,65536,
>
>                             #Fun<rabbit_variable_queue.2.51560583>,
>
>                             {0,nil}},
>
>                            {{client_msstate,msg_store_persistent,
>
>
> <<142,49,10,175,39,218,164,74,2,215,253,147,201,
>
>                                168,141,2>>,
>
>                              {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                 []}}},
>
>                              {state,303177,
>
>
> "d:/RabbitMQ/db/rabbit@STR-D-EVGAPP-02-mnesia/msg_store_persistent"},
>
>                              rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit@STR-D-EVGAPP-02-mnesia/msg_store_persistent",
>
>                              <0.251.0>,307274,299080,311371,315468},
>
>                             {client_msstate,msg_store_transient,
>
>
> <<18,152,170,252,55,149,243,37,252,234,131,180,53,
>
>                                66,14,182>>,
>
>                              {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                 []}}},
>
>                              {state,282692,
>
>
> "d:/RabbitMQ/db/rabbit@STR-D-EVGAPP-02-mnesia/msg_store_transient"},
>
>                              rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit@STR-D-EVGAPP-02-mnesia/msg_store_transient",
>
>                              <0.246.0>,286789,278595,290886,294983}},
>
>                            true,0,0,0,infinity,0,0,0,0,0,
>
>                            {rates,0.0,0.0,0.0,0.0,{1406,466811,43211}},
>
>                            {0,nil},
>
>                            {0,nil},
>
>                            {0,nil},
>
>                            {0,nil},
>
>                            0,0},
>
>                           undefined,undefined,
>
>                           {dict,0,16,16,8,80,48,
>
>
>    {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
>                           {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
>                           {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
>                           {state,
>
>                            {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                               []}}},
>
>                            delegate},
>
>                           undefined}
>
> ** Reason for termination ==
>
> ** {{function_clause,
>
>          [{rabbit_amqqueue_process,handle_call,
>
>               [{add_on_right,{20,<0.323.0>}},
>
>                {<0.323.0>,#Ref<0.0.0.2348>},
>
>                {q,{amqqueue,
>
>                       {resource,<<"eventing">>,queue,
>
>                           <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
>                       true,false,none,
>
>                       [{<<"x-dead-letter-exchange">>,longstr,
>
>                         <<"EnterpriseEventDeadLetter">>},
>
>                        {<<"x-dead-letter-routing-key">>,longstr,
>
>                         <<"SalesCenter2.BPClaims_ClosedClaim">>}],
>
>                       <5297.332.0>,
>
>                       [<5251.310.0>],
>
>                       [<5251.310.0>],
>
>                       [{vhost,<<"eventing">>},
>
>                        {name,<<"ha-all">>},
>
>                        {pattern,<<"^.*">>},
>
>                        {'apply-to',<<"queues">>},
>
>                        {definition,
>
>                            [{<<"ha-mode">>,<<"all">>},
>
>                             {<<"ha-sync-mode">>,<<"automatic">>}]},
>
>                        {priority,0}],
>
>
> [{<5251.311.0>,<5251.310.0>},{<5297.333.0>,<5297.332.0>}],
>
>                       []},
>
>                   none,false,rabbit_mirror_queue_master,
>
>                   {state,
>
>                       {resource,<<"eventing">>,queue,
>
>                           <<"SalesCenter2.BPClaims_ClosedClaim">>},
>
>                       <5297.333.0>,<5297.4223.0>,rabbit_variable_queue,
>
>                       {vqstate,
>
>                           {0,{[],[]}},
>
>                           {0,{[],[]}},
>
>                           {delta,undefined,0,undefined},
>
>                           {0,{[],[]}},
>
>                           {0,{[],[]}},
>
>                           0,
>
>                           {0,nil},
>
>                           {0,nil},
>
>                           {qistate,
>
>
>           "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/queues/6GQ2ZHJGDF7X2GH1FS0QPJX2O",
>
>                               {{dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                     [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                      [],[]}}},
>
>                                []},
>
>                               undefined,0,65536,
>
>                               #Fun<rabbit_variable_queue.2.51560583>,
>
>                               {0,nil}},
>
>                           {{client_msstate,msg_store_persistent,
>
>
> <<26,131,31,121,222,176,41,144,168,192,42,159,
>
>                                  222,13,179,107>>,
>
>                                {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                     [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                      [],[]}}},
>
>                                {state,311371,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent"},
>
>                                rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_persistent",
>
>                                <5297.266.0>,315468,307274,319565,323662},
>
>                            {client_msstate,msg_store_transient,
>
>
> <<32,49,130,217,86,65,129,214,32,140,24,34,254,
>
>                                  129,171,73>>,
>
>                                {dict,0,16,16,8,80,48,
>
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                     [],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
>                                      [],[]}}},
>
>                                {state,290886,
>
>
>        "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient"},
>
>                                rabbit_msg_store_ets_index,
>
>
> "d:/RabbitMQ/db/rabbit@DIX-D-EVGAPP-02-mnesia/msg_store_transient",
>
>                                <5297.261.0>,294983,286789,299080,303177}},
>
>                           true,0,0,0,infinity,0,0,0,0,0,
>
>                           {rates,0.0,0.0,0.0,0.0,{1406,466723,462160}},
>
>                           {0,nil},
>
>                           {0,nil},
>
>                           {0,nil},
>
>                           {0,nil},
>
>                           0,0},
>
>                       {dict,0,16,16,8,80,48,
>
>                           {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
>                       [],
>
>                       {set,0,16,16,8,80,48,
>
>                           {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}},
>
>                   {state,{queue,[],[],0},{active,1406466722448182,1.0}},
>
>                   undefined,undefined,undefined,undefined,
>
>                   {state,fine,5000,undefined},
>
>                   {0,nil},
>
>                   undefined,undefined,undefined,
>
>                   {state,
>
>                       {dict,0,16,16,8,80,48,
>
>                           {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>
>                       delegate},
>
>                   {resource,<<"eventing">>,exchange,
>
>                       <<"EnterpriseEventDeadLetter">>},
>
>
> <<"SalesCenter2.BPClaims_ClosedClaim">>,undefined,2,running}],
>
>               []},
>
>           {gen_server2,handle_msg,2,[]},
>
>           {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]},
>
>
> {gen_server2,call,[<5297.332.0>,{add_on_right,{20,<0.323.0>}},infinity]}}
>
> This e-mail is confidential.  If you are not the intended recipient, you
> must not disclose or use the information contained in it.  If you have
> received this e-mail in error, please tell us immediately by return
> e-mail and delete the document. No recipient may use the information in
> this e-mail in violation of any civil or criminal statute. Sentry
> disclaims all liability for any unauthorized uses of this e-mail or its
> contents. Sentry accepts no liability or responsibility for any damage
> caused by any virus transmitted with this e-mail.
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
> please subscribe to the new list!
>
> [hidden email]
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>


_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

Loading...