RabbitMQ server 3.1.3: "badmatch, {error, not_found}" (server down)

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

RabbitMQ server 3.1.3: "badmatch, {error, not_found}" (server down)

Badjatya, Nikunj

Hello All,

 

My RabbitMQ server is down and I am seeing following error in RabbitMQ logs.

 

{{{

service@server:/var/log/rabbitmq> cat vim rabbit\@localhost.log

 

[snip]

 

=INFO REPORT==== 2-Jun-2016::09:35:58 ===

Error description:

   {badmatch,{error,not_found}}

 

Log files (may contain more information):

   /var/log/rabbitmq/[hidden email]

   /var/log/rabbitmq/[hidden email]

 

Stack trace:

   [{rabbit_binding,recover_semi_durable_route,3,[]},

    {rabbit_binding,'-recover/2-lc$^0/1-0-',3,[]},

    {rabbit_binding,recover,2,[]},

    {rabbit,recover,0,[]},

    {rabbit,'-run_boot_step/1-lc$^1/1-1-',1,[]},

    {rabbit,run_boot_step,1,[]},

    {rabbit,'-start/2-lc$^0/1-0-',1,[]},

    {rabbit,start,2,[]}]

 

 

=INFO REPORT==== 2-Jun-2016::09:35:59 ===

Error description:

   {could_not_start,rabbit,

       {bad_return,

           {{rabbit,start,[normal,[]]},

            {'EXIT',

                {rabbit,failure_during_boot,{badmatch,{error,not_found}}}}}}}

 

Log files (may contain more information):

   /var/log/rabbitmq/[hidden email]

   [hidden email]

 

[snip]

 

}}}

 

Manual reset and start_app does not work either.

 

I noticed this error first time when tried to create a rabbitmq user using rabbitmqctl.

 

Any idea what is causing this issue?

 

I am using RabbitMQ server “3.1.3-1.suse” on SLES-11 SP3 machine.

 

Thanks in advance for your time.

 

 

Regards,

Nikunj


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: RabbitMQ server 3.1.3: "badmatch, {error, not_found}" (server down)

Michael Klishin-2
On 3 June 2014 at 14:15:52, Badjatya, Nikunj ([hidden email]) wrote:

> >
> Stack trace:
> [{rabbit_binding,recover_semi_durable_route,3,[]},
> {rabbit_binding,'-recover/2-lc$^0/1-0-',3,[]},
> {rabbit_binding,recover,2,[]},
> {rabbit,recover,0,[]},
> {rabbit,'-run_boot_step/1-lc$^1/1-1-',1,[]},
> {rabbit,run_boot_step,1,[]},
> {rabbit,'-start/2-lc$^0/1-0-',1,[]},
> {rabbit,start,2,[]}]

See http://markmail.org/thread/26j56abreqwd32fq and http://markmail.org/thread/ntgfmmmnfrhjodtj.

This is a known bug (#25783) which is not resolved yet. 
--  
MK  

Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: RabbitMQ server 3.1.3: "badmatch, {error, not_found}" (server down)

Simon MacMullen-2
On 03/06/14 11:41, Michael Klishin wrote:
> This is a known bug (#25783) which is not resolved yet.

But one which we don't fully understand. If this is happening
consistently at startup then please could you zip the Mnesia directory
and send it to me off-list?

Cheers, Simon

--
Simon MacMullen
RabbitMQ, Pivotal
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: RabbitMQ server 3.1.3: "badmatch, {error, not_found}" (server down)

Matthias Radestock-3
In reply to this post by Michael Klishin-2
On 03/06/14 11:41, Michael Klishin wrote:

> On 3 June 2014 at 14:15:52, Badjatya, Nikunj ([hidden email]) wrote:
>>>
>> Stack trace:
>> [{rabbit_binding,recover_semi_durable_route,3,[]},
>> {rabbit_binding,'-recover/2-lc$^0/1-0-',3,[]},
>> {rabbit_binding,recover,2,[]},
>> {rabbit,recover,0,[]},
>> {rabbit,'-run_boot_step/1-lc$^1/1-1-',1,[]},
>> {rabbit,run_boot_step,1,[]},
>> {rabbit,'-start/2-lc$^0/1-0-',1,[]},
>> {rabbit,start,2,[]}]
>
> See http://markmail.org/thread/26j56abreqwd32fq and http://markmail.org/thread/ntgfmmmnfrhjodtj.
>
> This is a known bug (#25783) which is not resolved yet.

Actually, there is a bug with the above signature that we fixed in 3.2.0

25822 prevent crash at startup when starting a clustered node hosting a
       durable non-HA queue which had been bound to a transient exchange
which
       was deleted when the node was down (since 2.5.0)

Does that at all sound like what might be happening?

If not then, as Simon says, please send him a copy of the mnesia dir
off-list.

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: RabbitMQ server 3.1.3: "badmatch, {error, not_found}" (server down)

Badjatya, Nikunj
Thank you all for quick replies.
Well this issue was not reproducible and we hit this only once.

I didn’t quite understand the reason given in http://markmail.org/thread/26j56abreqwd32fq#query:+page:1+mid:2j2la4yehcxferjy+state:results link.

Can you please explain it in simpler way?

I am not deleting node as part of any operation. Here are the operations which I am doing currently using Pika python library:

1. Create user using rabbitmqctl
2. Connect to localhost using BlockingConnection
3. get channel
4. create exchange if not done already
5. publish using basic_publish
6. Close connection
6. Delete user using rabbitmqctl


Thanks
Nikunj


-----Original Message-----
From: Matthias Radestock [mailto:[hidden email]]
Sent: Tuesday, June 03, 2014 4:55 PM
To: Discussions about RabbitMQ
Cc: Badjatya, Nikunj
Subject: Re: [rabbitmq-discuss] RabbitMQ server 3.1.3: "badmatch, {error, not_found}" (server down)

On 03/06/14 11:41, Michael Klishin wrote:

> On 3 June 2014 at 14:15:52, Badjatya, Nikunj ([hidden email]) wrote:
>>>
>> Stack trace:
>> [{rabbit_binding,recover_semi_durable_route,3,[]},
>> {rabbit_binding,'-recover/2-lc$^0/1-0-',3,[]},
>> {rabbit_binding,recover,2,[]},
>> {rabbit,recover,0,[]},
>> {rabbit,'-run_boot_step/1-lc$^1/1-1-',1,[]},
>> {rabbit,run_boot_step,1,[]},
>> {rabbit,'-start/2-lc$^0/1-0-',1,[]},
>> {rabbit,start,2,[]}]
>
> See http://markmail.org/thread/26j56abreqwd32fq and http://markmail.org/thread/ntgfmmmnfrhjodtj.
>
> This is a known bug (#25783) which is not resolved yet.

Actually, there is a bug with the above signature that we fixed in 3.2.0

25822 prevent crash at startup when starting a clustered node hosting a
       durable non-HA queue which had been bound to a transient exchange which
       was deleted when the node was down (since 2.5.0)

Does that at all sound like what might be happening?

If not then, as Simon says, please send him a copy of the mnesia dir off-list.

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: RabbitMQ server 3.1.3: "badmatch, {error, not_found}" (server down)

Badjatya, Nikunj
Hello Folks,

Any thoughts on this?

Thanks


-----Original Message-----
From: rabbitmq-discuss [mailto:[hidden email]] On Behalf Of Badjatya, Nikunj
Sent: Wednesday, June 04, 2014 8:52 PM
To: Matthias Radestock; Discussions about RabbitMQ
Subject: Re: [rabbitmq-discuss] RabbitMQ server 3.1.3: "badmatch, {error, not_found}" (server down)

Thank you all for quick replies.
Well this issue was not reproducible and we hit this only once.

I didn’t quite understand the reason given in http://markmail.org/thread/26j56abreqwd32fq#query:+page:1+mid:2j2la4yehcxferjy+state:results link.

Can you please explain it in simpler way?

I am not deleting node as part of any operation. Here are the operations which I am doing currently using Pika python library:

1. Create user using rabbitmqctl
2. Connect to localhost using BlockingConnection 3. get channel 4. create exchange if not done already 5. publish using basic_publish 6. Close connection 6. Delete user using rabbitmqctl


Thanks
Nikunj


-----Original Message-----
From: Matthias Radestock [mailto:[hidden email]]
Sent: Tuesday, June 03, 2014 4:55 PM
To: Discussions about RabbitMQ
Cc: Badjatya, Nikunj
Subject: Re: [rabbitmq-discuss] RabbitMQ server 3.1.3: "badmatch, {error, not_found}" (server down)

On 03/06/14 11:41, Michael Klishin wrote:

> On 3 June 2014 at 14:15:52, Badjatya, Nikunj ([hidden email]) wrote:
>>>
>> Stack trace:
>> [{rabbit_binding,recover_semi_durable_route,3,[]},
>> {rabbit_binding,'-recover/2-lc$^0/1-0-',3,[]},
>> {rabbit_binding,recover,2,[]},
>> {rabbit,recover,0,[]},
>> {rabbit,'-run_boot_step/1-lc$^1/1-1-',1,[]},
>> {rabbit,run_boot_step,1,[]},
>> {rabbit,'-start/2-lc$^0/1-0-',1,[]},
>> {rabbit,start,2,[]}]
>
> See http://markmail.org/thread/26j56abreqwd32fq and http://markmail.org/thread/ntgfmmmnfrhjodtj.
>
> This is a known bug (#25783) which is not resolved yet.

Actually, there is a bug with the above signature that we fixed in 3.2.0

25822 prevent crash at startup when starting a clustered node hosting a
       durable non-HA queue which had been bound to a transient exchange which
       was deleted when the node was down (since 2.5.0)

Does that at all sound like what might be happening?

If not then, as Simon says, please send him a copy of the mnesia dir off-list.

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: RabbitMQ server 3.1.3: "badmatch, {error, not_found}" (server down)

Badjatya, Nikunj
In reply to this post by Matthias Radestock-3
Hello All,

Any thoughts on this?

Thanks.


-----Original Message-----
From: Badjatya, Nikunj
Sent: Wednesday, June 04, 2014 8:52 PM
To: 'Matthias Radestock'; 'Discussions about RabbitMQ'
Subject: RE: [rabbitmq-discuss] RabbitMQ server 3.1.3: "badmatch, {error, not_found}" (server down)

Thank you all for quick replies.
Well this issue was not reproducible and we hit this only once.

I didn’t quite understand the reason given in http://markmail.org/thread/26j56abreqwd32fq#query:+page:1+mid:2j2la4yehcxferjy+state:results link.

Can you please explain it in simpler way?

I am not deleting node as part of any operation. Here are the operations which I am doing currently using Pika python library:

1. Create user using rabbitmqctl
2. Connect to localhost using BlockingConnection 3. get channel 4. create exchange if not done already 5. publish using basic_publish 6. Close connection 6. Delete user using rabbitmqctl


Thanks
Nikunj


-----Original Message-----
From: Matthias Radestock [mailto:[hidden email]]
Sent: Tuesday, June 03, 2014 4:55 PM
To: Discussions about RabbitMQ
Cc: Badjatya, Nikunj
Subject: Re: [rabbitmq-discuss] RabbitMQ server 3.1.3: "badmatch, {error, not_found}" (server down)

On 03/06/14 11:41, Michael Klishin wrote:

> On 3 June 2014 at 14:15:52, Badjatya, Nikunj ([hidden email]) wrote:
>>>
>> Stack trace:
>> [{rabbit_binding,recover_semi_durable_route,3,[]},
>> {rabbit_binding,'-recover/2-lc$^0/1-0-',3,[]},
>> {rabbit_binding,recover,2,[]},
>> {rabbit,recover,0,[]},
>> {rabbit,'-run_boot_step/1-lc$^1/1-1-',1,[]},
>> {rabbit,run_boot_step,1,[]},
>> {rabbit,'-start/2-lc$^0/1-0-',1,[]},
>> {rabbit,start,2,[]}]
>
> See http://markmail.org/thread/26j56abreqwd32fq and http://markmail.org/thread/ntgfmmmnfrhjodtj.
>
> This is a known bug (#25783) which is not resolved yet.

Actually, there is a bug with the above signature that we fixed in 3.2.0

25822 prevent crash at startup when starting a clustered node hosting a
       durable non-HA queue which had been bound to a transient exchange which
       was deleted when the node was down (since 2.5.0)

Does that at all sound like what might be happening?

If not then, as Simon says, please send him a copy of the mnesia dir off-list.

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: RabbitMQ server 3.1.3: "badmatch, {error, not_found}" (server down)

Michael Klishin-2
On 6 June 2014 at 14:16:26, Badjatya, Nikunj ([hidden email]) wrote:
> > Any thoughts on this?

Try 3.3.1 and if the issue persists, send us an archive of your Mnesia directory
off-list.
--
MK

Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss