Quantcast

Queue will not delete

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Queue will not delete

Ganann, Kale

Hi all,

 

I have a queue that won’t delete in RabbitMQ 3.2.4.  When I try to access it via the management plug-in, I get:

 

TypeError: Cannot read property 'ram_msg_count' of undefined

TypeError: Cannot read property 'ram_msg_count' of undefined

And nothing displays.  When I try to delete it via the API, it hangs and I get a timeout.  I can list the queue via the API, but policies don’t apply to the queue, and message counts show as “?” in the management plug-in.  I tried having the cluster forget the node that owned the queue, but it remained despite this.  I bypassed the VIP and connected directly to the API on the node itself, and it gave the same results.  Additionally, nothing was noted in the normal or SASL logs to denote the attempt – the only way you can even tell I was there was the access log from the management plug-in.  I ran rabbitmqctl report, and am attaching what I got, but it keeps freezing partway through, which I’m guessing is it trying to access whatever is stuck.

 

As always, any help you can provide is appreciated.

 

Thanks,

Kale


_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

report.txt (279K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Queue will not delete

Michael Klishin-4
This list has moved to rabbitmq-users. Please start all new discussions there:
https://groups.google.com/forum/#!forum/rabbitmq-users 

On 26 August 2014 at 23:21:49, Ganann, Kale ([hidden email]) wrote:
> > I have a queue that won’t delete in RabbitMQ 3.2.4.

Sounds like bug 26180 which was fixed in 3.3.2:
http://www.rabbitmq.com/release-notes/README-3.3.2.txt
--  
MK  

Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Queue will not delete

Ganann, Kale
Does that only apply when the socket limit is reached?  Because we're nowhere near our socket connection limits.  The highest is at 332, and we have 943626 available per node.  That said, is there any way to identify the bad connection?  Our last attempt to upgrade to 3.3.x was disastrous; we had to roll back everything.  If we can fix this at our current rev it would be much appreciated.

Thanks for the information,
Kale



-----Original Message-----
From: Michael Klishin [mailto:[hidden email]]
Sent: Tuesday, August 26, 2014 2:24 PM
To: Ganann, Kale; Legacy list about RabbitMQ
Subject: Re: [rabbitmq-discuss] Queue will not delete

This list has moved to rabbitmq-users. Please start all new discussions there:
https://groups.google.com/forum/#!forum/rabbitmq-users 

On 26 August 2014 at 23:21:49, Ganann, Kale ([hidden email]) wrote:
> > I have a queue that won’t delete in RabbitMQ 3.2.4.

Sounds like bug 26180 which was fixed in 3.3.2:
http://www.rabbitmq.com/release-notes/README-3.3.2.txt
--  
MK  

Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Queue will not delete

Michael Klishin-4
 On 26 August 2014 at 23:56:34, Ganann, Kale ([hidden email]) wrote:
> If we can fix this at our current rev it would be much appreciated.  

I'm afraid investigating how to do that would take a while, and may yield
no results.

Have you reported any 3.3.x upgrade issues to us? What was the problem?
--  
MK  

Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Queue will not delete

Michael Klishin-4
In reply to this post by Ganann, Kale
 On 26 August 2014 at 23:21:49, Ganann, Kale ([hidden email]) wrote:
> >
> I have a queue that won’t delete in RabbitMQ 3.2.4. When I try to  
> access it via the management plug-in, I get:
> TypeError: Cannot read property 'ram_msg_count' of undefined  
>  
>  
> TypeError: Cannot read property 'ram_msg_count' of undefined  
>  

What is in the log files? How do you delete the queue, using a Rabbit client
or HTTP API?
--  
MK  

Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Queue will not delete

Ganann, Kale
As I said, nothing is in the log files.  The only reason you can tell I even attempted it is I show up in the access logs for the management plug in.

I have tried to delete via the management plug in and via the HTTP API.

I reported some of the issues we were having moving to 3.3 at the time, largely around (but not limited to) federation completely and totally failing for us once we moved up, but we weren't yielding great results and had to roll back for time constraints to get our environments up and running.

-----Original Message-----
From: Michael Klishin [mailto:[hidden email]]
Sent: Tuesday, August 26, 2014 3:01 PM
To: Ganann, Kale; Legacy list about RabbitMQ
Subject: Re: [rabbitmq-discuss] Queue will not delete

 On 26 August 2014 at 23:21:49, Ganann, Kale ([hidden email]) wrote:
> >
> I have a queue that won’t delete in RabbitMQ 3.2.4. When I try to
> access it via the management plug-in, I get:
> TypeError: Cannot read property 'ram_msg_count' of undefined
>  
>  
> TypeError: Cannot read property 'ram_msg_count' of undefined
>  

What is in the log files? How do you delete the queue, using a Rabbit client or HTTP API?
--
MK  

Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Queue will not delete

Michael Klishin-4
 On 27 August 2014 at 00:05:39, Ganann, Kale ([hidden email]) wrote:
> > I reported some of the issues we were having moving to 3.3 at the  
> time, largely around (but not limited to) federation completely  
> and totally failing for us once we moved up, but we weren't yielding  
> great results and had to roll back for time constraints to get  
> our environments up and running.

Understood.

What version exactly was that with?

There were Federation fixes in the 3.3.x series:

http://www.rabbitmq.com/changelog.html
http://www.rabbitmq.com/release-notes/README-3.3.2.txt
http://www.rabbitmq.com/release-notes/README-3.3.5.txt
--  
MK  

Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Queue will not delete

Ganann, Kale
In reply to this post by Ganann, Kale
Hmm.  Here's something interesting - I tried to overwrite it, and got this back from the api:

PS C:\Users\kganann.CCP> Write-RMQQueue -RabbitMQServer $RMQDev -Credential $Cred -vHost "master" -Name "q.d2.ha.gemini.internalIndexing"
Invoke-WebRequest : {"error":"bad_request","reason":"406 PRECONDITION_FAILED - parameters for queue
'q.d2.ha.gemini.internalIndexing' in vhost 'master' not equivalent"}
At C:\Windows\system32\WindowsPowerShell\v1.0\Modules\rabbitmq\RabbitMQ.psm1:1074 char:17
+         $RawResults = Invoke-WebRequest -Uri "http://$RabbitMQServer`:$Port/api/queues ...
+    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebExc
   eption
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
ConvertFrom-Json : Cannot bind argument to parameter 'InputObject' because it is null.
At C:\Windows\system32\WindowsPowerShell\v1.0\Modules\rabbitmq\RabbitMQ.psm1:1077 char:30
+     $Results = ConvertFrom-Json $RawResults
+                                 ~~~~~~~~~~~
    + CategoryInfo          : InvalidData: (:) [ConvertFrom-Json], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,Microsoft.PowerShell.Commands.ConvertFrom
   JsonCommand

-----Original Message-----
From: rabbitmq-discuss [mailto:[hidden email]] On Behalf Of Ganann, Kale
Sent: Tuesday, August 26, 2014 3:06 PM
To: Michael Klishin; Legacy list about RabbitMQ
Subject: Re: [rabbitmq-discuss] Queue will not delete

As I said, nothing is in the log files.  The only reason you can tell I even attempted it is I show up in the access logs for the management plug in.

I have tried to delete via the management plug in and via the HTTP API.

I reported some of the issues we were having moving to 3.3 at the time, largely around (but not limited to) federation completely and totally failing for us once we moved up, but we weren't yielding great results and had to roll back for time constraints to get our environments up and running.

-----Original Message-----
From: Michael Klishin [mailto:[hidden email]]
Sent: Tuesday, August 26, 2014 3:01 PM
To: Ganann, Kale; Legacy list about RabbitMQ
Subject: Re: [rabbitmq-discuss] Queue will not delete

 On 26 August 2014 at 23:21:49, Ganann, Kale ([hidden email]) wrote:
> >
> I have a queue that won’t delete in RabbitMQ 3.2.4. When I try to
> access it via the management plug-in, I get:
> TypeError: Cannot read property 'ram_msg_count' of undefined
>  
>  
> TypeError: Cannot read property 'ram_msg_count' of undefined
>  

What is in the log files? How do you delete the queue, using a Rabbit client or HTTP API?
--
MK  

Staff Software Engineer, Pivotal/RabbitMQ _______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Queue will not delete

Ganann, Kale
In reply to this post by Michael Klishin-4
We went to 3.3.4.  We're making a move to get off federation - we've had too many issues with it, so upgrading might be in the cards again, but I'll have to pitch to the managers.

-----Original Message-----
From: Michael Klishin [mailto:[hidden email]]
Sent: Tuesday, August 26, 2014 3:13 PM
To: Legacy list about RabbitMQ; Ganann, Kale
Subject: RE: [rabbitmq-discuss] Queue will not delete

 On 27 August 2014 at 00:05:39, Ganann, Kale ([hidden email]) wrote:
> > I reported some of the issues we were having moving to 3.3 at the
> time, largely around (but not limited to) federation completely and
> totally failing for us once we moved up, but we weren't yielding great
> results and had to roll back for time constraints to get our
> environments up and running.

Understood.

What version exactly was that with?

There were Federation fixes in the 3.3.x series:

http://www.rabbitmq.com/changelog.html
http://www.rabbitmq.com/release-notes/README-3.3.2.txt
http://www.rabbitmq.com/release-notes/README-3.3.5.txt
--
MK  

Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Queue will not delete

Michael Klishin-4
In reply to this post by Ganann, Kale
On 27 August 2014 at 00:15:19, Ganann, Kale ([hidden email]) wrote:
> > Write-RMQQueue -RabbitMQServer $RMQDev -Credential $Cred  
> -vHost "master" -Name "q.d2.ha.gemini.internalIndexing"  
> Invoke-WebRequest : {"error":"bad_request","reason":"406  
> PRECONDITION_FAILED - parameters for queue
> 'q.d2.ha.gemini.internalIndexing' in vhost 'master' not  
> equivalent"}

If by "overwrite" you mean "redeclare", then it is nothing unexpected:
you simply provided different parameters (e.g. durability). When you provide the correct ones, the operation will be a no-op.

So I'm afraid this is a red herring.
--  
MK  

Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Queue will not delete

Ganann, Kale
Shoot.

Since the bug is listed as being related to a connection, would restarting the whole cluster clear the broken connection?

-----Original Message-----
From: Michael Klishin [mailto:[hidden email]]
Sent: Tuesday, August 26, 2014 3:17 PM
To: Ganann, Kale; Legacy list about RabbitMQ
Subject: Re: [rabbitmq-discuss] Queue will not delete

On 27 August 2014 at 00:15:19, Ganann, Kale ([hidden email]) wrote:
> > Write-RMQQueue -RabbitMQServer $RMQDev -Credential $Cred
> -vHost "master" -Name "q.d2.ha.gemini.internalIndexing"  
> Invoke-WebRequest : {"error":"bad_request","reason":"406
> PRECONDITION_FAILED - parameters for queue
> 'q.d2.ha.gemini.internalIndexing' in vhost 'master' not equivalent"}

If by "overwrite" you mean "redeclare", then it is nothing unexpected:
you simply provided different parameters (e.g. durability). When you provide the correct ones, the operation will be a no-op.

So I'm afraid this is a red herring.
--
MK  

Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Queue will not delete

Michael Klishin-4
On 27 August 2014 at 00:23:46, Ganann, Kale ([hidden email]) wrote:
> > Since the bug is listed as being related to a connection, would  
> restarting the whole cluster clear the broken connection?  

Connections are node-local. You definitely don't need to restart the entire
cluster. 
--  
MK  

Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Queue will not delete

Ganann, Kale
But I would have to know which node the connection was linked to.  I'm not saying all at once, but I would need to cycle through the nodes until it got wiped, correct?

-----Original Message-----
From: Michael Klishin [mailto:[hidden email]]
Sent: Tuesday, August 26, 2014 3:25 PM
To: Legacy list about RabbitMQ; Ganann, Kale
Subject: RE: [rabbitmq-discuss] Queue will not delete

On 27 August 2014 at 00:23:46, Ganann, Kale ([hidden email]) wrote:
> > Since the bug is listed as being related to a connection, would
> restarting the whole cluster clear the broken connection?  

Connections are node-local. You definitely don't need to restart the entire cluster.
--
MK  

Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Queue will not delete

Michael Klishin-4
On 27 August 2014 at 00:32:16, Ganann, Kale ([hidden email]) wrote:
> > But I would have to know which node the connection was linked
> to. I'm not saying all at once, but I would need to cycle through
> the nodes until it got wiped, correct?

If your apps can recover from server-sent connection.close, you can
close all connections via HTTP API in a loop.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Loading...