Quantcast

Message timeouts

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Message timeouts

Alex Grönholm-3
I've developed an RPC system using RabbitMQ (with different semantics than the built-in RPC). I'd like to get the server to send the client some kind of a notification if a message it has sent will never be delivered (like if the queue which it was routed to was deleted before the message was consumed from it). Is there any way to accomplish this? Client side timeouts are not good enough either because the request might get processed at some point then anyway.

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Message timeouts

Valentin Bernard
Hi,

If your clients publish messages with the mandatory flag set, they
will get a Basic.Return from the server if the message can't be routed
to a queue. Does that solve your problem?

Java example: http://www.rabbitmq.com/api-guide.html#returning

Cheers,

Valentin.

On 2 déc, 12:33, Alex Grönholm <[hidden email]> wrote:

> I've developed an RPC system using RabbitMQ (with different semantics than
> the built-in RPC). I'd like to get the server to send the client some kind
> of a notification if a message it has sent will never be delivered (like if
> the queue which it was routed to was deleted before the message was
> consumed from it). Is there any way to accomplish this? Client side
> timeouts are not good enough either because the request might get processed
> at some point then anyway.
>
> _______________________________________________
> rabbitmq-discuss mailing list
> [hidden email]://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Message timeouts

Simon MacMullen-2
If on the other hand Alex is interested in learning about messages that
have been deleted once they've made it to queues then there's currently
no way to find out about that.

We are currently looking at implementing dead lettering, which might
help with this.

Cheers, Simon

On 02/12/11 12:12, Valentin BERNARD wrote:

> Hi,
>
> If your clients publish messages with the mandatory flag set, they
> will get a Basic.Return from the server if the message can't be routed
> to a queue. Does that solve your problem?
>
> Java example: http://www.rabbitmq.com/api-guide.html#returning
>
> Cheers,
>
> Valentin.
>
> On 2 déc, 12:33, Alex Grönholm<[hidden email]>  wrote:
>> I've developed an RPC system using RabbitMQ (with different semantics than
>> the built-in RPC). I'd like to get the server to send the client some kind
>> of a notification if a message it has sent will never be delivered (like if
>> the queue which it was routed to was deleted before the message was
>> consumed from it). Is there any way to accomplish this? Client side
>> timeouts are not good enough either because the request might get processed
>> at some point then anyway.
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> [hidden email]://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> _______________________________________________
> rabbitmq-discuss mailing list
> [hidden email]
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


--
Simon MacMullen
RabbitMQ, VMware
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Message timeouts

Alex Grönholm-3
In reply to this post by Valentin Bernard
That only happens if the queue is not there to begin with. If the queue is there and it expires, no return message is ever generated.

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Message timeouts

Alex Grönholm-3
In reply to this post by Simon MacMullen-2
Yes, that is exactly my problem. Seems there is no solution then :(
I want to keep my "rpc" queue alive even when the RPC server software is being upgraded so that users will only see a delay in the responses (and no lost requests).

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Message timeouts

Matthias Radestock-3
On 02/12/11 12:36, Alex Grönholm wrote:
> I want to keep my "rpc" queue alive even when the RPC server software is
> being upgraded so that users will only see a delay in the responses (and
> no lost requests).

So just make sure the rpc queue stays around during the upgrade.

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Message timeouts

Alex Grönholm-3
That was the plan from the start, but I wanted to cover cases where something goes wrong (server starts with rabbitmq but not the rpc server) in which case the client would be stuck waiting forever.
But I guess that dead lettering feature will fix that.

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Message timeouts

Steve Powell-2
In reply to this post by Matthias Radestock-3
Dear Alex,

Just to be precise about this: the 'standard' RpcServer uses a passed in queue
or creates its queue as follows:

        queueDeclare()

which is equivalent to:

        queueDeclare("", false, true, true, null)

which creates a non-durable, exclusive, auto-delete queue, named by the broker.

If you want your queue to survive the connection, it mustn't be exclusive to the
connection that creates it.

In order to survive a server failure you would make it durable.

To re-use the same queue later (after upgrade?), you had better name it yourself
(and re-use the name each time).

If the queue should persist after the last consumer (RpcServer?) is cancelled
from the queue, then it should not be auto-delete.

Does this give you the flexibility you require?

I do not know  your situation exactly, but it looks as though you need a named
non-exclusive queue, at the very least.
If you expect each server instance to be solely responsible for the request
queue it serves, then a simple naming scheme will suffice, but you still have to
be careful to have at most one instance of that server running at any one time.

Of course, having more than one server processing requests from a queue is
alright, too; if you are happy with way requests are shared out.

Steve Powell
[hidden email]
[wrk: +44-2380-111-528] [mob: +44-7815-838-558]

On 2 Dec 2011, at 12:44, Matthias Radestock wrote:

> On 02/12/11 12:36, Alex Grönholm wrote:
>> I want to keep my "rpc" queue alive even when the RPC server software is
>> being upgraded so that users will only see a delay in the responses (and
>> no lost requests).
>
> So just make sure the rpc queue stays around during the upgrade.
>
> Matthias.
> _______________________________________________
> rabbitmq-discuss mailing list
> [hidden email]
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Message timeouts

Tony Garnock-Jones-6
In reply to this post by Alex Grönholm-3
This is where the "mandatory" feature comes in.

1. Declare your queue as durable.
2. Send your RPC requests with "mandatory" flag.

Then:
  • if rabbitmq is down, the RPC client won't be able to connect. Clear.
  • if rabbitmq is up, but the RPC server is down, and noone has created the queue yet, then the request will be Basic.Return'd to you. Also clear.
  • if rabbitmq is up, but the RPC server is down, and the queue was previously created by somebody, then the request will be enqueued. As long as the RPC server returns to service in a timely way, everyone's happy.
  • if rabbitmq is up, and the RPC server is up, normal service ensues.

Regards,
  Tony



On 2 December 2011 09:21, Alex Grönholm <[hidden email]> wrote:
That was the plan from the start, but I wanted to cover cases where something goes wrong (server starts with rabbitmq but not the rpc server) in which case the client would be stuck waiting forever.
But I guess that dead lettering feature will fix that.

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss



_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Message timeouts

Alex Grönholm-3

if rabbitmq is up, but the RPC server is down, and the queue was previously created by somebody, then the request will be enqueued. As long as the RPC server returns to service in a timely way, everyone's happy.

And if it doesn't? How can I get the request cancelled then? I'm not sure I should rely on client generated timestamps. Their clocks could be wrong.

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Message timeouts

Alex Grönholm-3
In reply to this post by Steve Powell-2
Dear Steve,

I know how to create queues properly. That was never the issue. The issue was that if I created a leased queue, take the consumer offline and leave it that way and the queue expires, the client is never notified that the message was dropped and it will continue to wait for a response forever. That is not how I want it to work. I expect the server to send some return message when the queue dies. The mandatory flag does not help since it only returns the message if it's not routed to a queue (which it is). But like I said before, the dead lettering feature should help with this.

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Message timeouts

Tony Garnock-Jones-6
In reply to this post by Alex Grönholm-3
On 4 December 2011 21:45, Alex Grönholm <[hidden email]> wrote:
And if it doesn't? How can I get the request cancelled then? I'm not sure I should rely on client generated timestamps. Their clocks could be wrong.

You could try per-queue message TTLs: http://www.rabbitmq.com/extensions.html#queue-ttl

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Message timeouts

Steve Powell-2
In reply to this post by Alex Grönholm-3
Sorry Alex, I'm sure you do. I'm learning and trying to understand your
situation.

Although it is reasonable to create a "leased" queue (with an 'x-expires'
argument) for replies to Rpc requests (on the grounds that the clients cannot
be replied upon to drain them), I do not think it reasonable to do this for the
server queues unless you can tolerate requests being (quietly) lost.

If you don't want the server queue requests to be 'lost' then don't allow the
queues to expire!

On the other hand your clients apparently wait forever.  Is there a reason I
don't understand why the server queues expire but the clients don't have a
timeout for the responses?

Well, I think I can answer my own question -- please confirm if this is right:

You want requests to either be processed or (reliably) not processed.  You
never want the clients to be in any doubt about whether the request was
processed or not.  So you wait forever in the client.  However, requests that
get discarded never return so the client doesn't know.  So you want expiry to
trigger replies to the clients.  (Per-queue message TTLs also don't return.)

If the server dies there is still the possibility that it did so after pulling
the request from the queue and before posting a reply -- unless the requests are
'transactionally' processed. So it looks like you need the request processing to
be a transaction so that all requests not in the queue (when it expires) have
definitely been processed.

Let's see -- how would we do this without a dead-letter mechanism?  Well, apart
from making the server transactional, it might be useful to have an 'offline'
server that can take over any server's queue when it fails and either deal with
the requests [by issuing an 'offline' response?] or arrange for the server to
be restarted (and/or upgraded).

In one of your original posts:

> "I want to keep my "rpc" queue alive even when the RPC server software is
> being upgraded so that users will only see a delay in the responses (and no
> lost requests)"

it looks like you could actually schedule a takeover of the 'offline' server
from the RpcServer, and have complete control of the policy for handling
requests in 'dead' server queues.

A dead-letter mechanism would be convenient in this case, because you would not
then have to 'take-over' any queues and the 'offline' server could be run
permanently, without a need to monitor the servers for failure.

Have I understood now?

Steve Powell  (a learning bunny)
----------some more definitions from the SPD----------
avoirdupois (phr.) 'Would you like peas with that?'
distribute (v.) To denigrate an award ceremony.
definite (phr.) 'It's hard of hearing, I think.'
modest (n.) The most mod.

On 5 Dec 2011, at 02:48, Alex Grönholm wrote:

> Dear Steve,
>
> I know how to create queues properly. That was never the issue. The issue was that if I created a leased queue, take the consumer offline and leave it that way and the queue expires, the client is never notified that the message was dropped and it will continue to wait for a response forever. That is not how I want it to work. I expect the server to send some return message when the queue dies. The mandatory flag does not help since it only returns the message if it's not routed to a queue (which it is). But like I said before, the dead lettering feature should help with this.
> _______________________________________________
> rabbitmq-discuss mailing list
> [hidden email]
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Message timeouts

Alex Grönholm-3
Sorry for taking this long to respond. I've been otherwise engaged and forgot about this conversation.
Thank you for your replies.

If you don't want the server queue requests to be 'lost' then don't allow the
queues to expire!

On the other hand your clients apparently wait forever.  Is there a reason I
don't understand why the server queues expire but the clients don't have a
timeout for the responses?

If the upgrade of the server software fails and the developers do nothing, the queue expires eventually so that stale requests are dropped and the clients will start getting proper error messages informing them of this condition.
This is not the ideal solution of course (per-message TTL and return notification would be).

You want requests to either be processed or (reliably) not processed.  You
never want the clients to be in any doubt about whether the request was
processed or not.  So you wait forever in the client.  However, requests that
get discarded never return so the client doesn't know.  So you want expiry to
trigger replies to the clients.  (Per-queue message TTLs also don't return.)

 Sounds about right.

Let's see -- how would we do this without a dead-letter mechanism?  Well, apart
from making the server transactional, it might be useful to have an 'offline'
server that can take over any server's queue when it fails and either deal with
the requests [by issuing an 'offline' response?] or arrange for the server to
be restarted (and/or upgraded).

This seems fairly complicated. I'll have to look into transactions to see if they will help in my case.

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Message timeouts

Alex Grönholm-3
11.12.2011 14:14, Alex Grönholm kirjoitti:
Sorry for taking this long to respond. I've been otherwise engaged and forgot about this conversation.
Thank you for your replies.

If you don't want the server queue requests to be 'lost' then don't allow the
queues to expire!

On the other hand your clients apparently wait forever.  Is there a reason I
don't understand why the server queues expire but the clients don't have a
timeout for the responses?

If the upgrade of the server software fails and the developers do nothing, the queue expires eventually so that stale requests are dropped and the clients will start getting proper error messages informing them of this condition.
This is not the ideal solution of course (per-message TTL and return notification would be).

You want requests to either be processed or (reliably) not processed.  You
never want the clients to be in any doubt about whether the request was
processed or not.  So you wait forever in the client.  However, requests that
get discarded never return so the client doesn't know.  So you want expiry to
trigger replies to the clients.  (Per-queue message TTLs also don't return.)

 Sounds about right.

Let's see -- how would we do this without a dead-letter mechanism?
Now that the dead-letter mechanism is part of the server, I don't still don't see how this could be done. The dead lettered RPC commands would still have to be processed somewhere.
 Well, apart
from making the server transactional, it might be useful to have an 'offline'
server that can take over any server's queue when it fails and either deal with
the requests [by issuing an 'offline' response?] or arrange for the server to
be restarted (and/or upgraded).

This seems fairly complicated. I'll have to look into transactions to see if they will help in my case.


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Message timeouts

Matthias Radestock-3
Alex,

On 26/04/12 06:08, Alex Grönholm wrote:

>> You want requests to either be processed or (reliably) not
>> processed. You never want the clients to be in any doubt about
>> whether the request was processed or not. So you wait forever in
>> the client. However, requests that get discarded never return so
>> the client doesn't know. So you want expiry to trigger replies to
>> the clients. (Per-queue message TTLs also don't return.) >>
>>
>>  Sounds about right.
>>
>> Let's see -- how would we do this without a dead-letter mechanism? >>
> Now that the dead-letter mechanism is part of the server, I don't still
> don't see how this could be done. The dead lettered RPC commands would
> still have to be processed somewhere.

Right. Ideally you'd need a way to specify a dead-letter destination
*per message*. However, working with what is possible today, instead you
could just have a single "dead request" queue to which all expired
requests get routed. And then have a small app simply consuming messages
from that queue and replying to them with some sort of error indication.

Regards,

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Message timeouts

Alex Grönholm-3
30.04.2012 19:59, Matthias Radestock kirjoitti:

> Alex,
>
> On 26/04/12 06:08, Alex Grönholm wrote:
>>> You want requests to either be processed or (reliably) not
>>> processed. You never want the clients to be in any doubt about
>>> whether the request was processed or not. So you wait forever in
>>> the client. However, requests that get discarded never return so
>>> the client doesn't know. So you want expiry to trigger replies to
>>> the clients. (Per-queue message TTLs also don't return.) >>
>>>
>>>  Sounds about right.
>>>
>>> Let's see -- how would we do this without a dead-letter mechanism? >>
>> Now that the dead-letter mechanism is part of the server, I don't still
>> don't see how this could be done. The dead lettered RPC commands would
>> still have to be processed somewhere.
>
> Right. Ideally you'd need a way to specify a dead-letter destination
> *per message*. However, working with what is possible today, instead
> you could just have a single "dead request" queue to which all expired
> requests get routed. And then have a small app simply consuming
> messages from that queue and replying to them with some sort of error
> indication.
That did occur to me, but if I was able to launch an "answering machine"
app, why not the real one too while I'm at it?

Thanks for the answer, but is there any hope for a real solution? Could
this be cleanly done with a plugin?
>
> Regards,
>
> Matthias.

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Message timeouts

Matthias Radestock-3
Alex,

On 01/05/12 23:38, Alex Grönholm wrote:

> 30.04.2012 19:59, Matthias Radestock kirjoitti:
>> Right. Ideally you'd need a way to specify a dead-letter destination
>> *per message*. However, working with what is possible today, instead
>> you could just have a single "dead request" queue to which all expired
>> requests get routed. And then have a small app simply consuming
>> messages from that queue and replying to them with some sort of error
>> indication.
> That did occur to me, but if I was able to launch an "answering machine"
> app, why not the real one too while I'm at it?
>
> Thanks for the answer, but is there any hope for a real solution? Could
> this be cleanly done with a plugin?

Yep. It could be done cleanly with a plugin that employs the Erlang
client in 'direct' mode to implement said "answering machine" app.

Of course if you take that approach to the extreme then you'd write your
*entire* app as a plugin ;)

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Message timeouts

Alex Grönholm-3
02.05.2012 08:26, Matthias Radestock kirjoitti:

> Alex,
>
> On 01/05/12 23:38, Alex Grönholm wrote:
>> 30.04.2012 19:59, Matthias Radestock kirjoitti:
>>> Right. Ideally you'd need a way to specify a dead-letter destination
>>> *per message*. However, working with what is possible today, instead
>>> you could just have a single "dead request" queue to which all expired
>>> requests get routed. And then have a small app simply consuming
>>> messages from that queue and replying to them with some sort of error
>>> indication.
>> That did occur to me, but if I was able to launch an "answering machine"
>> app, why not the real one too while I'm at it?
>>
>> Thanks for the answer, but is there any hope for a real solution? Could
>> this be cleanly done with a plugin?
>
> Yep. It could be done cleanly with a plugin that employs the Erlang
> client in 'direct' mode to implement said "answering machine" app.
>
> Of course if you take that approach to the extreme then you'd write
> your *entire* app as a plugin ;)
>
> Matthias.
No thanks, I've looked into the Erlang language and while I like many of
the ideas and patterns it employs, Python's syntax still wins big time
for its power and efficiency.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Loading...