Quantcast

Confirmation IDs

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Confirmation IDs

Attila Nagy
Hi,

According to https://www.rabbitmq.com/confirms.html:
"Once a channel is in confirm mode, both the broker and the client count
messages (counting starts at 1 on the first confirm.select)."

Which seems to be easy for the first sight, but I think it may
unnecessarily increase complexity on the client side.

My case: I process a file, containing multiple records. I do this in a
FIFO manner: reading the file from its head towards its tail, sending
each record with AMQP and waiting for confirms.
The problem here is that I have to keep a local delivery tag to file
pointer table, so I know which delivery tag means which file pointer.

I think it would be much easier and flexible if the confirmation system
would allow data to be "inherited" from the original message, like its
properties or headers.
Something like receipts in STOMP.

That way the state information would be in the broker and the client
code would be clearer.

What do you think?
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Confirmation IDs

Michael Klishin-2
On 27 July 2014 at 15:54:44, Nagy, Attila ([hidden email]) wrote:
> > My case: I process a file, containing multiple records. I do  
> this in a
> FIFO manner: reading the file from its head towards its tail,  
> sending
> each record with AMQP and waiting for confirms.
> The problem here is that I have to keep a local delivery tag to file  
> pointer table, so I know which delivery tag means which file pointer.  

Yes, you need to correlate messages with delivery tags.

> I think it would be much easier and flexible if the confirmation  
> system
> would allow data to be "inherited" from the original message,  
> like its
> properties or headers.

So, calculated from the message? That won't work very well if you publish
exactly the same message N times in a row.

Either way, it is probably too late to change how publisher confirms work.
We can't just break things for all the existing users out there. 
--  
MK  

Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Confirmation IDs

Attila Nagy
On 07/27/14 16:25, Michael Klishin wrote:
> On 27 July 2014 at 15:54:44, Nagy, Attila ([hidden email]) wrote:
>>> My case: I process a file, containing multiple records. I do
>> this in a
>> FIFO manner: reading the file from its head towards its tail,
>> sending
>> each record with AMQP and waiting for confirms.
>> The problem here is that I have to keep a local delivery tag to file
>> pointer table, so I know which delivery tag means which file pointer.
> Yes, you need to correlate messages with delivery tags.
Yeah, that's my problem. BTW, are confirmations guaranteed to be in the
order the original messages have been submitted?
I can see no information about that, except the existence of multiple
field, which I guess would be somewhat meaningless if the confirmations
wouldn't be in original order.
>
>> I think it would be much easier and flexible if the confirmation
>> system
>> would allow data to be "inherited" from the original message,
>> like its
>> properties or headers.
> So, calculated from the message? That won't work very well if you publish
> exactly the same message N times in a row.
Nope, from the record's identifier. I have an incoming task queue (for
example a file which is appended with records, or a database which has
rows), from which I can have a unique identifier which is always
available and can directly point to the record.
So counting (because publish doesn't give it back) the sent messages
(and resetting the counter on reconnects) and maintaining
counter->original message pointers are unnecessary indirections to
achieve this functionality.

BTW, RabbitMQ's STOMP plugin does this, you can set a receipt string and
the message will be acknowledged back with that string, whatever it is.
But STOMP is not really an option if you want to do serious things in
RabbitMQ. :)

>
> Either way, it is probably too late to change how publisher confirms work
Isn't it possible to add additional info in the ACK frame without
breaking current clients?
Or making it only appear when publish has a given pattern? (like with STOMP)

Thanks,
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Confirmation IDs

Michael Klishin-2
On 27 July 2014 at 23:01:12, Nagy, Attila ([hidden email]) wrote:
> > Yeah, that's my problem. BTW, are confirmations guaranteed  
> to be in the
> order the original messages have been submitted?

Yes.

> Nope, from the record's identifier. I have an incoming task queue  
> (for
> example a file which is appended with records, or a database which  
> has
> rows), from which I can have a unique identifier which is always  
> available and can directly point to the record.
> So counting (because publish doesn't give it back) the sent messages  
> (and resetting the counter on reconnects) and maintaining
> counter->original message pointers are unnecessary indirections  
> to
> achieve this functionality.

Coming up with a good identifier for
an arbitrary message is much harder and there's no consensus on what's the best
way to do this.
Especially if you try to come up with a solution for multiple protocols.

> BTW, RabbitMQ's STOMP plugin does this, you can set a receipt  
> string and
> the message will be acknowledged back with that string, whatever  
> it is.

STOMP has this feature but AMQP 0-9-1 and MQTT do not, unfortunately.

> But STOMP is not really an option if you want to do serious things  
> in
> RabbitMQ. :)

I can't see why not.

> Isn't it possible to add additional info in the ACK frame without 
> breaking current clients?
> Or making it only appear when publish has a given pattern? (like  
> with STOMP)

For AMQP 0-9-1 clients, it is not. STOMP header values can be of arbitrary length and many
headers are optional. AMQP 0-9-1 methods have a fixed (mandatory) attributes
and only *some* have optional ones ("extra arguments", as an attribute table).
Unfortunately, basic.ack is not one of the latter group.
--  
MK  

Staff Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Confirmation IDs

Attila Nagy
On 07/27/14 21:30, Michael Klishin wrote:

>> Nope, from the record's identifier. I have an incoming task queue
>> (for
>> example a file which is appended with records, or a database which
>> has
>> rows), from which I can have a unique identifier which is always
>> available and can directly point to the record.
>> So counting (because publish doesn't give it back) the sent messages
>> (and resetting the counter on reconnects) and maintaining
>> counter->original message pointers are unnecessary indirections
>> to
>> achieve this functionality.
> Coming up with a good identifier for
> an arbitrary message is much harder and there's no consensus on what's the best
> way to do this.
> Especially if you try to come up with a solution for multiple protocols.
I see three approaches:
1. let the server and the client -independently- count the messages and
use that (the current solution for AMQP in RabbitMQ)
2. let the server count (or otherwise mark them with a unique ID) the
messages and return it to the client, so it will know about that
3. let the client specify one

I think the first has many problems, it's not flexible, the client has
to maintain translation tables and has to count exactly the same way as
the server.
This way the counter must be implemented in the lowest level (in the
AMQP library, because it's the only one which knows about the connection
state, which can reset the counter) and higher levels need to sync with
that, otherwise they won't be able to tell which messages have been
confirmed.
How do other libraries work in this regard?

Pika's publisher example for example is both wrong (it will die on
reconnects due to the counter resetting) and ugly. :)
http://pika.readthedocs.org/en/latest/examples/asynchronous_publisher_example.html

I've just made a quick peek into the java client, so I may be wrong. I
guess it has these problems too, the counter can be queried
independently of the publishing, therefore it can't be used in multi
threaded environments (well, at least you have to open one channel per
thread, or put a lock on the client so there won't be a race condition
between querying the counter and sending the message).

>> But STOMP is not really an option if you want to do serious things
>> in
>> RabbitMQ. :)
> I can't see why not.
I've tried all STOMP brokers and finally stick with RabbitMQ, because it
was the best in many ways. But as in most of the other brokers, STOMP
offers a limited control on what the broker does in the background,
which is unsuitable for a lot of use cases.
For example, for me serious means exact access control. And because
STOMP does abritrary bindings and declares, the clients can't have
strict policies, they will have always more power than it's needed with
AMQP.

>
>> Isn't it possible to add additional info in the ACK frame without
>> breaking current clients?
>> Or making it only appear when publish has a given pattern? (like
>> with STOMP)
> For AMQP 0-9-1 clients, it is not. STOMP header values can be of arbitrary length and many
> headers are optional. AMQP 0-9-1 methods have a fixed (mandatory) attributes
> and only *some* have optional ones ("extra arguments", as an attribute table).
> Unfortunately, basic.ack is not one of the latter group.
>
OK, then could you please consider the following?
With basic.publish setting a custom header replaces the message's
confirm sequence number with the header's value (with type
compatibility, so this will be a limitation, only 64(?) bit unsigned
ints could be used) and will basic.ack with that.

I mean this would be a normal publish (if confirms are turned on, it
will have the current counter as the sequence number):

         properties  =  pika.BasicProperties(app_id='example-publisher',
                                           headers={'key':'value'})

         self._channel.basic_publish(self.EXCHANGE,  self.ROUTING_KEY,
                                     message,
                                     properties)
         self._message_number  +=  1


And this will have 123456 as its sequence number:

         properties  =  pika.BasicProperties(app_id='example-publisher',
                                           headers={'key':'value', 'X-RabbitMQ-confirm-seq':123456})

         self._channel.basic_publish(self.EXCHANGE,  self.ROUTING_KEY,
                                     message,
                                     properties)

The consequences are to be borne by the client.
_______________________________________________
rabbitmq-discuss mailing list has moved to https://groups.google.com/forum/#!forum/rabbitmq-users,
please subscribe to the new list!

[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Loading...