Age of message

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

Age of message

cw storm
Is it possible to get the age of the message from the queue?  I want to get a total count of message that are x old.

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Age of message

cw storm
We want to get an alert when messages haven't been ack since the time it arrived in the queue.  


On Thu, Mar 20, 2014 at 3:38 PM, cw storm <[hidden email]> wrote:
Is it possible to get the age of the message from the queue?  I want to get a total count of message that are x old.


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Age of message

Alvaro Videla-2
On Thu, Mar 20, 2014 at 8:49 PM, cw storm <[hidden email]> wrote:
> We want to get an alert when messages haven't been ack since the time it
> arrived in the queue.

I don't think RabbitMQ could give you that functionality. In any case,
have you looked into TTLs? http://www.rabbitmq.com/ttl.html
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Age of message

cw storm
TTL applies to expiring the message.  I don't want to set an expiration on the message.  What we're trying to do is report on stale messages.  Trying to find a reasonable design or approach on this topic.  Interested in how everyone does it.


On Thu, Mar 20, 2014 at 3:54 PM, Alvaro Videla <[hidden email]> wrote:
On Thu, Mar 20, 2014 at 8:49 PM, cw storm <[hidden email]> wrote:
> We want to get an alert when messages haven't been ack since the time it
> arrived in the queue.

I don't think RabbitMQ could give you that functionality. In any case,
have you looked into TTLs? http://www.rabbitmq.com/ttl.html
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Age of message

Alvaro Videla-2
If you are not going to process the staled messages, then if they have
TTL set, they will be dead lettered, by setting up a queue on that
dead letter exchange, you could then report on those "staled"
messages.

RabbitMQ doesn't support browsing a queue and selecting message with
certain age.

On Thu, Mar 20, 2014 at 9:07 PM, cw storm <[hidden email]> wrote:

> TTL applies to expiring the message.  I don't want to set an expiration on
> the message.  What we're trying to do is report on stale messages.  Trying
> to find a reasonable design or approach on this topic.  Interested in how
> everyone does it.
>
>
> On Thu, Mar 20, 2014 at 3:54 PM, Alvaro Videla <[hidden email]>
> wrote:
>>
>> On Thu, Mar 20, 2014 at 8:49 PM, cw storm <[hidden email]> wrote:
>> > We want to get an alert when messages haven't been ack since the time it
>> > arrived in the queue.
>>
>> I don't think RabbitMQ could give you that functionality. In any case,
>> have you looked into TTLs? http://www.rabbitmq.com/ttl.html
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> [hidden email]
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> [hidden email]
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Age of message

cw storm
I guess the client could include a timestamp during the publish and put it in a header attribute.  The consumer can use that information and match that against the current date time.

You see, there could be an issue with the client not setting the acknowledge flag so that the message can be remove from the queue.  What happens is that all other messages are back up until that one message is acknowledge by the client.  At that time, anything can happen with the environment.  The client server could be unavailable or that some other components is working overtime which prevent the consumer service to complete processing it correctly.  If everything goes well, then the last step is to set that flag and the message will either be acknowledge or rejected.  If rejected, it'll go back to the ready state and will process again until it's acknowledge.  So, I'm trying to come up with a good approach on reporting "stale" messages. I could get the total messages waiting in the queue and if it's greater than say 100, then send an alert.  But during runtime, one could get 1k messages in a queue and it just take time to process so it's not really "stale".  But I guess this would be good for now so that support person can phyiscially monitor it.


On Thu, Mar 20, 2014 at 4:24 PM, Alvaro Videla <[hidden email]> wrote:
If you are not going to process the staled messages, then if they have
TTL set, they will be dead lettered, by setting up a queue on that
dead letter exchange, you could then report on those "staled"
messages.

RabbitMQ doesn't support browsing a queue and selecting message with
certain age.

On Thu, Mar 20, 2014 at 9:07 PM, cw storm <[hidden email]> wrote:
> TTL applies to expiring the message.  I don't want to set an expiration on
> the message.  What we're trying to do is report on stale messages.  Trying
> to find a reasonable design or approach on this topic.  Interested in how
> everyone does it.
>
>
> On Thu, Mar 20, 2014 at 3:54 PM, Alvaro Videla <[hidden email]>
> wrote:
>>
>> On Thu, Mar 20, 2014 at 8:49 PM, cw storm <[hidden email]> wrote:
>> > We want to get an alert when messages haven't been ack since the time it
>> > arrived in the queue.
>>
>> I don't think RabbitMQ could give you that functionality. In any case,
>> have you looked into TTLs? http://www.rabbitmq.com/ttl.html
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> [hidden email]
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> [hidden email]
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Age of message

Michael Klishin-2

On 21 Mar 2014, at 00:49, cw storm <[hidden email]> wrote:

> I guess the client could include a timestamp during the publish and put it in a header attribute.  The consumer can use that information and match that against the current date time.

There is a message property called “timestamp”. Note that publishers and consumers then should synchronise
their clocks using NTP or similar.

MK

Software Engineer, Pivotal/RabbitMQ


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Age of message

cw storm
Thanks MK.  Is the below client to perform the synchronization easier?



On Thu, Mar 20, 2014 at 9:28 PM, Michael Klishin <[hidden email]> wrote:

On 21 Mar 2014, at 00:49, cw storm <[hidden email]> wrote:

> I guess the client could include a timestamp during the publish and put it in a header attribute.  The consumer can use that information and match that against the current date time.

There is a message property called “timestamp”. Note that publishers and consumers then should synchronise
their clocks using NTP or similar.

MK

Software Engineer, Pivotal/RabbitMQ


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Age of message

Michael Klishin-2

On 21 Mar 2014, at 16:32, cw storm <[hidden email]> wrote:

> Is the below client to perform the synchronization easier?

Your *app* should not perform time synchronisation. NTP daemon should, in concert
with OS and a pool of NTP servers worldwide.

See http://en.wikipedia.org/wiki/Network_Time_Protocol.


MK

Software Engineer, Pivotal/RabbitMQ


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Age of message

cw storm
In reply to this post by cw storm


On Fri, Mar 21, 2014 at 8:32 AM, cw storm <[hidden email]> wrote:
Thanks MK.  Is the below client to perform the synchronization easier?



On Thu, Mar 20, 2014 at 9:28 PM, Michael Klishin <[hidden email]> wrote:

On 21 Mar 2014, at 00:49, cw storm <[hidden email]> wrote:

> I guess the client could include a timestamp during the publish and put it in a header attribute.  The consumer can use that information and match that against the current date time.

There is a message property called “timestamp”. Note that publishers and consumers then should synchronise
their clocks using NTP or similar.

MK

Software Engineer, Pivotal/RabbitMQ


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss



_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Age of message

cw storm
MK,

So you're saying that the OS has to sync with the NTP server.  Once that's confirm, then the message property of "timestamp" can be use to compare to the local time on the client server, right?


On Fri, Mar 21, 2014 at 8:41 AM, cw storm <[hidden email]> wrote:


On Fri, Mar 21, 2014 at 8:32 AM, cw storm <[hidden email]> wrote:
Thanks MK.  Is the below client to perform the synchronization easier?



On Thu, Mar 20, 2014 at 9:28 PM, Michael Klishin <[hidden email]> wrote:

On 21 Mar 2014, at 00:49, cw storm <[hidden email]> wrote:

> I guess the client could include a timestamp during the publish and put it in a header attribute.  The consumer can use that information and match that against the current date time.

There is a message property called “timestamp”. Note that publishers and consumers then should synchronise
their clocks using NTP or similar.

MK

Software Engineer, Pivotal/RabbitMQ


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss




_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Age of message

cw storm
Our requirement does not have to be very accurate in terms of the date time.  Just as long as we have a date time to capture and compare that data time with the local time on the client server during the consumption process.


On Fri, Mar 21, 2014 at 8:46 AM, cw storm <[hidden email]> wrote:
MK,

So you're saying that the OS has to sync with the NTP server.  Once that's confirm, then the message property of "timestamp" can be use to compare to the local time on the client server, right?


On Fri, Mar 21, 2014 at 8:41 AM, cw storm <[hidden email]> wrote:


On Fri, Mar 21, 2014 at 8:32 AM, cw storm <[hidden email]> wrote:
Thanks MK.  Is the below client to perform the synchronization easier?



On Thu, Mar 20, 2014 at 9:28 PM, Michael Klishin <[hidden email]> wrote:

On 21 Mar 2014, at 00:49, cw storm <[hidden email]> wrote:

> I guess the client could include a timestamp during the publish and put it in a header attribute.  The consumer can use that information and match that against the current date time.

There is a message property called “timestamp”. Note that publishers and consumers then should synchronise
their clocks using NTP or similar.

MK

Software Engineer, Pivotal/RabbitMQ


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss





_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Age of message

Michael Klishin-2
In reply to this post by cw storm

On 21 Mar 2014, at 16:46, cw storm <[hidden email]> wrote:

> So you're saying that the OS has to sync with the NTP server.  Once that's confirm, then the message property of "timestamp" can be use to compare to the local time on the client server, right?

After that you still can get inaccurate times from some machines but the possibility
of that is greatly reduced. Syncing over NTP happens periodically. For a lot of apps,
syncing once every few hours is good enough.

MK

Software Engineer, Pivotal/RabbitMQ


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Age of message

cw storm
MK,

In order to use the timestamp of the message property, don't you have to issue the "setTimestamp" to populate it? If so, isn't setTimestamp deprecated?


On Fri, Mar 21, 2014 at 8:51 AM, Michael Klishin <[hidden email]> wrote:

On 21 Mar 2014, at 16:46, cw storm <[hidden email]> wrote:

> So you're saying that the OS has to sync with the NTP server.  Once that's confirm, then the message property of "timestamp" can be use to compare to the local time on the client server, right?

After that you still can get inaccurate times from some machines but the possibility
of that is greatly reduced. Syncing over NTP happens periodically. For a lot of apps,
syncing once every few hours is good enough.

MK

Software Engineer, Pivotal/RabbitMQ


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Age of message

Matthias Radestock-3
On 24/03/14 15:31, cw storm wrote:
> In order to use the timestamp of the message property, don't you have to
> issue the "*setTimestamp
> <http://www.rabbitmq.com/releases/rabbitmq-java-client/v3.1.3/rabbitmq-java-client-javadoc-3.1.3/com/rabbitmq/client/AMQP.BasicProperties.html#setTimestamp(java.util.Date)>"
> to populate it? If so, isn't **setTimestamp
> <http://www.rabbitmq.com/releases/rabbitmq-java-client/v3.1.3/rabbitmq-java-client-javadoc-3.1.3/com/rabbitmq/client/AMQP.BasicProperties.html#setTimestamp(java.util.Date)> deprecated?*

You are meant to use the Builder classes, as described in
http://www.rabbitmq.com/api-guide.html#classoverview

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Age of message

cw storm
So, something like props.Timestamp = DateTime.Now

However, during the consumption process, is it easy enough to compare the timestamp value?  I'm guessing that's why you mentioned that the client local server needs to be in sync with the NTP, right?


On Mon, Mar 24, 2014 at 11:38 AM, Matthias Radestock <[hidden email]> wrote:
On 24/03/14 15:31, cw storm wrote:
In order to use the timestamp of the message property, don't you have to
issue the "*setTimestamp
<http://www.rabbitmq.com/releases/rabbitmq-java-client/v3.1.3/rabbitmq-java-client-javadoc-3.1.3/com/rabbitmq/client/AMQP.BasicProperties.html#setTimestamp(java.util.Date)>"
to populate it? If so, isn't **setTimestamp
<http://www.rabbitmq.com/releases/rabbitmq-java-client/v3.1.3/rabbitmq-java-client-javadoc-3.1.3/com/rabbitmq/client/AMQP.BasicProperties.html#setTimestamp(java.util.Date)> deprecated?*

You are meant to use the Builder classes, as described in http://www.rabbitmq.com/api-guide.html#classoverview

Matthias.

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Age of message

Michael Klishin-2
 On 24 March 2014 at 21:11:12, cw storm ([hidden email]) wrote:
> > However, during the consumption process, is it easy enough  
> to compare the timestamp value?

About as easy as comparing two dates.
--  
MK  

Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Age of message

cw storm
MK,

Is it safe to compare whatever value in the timestamp property since it's store as a 64-bit time format? or do I have to painly go through converting it to an actual date time and comparing it?  I'm leaning toward the former.


On Mon, Mar 24, 2014 at 1:22 PM, Michael Klishin <[hidden email]> wrote:
 On 24 March 2014 at 21:11:12, cw storm ([hidden email]) wrote:
> > However, during the consumption process, is it easy enough
> to compare the timestamp value?

About as easy as comparing two dates.
--
MK

Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Age of message

Michael Klishin-2
 On 25 March 2014 at 20:03:09, cw storm ([hidden email]) wrote:
> > Is it safe to compare whatever value in the timestamp property  
> since it's store as a 64-bit time format?

I don’t see how it can be unsafe, barring the aforementioned
clock synchronisation issue.
--  
MK  

Software Engineer, Pivotal/RabbitMQ
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss