ANN New project QDB: Persistent Message Queues With Replay

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

ANN New project QDB: Persistent Message Queues With Replay

David Tinker
QDB provides persistent message queues with the ability to efficiently
replay old messages by timestamp or id. It can push messages from its
queues directly to RabbitMQ and has simple REST interface.

http://qdb.io/

QDB is much more efficient than stuffing messages into a database
(MongoDB, MySQL etc.) before sending them to RabbitMQ for processing.

Cheers
David
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: ANN New project QDB: Persistent Message Queues With Replay

Tom Anderson
On 01/08/13 11:44, David Tinker wrote:
QDB provides persistent message queues with the ability to efficiently
replay old messages by timestamp or id. It can push messages from its
queues directly to RabbitMQ and has simple REST interface.

http://qdb.io/

Interesting. Thanks!

Can you feed it from RabbitMQ too? If not, any plans to make that possible? A replayable recent history of certain queues could be insanely useful for us. If not, it looks like it wouldn't be that hard for us to add if we wanted it.

A few minor remarks:

- "QDB depends on two libraries that are not yet on Maven central ... These need to be built and installed in your local Maven repository before building QDB." - have you considered putting artifacts for these up in a non-Central repository? I do that via a repo hosted on my Bitbucket site - see https://bitbucket.org/twic/twic.bitbucket.org/src and https://bitbucket.org/twic/twic.bitbucket.org/src/default/release.sh for the script that updates it - and find it fairly easy. I think this is even easier on GitHub.

- Building and installing locally is rendered less than entirely smooth by (a) the fact that both projects use the signing plugin, which means i would have to set up GPG etc just to install locally (i commented out the signing declarations instead), (b) qdb-kvstore refers to a repo with url "/Users/david/.m2/repository", (c) qdb-kvstore depends on version 0.4.0-SNAPSHOT of qdb-buffer, but if you check out and build qdb-buffer, you make version 0.4.1-SNAPSHOT.

- The qdb-server build at revision c5c7ac3b7a11 has one failing test, because this:

    def "Format timestamp works"() {
        Date d = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSSZ").parse("2013-06-16T21:04:32.123+0200")
        String s = DateTimeParser.INSTANCE.formatTimestamp(d)

        expect:
        s == "2013-06-16T21:04:32.123+0200"
    }

Won't pass if you don't live in South Africa, or somewhere else with a +0200 timezone.

And some handwaving:

- If you ever fancy using a different storage engine, you might be interested in http://www.mapdb.org/ . Not sure if it has the event notification, though, nor if its performance characteristics are really what you need.

- For qdb-buffer, it might be helpful to write a comparison between what seems like your main use case (buffering messages heading towards RabbitMQ to protect against RabbitMQ doing down) and running a local RabbitMQ on every box as a first hop.

tom

--

Tom Anderson | Developer | +44 20 7826 4312 | timgroup.com

STATEMENT OF CONFIDENTIALITY: The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain confidential or privileged information. If you are not the intended recipient, please notify Tom Anderson at TIM Group at [hidden email] and destroy all copies of this message and any attachments.

TIM Group is the trading name for YouDevise Limited. YouDevise Limited is registered in England, No. 3331176. Registered office: 3 Copthall Avenue, London, EC2R 7BH.


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: ANN New project QDB: Persistent Message Queues With Replay

David Tinker
Thanks for all the comments and testing. I will fix the build scripts
for qdb-buffer and qdb-kvstore and sort out the Maven stuff and the
failing test.

Yes I definitely do plan to add support for pulling messages from
RabbitMQ queues as well. We need that here at work.

As for the QDB vs local RabbitMQ use-case thats a good idea.



On Thu, Aug 1, 2013 at 2:00 PM, Tom Anderson <[hidden email]> wrote:

> On 01/08/13 11:44, David Tinker wrote:
>
> QDB provides persistent message queues with the ability to efficiently
> replay old messages by timestamp or id. It can push messages from its
> queues directly to RabbitMQ and has simple REST interface.
>
> http://qdb.io/
>
>
> Interesting. Thanks!
>
> Can you feed it from RabbitMQ too? If not, any plans to make that possible?
> A replayable recent history of certain queues could be insanely useful for
> us. If not, it looks like it wouldn't be that hard for us to add if we
> wanted it.
>
> A few minor remarks:
>
> - "QDB depends on two libraries that are not yet on Maven central ... These
> need to be built and installed in your local Maven repository before
> building QDB." - have you considered putting artifacts for these up in a
> non-Central repository? I do that via a repo hosted on my Bitbucket site -
> see https://bitbucket.org/twic/twic.bitbucket.org/src and
> https://bitbucket.org/twic/twic.bitbucket.org/src/default/release.sh for the
> script that updates it - and find it fairly easy. I think this is even
> easier on GitHub.
>
> - Building and installing locally is rendered less than entirely smooth by
> (a) the fact that both projects use the signing plugin, which means i would
> have to set up GPG etc just to install locally (i commented out the signing
> declarations instead), (b) qdb-kvstore refers to a repo with url
> "/Users/david/.m2/repository", (c) qdb-kvstore depends on version
> 0.4.0-SNAPSHOT of qdb-buffer, but if you check out and build qdb-buffer, you
> make version 0.4.1-SNAPSHOT.
>
> - The qdb-server build at revision c5c7ac3b7a11 has one failing test,
> because this:
>
>     def "Format timestamp works"() {
>         Date d = new
> SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSSZ").parse("2013-06-16T21:04:32.123+0200")
>         String s = DateTimeParser.INSTANCE.formatTimestamp(d)
>
>         expect:
>         s == "2013-06-16T21:04:32.123+0200"
>     }
>
> Won't pass if you don't live in South Africa, or somewhere else with a +0200
> timezone.
>
> And some handwaving:
>
> - If you ever fancy using a different storage engine, you might be
> interested in http://www.mapdb.org/ . Not sure if it has the event
> notification, though, nor if its performance characteristics are really what
> you need.
>
> - For qdb-buffer, it might be helpful to write a comparison between what
> seems like your main use case (buffering messages heading towards RabbitMQ
> to protect against RabbitMQ doing down) and running a local RabbitMQ on
> every box as a first hop.
>
> tom
>
> --
>
> Tom Anderson | Developer | +44 20 7826 4312 | timgroup.com
>
> STATEMENT OF CONFIDENTIALITY: The information contained in this electronic
> message and any attachments to this message are intended for the exclusive
> use of the addressee(s) and may contain confidential or privileged
> information. If you are not the intended recipient, please notify Tom
> Anderson at TIM Group at [hidden email] and destroy all copies of
> this message and any attachments.
>
> TIM Group is the trading name for YouDevise Limited. YouDevise Limited is
> registered in England, No. 3331176. Registered office: 3 Copthall Avenue,
> London, EC2R 7BH.
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> [hidden email]
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>



--
http://twoogleplus.com/ Online service to cross-post tweets to Google+
http://engage.calibreapps.com/ Engage - Web based goal setting and
performance management
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: ANN New project QDB: Persistent Message Queues With Replay

David Tinker
I have fixed all of the assorted build problems (version issue,
signing required etc.) and the "failing outside of GMT+2" test.

With any luck the qdb-buffer and qdb-kvstore dependencies will also be
on Maven Central in a couple of hours.

I should have time to implement queue "inputs" this weekend. The first
of which will have type=rabbitmq and will pull messages from a
RabbitMQ queue.

On Thu, Aug 1, 2013 at 2:32 PM, David Tinker <[hidden email]> wrote:

> Thanks for all the comments and testing. I will fix the build scripts
> for qdb-buffer and qdb-kvstore and sort out the Maven stuff and the
> failing test.
>
> Yes I definitely do plan to add support for pulling messages from
> RabbitMQ queues as well. We need that here at work.
>
> As for the QDB vs local RabbitMQ use-case thats a good idea.
>
>
>
> On Thu, Aug 1, 2013 at 2:00 PM, Tom Anderson <[hidden email]> wrote:
>> On 01/08/13 11:44, David Tinker wrote:
>>
>> QDB provides persistent message queues with the ability to efficiently
>> replay old messages by timestamp or id. It can push messages from its
>> queues directly to RabbitMQ and has simple REST interface.
>>
>> http://qdb.io/
>>
>>
>> Interesting. Thanks!
>>
>> Can you feed it from RabbitMQ too? If not, any plans to make that possible?
>> A replayable recent history of certain queues could be insanely useful for
>> us. If not, it looks like it wouldn't be that hard for us to add if we
>> wanted it.
>>
>> A few minor remarks:
>>
>> - "QDB depends on two libraries that are not yet on Maven central ... These
>> need to be built and installed in your local Maven repository before
>> building QDB." - have you considered putting artifacts for these up in a
>> non-Central repository? I do that via a repo hosted on my Bitbucket site -
>> see https://bitbucket.org/twic/twic.bitbucket.org/src and
>> https://bitbucket.org/twic/twic.bitbucket.org/src/default/release.sh for the
>> script that updates it - and find it fairly easy. I think this is even
>> easier on GitHub.
>>
>> - Building and installing locally is rendered less than entirely smooth by
>> (a) the fact that both projects use the signing plugin, which means i would
>> have to set up GPG etc just to install locally (i commented out the signing
>> declarations instead), (b) qdb-kvstore refers to a repo with url
>> "/Users/david/.m2/repository", (c) qdb-kvstore depends on version
>> 0.4.0-SNAPSHOT of qdb-buffer, but if you check out and build qdb-buffer, you
>> make version 0.4.1-SNAPSHOT.
>>
>> - The qdb-server build at revision c5c7ac3b7a11 has one failing test,
>> because this:
>>
>>     def "Format timestamp works"() {
>>         Date d = new
>> SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSSZ").parse("2013-06-16T21:04:32.123+0200")
>>         String s = DateTimeParser.INSTANCE.formatTimestamp(d)
>>
>>         expect:
>>         s == "2013-06-16T21:04:32.123+0200"
>>     }
>>
>> Won't pass if you don't live in South Africa, or somewhere else with a +0200
>> timezone.
>>
>> And some handwaving:
>>
>> - If you ever fancy using a different storage engine, you might be
>> interested in http://www.mapdb.org/ . Not sure if it has the event
>> notification, though, nor if its performance characteristics are really what
>> you need.
>>
>> - For qdb-buffer, it might be helpful to write a comparison between what
>> seems like your main use case (buffering messages heading towards RabbitMQ
>> to protect against RabbitMQ doing down) and running a local RabbitMQ on
>> every box as a first hop.
>>
>> tom
>>
>> --
>>
>> Tom Anderson | Developer | +44 20 7826 4312 | timgroup.com
>>
>> STATEMENT OF CONFIDENTIALITY: The information contained in this electronic
>> message and any attachments to this message are intended for the exclusive
>> use of the addressee(s) and may contain confidential or privileged
>> information. If you are not the intended recipient, please notify Tom
>> Anderson at TIM Group at [hidden email] and destroy all copies of
>> this message and any attachments.
>>
>> TIM Group is the trading name for YouDevise Limited. YouDevise Limited is
>> registered in England, No. 3331176. Registered office: 3 Copthall Avenue,
>> London, EC2R 7BH.
>>
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> [hidden email]
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>
>
>
> --
> http://twoogleplus.com/ Online service to cross-post tweets to Google+
> http://engage.calibreapps.com/ Engage - Web based goal setting and
> performance management



--
http://twoogleplus.com/ Online service to cross-post tweets to Google+
http://engage.calibreapps.com/ Engage - Web based goal setting and
performance management
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss