Quantcast

Can RabbitMQ handle big messages?

classic Classic list List threaded Threaded
27 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Can RabbitMQ handle big messages?

Zabrane Mickael
Hi there,

I've a set (~1 million) of high resolution PNG files  each one between 100MB and 1GB size.
I'd like to know if RabbitMQ is capable of handling such files as messages?

What's the max allowed message size?

Thx ...

Regards,
Zabrane


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Can RabbitMQ handle big messages?

Jerry Kuch-3
Hi, Zabrane:  In theory the AMQP protocol IIRC allows crazy large message payloads (2^64 bytes I believe).  In practice though, that's madness since you end up with potential copying and buffering along the way that could make a broker very unhealthy.

For moving bulk binary data as in your app, you'd be wise to fragment at the producer and reassemble at the destination/consumer. Keeping your chunks in the 100KB range might be a decent place to start experimenting.  

Sent from my iPhone (Brevity and typos are hopefully the result of 1-fingered typing rather than rudeness or illiteracy).


On Mar 10, 2012, at 10:28 AM, Zabrane Mickael <[hidden email]> wrote:

Hi there,

I've a set (~1 million) of high resolution PNG files  each one between 100MB and 1GB size.
I'd like to know if RabbitMQ is capable of handling such files as messages?

What's the max allowed message size?

Thx ...

Regards,
Zabrane

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Can RabbitMQ handle big messages?

Zabrane Mickael
Thanks Jerry.

I'll experiment with that advice and come back soon.

Regards,
Zabrane

On Mar 10, 2012, at 7:34 PM, Jerry Kuch wrote:

Hi, Zabrane:  In theory the AMQP protocol IIRC allows crazy large message payloads (2^64 bytes I believe).  In practice though, that's madness since you end up with potential copying and buffering along the way that could make a broker very unhealthy.

For moving bulk binary data as in your app, you'd be wise to fragment at the producer and reassemble at the destination/consumer. Keeping your chunks in the 100KB range might be a decent place to start experimenting.  

Sent from my iPhone (Brevity and typos are hopefully the result of 1-fingered typing rather than rudeness or illiteracy).


On Mar 10, 2012, at 10:28 AM, Zabrane Mickael <[hidden email]> wrote:

Hi there,

I've a set (~1 million) of high resolution PNG files  each one between 100MB and 1GB size.
I'd like to know if RabbitMQ is capable of handling such files as messages?

What's the max allowed message size?

Thx ...

Regards,
Zabrane

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss



_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Can RabbitMQ handle big messages?

Zabrane Mickael
In reply to this post by Jerry Kuch-3
Hi guys,

Before trying to rewrite the wheel, anyone faced this problem before and would like to
share some code (ie. splitting a big file are constructing it when reading from RabbitMQ queue)?

Thanks ...

Regards,
Zabrane


On Mar 10, 2012, at 7:34 PM, Jerry Kuch wrote:

Hi, Zabrane:  In theory the AMQP protocol IIRC allows crazy large message payloads (2^64 bytes I believe).  In practice though, that's madness since you end up with potential copying and buffering along the way that could make a broker very unhealthy.

For moving bulk binary data as in your app, you'd be wise to fragment at the producer and reassemble at the destination/consumer. Keeping your chunks in the 100KB range might be a decent place to start experimenting.  

Sent from my iPhone (Brevity and typos are hopefully the result of 1-fingered typing rather than rudeness or illiteracy).


On Mar 10, 2012, at 10:28 AM, Zabrane Mickael <[hidden email]> wrote:

Hi there,

I've a set (~1 million) of high resolution PNG files  each one between 100MB and 1GB size.
I'd like to know if RabbitMQ is capable of handling such files as messages?

What's the max allowed message size?

Thx ...

Regards,
Zabrane

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss



_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Can RabbitMQ handle big messages?

Josh Geisser

No code avilable anymore but I used to test >=512MB messages, no problems at all.

(Except of coarse it takes ages to upload/download, and getting any %-complete is almost impossible)

 

Used py-AMQP then.

 

FYI & cheers

Josh

 


Von: [hidden email] [mailto:[hidden email]] Im Auftrag von Zabrane Mickael
Gesendet: Montag, 12. März 2012 07:33
An: Jerry Kuch
Cc: [hidden email]
Betreff: Re: [rabbitmq-discuss] Can RabbitMQ handle big messages?

 

Hi guys,

 

Before trying to rewrite the wheel, anyone faced this problem before and would like to

share some code (ie. splitting a big file are constructing it when reading from RabbitMQ queue)?

 

Thanks ...

 

Regards,

Zabrane

 

 

On Mar 10, 2012, at 7:34 PM, Jerry Kuch wrote:



Hi, Zabrane:  In theory the AMQP protocol IIRC allows crazy large message payloads (2^64 bytes I believe).  In practice though, that's madness since you end up with potential copying and buffering along the way that could make a broker very unhealthy.

 

For moving bulk binary data as in your app, you'd be wise to fragment at the producer and reassemble at the destination/consumer. Keeping your chunks in the 100KB range might be a decent place to start experimenting.  

Sent from my iPhone (Brevity and typos are hopefully the result of 1-fingered typing rather than rudeness or illiteracy).

 


On Mar 10, 2012, at 10:28 AM, Zabrane Mickael <[hidden email]> wrote:

Hi there,

 

I've a set (~1 million) of high resolution PNG files  each one between 100MB and 1GB size.

I'd like to know if RabbitMQ is capable of handling such files as messages?

 

What's the max allowed message size?

 

Thx ...

 

Regards,

Zabrane

 

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

 

 

-- 
----
ASG at hnet

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Can RabbitMQ handle big messages?

michaelklishin
In reply to this post by Zabrane Mickael
Zabrane Mickael:

> Before trying to rewrite the wheel, anyone faced this problem before and would like to
> share some code (ie. splitting a big file are constructing it when reading from RabbitMQ queue)?

You can use Nanite's file streaming implementation as example:
https://github.com/ruby-amqp/nanite/blob/master/lib/nanite/streaming.rb

Keep in mind that the project is old but the chunking part is just as relevant
today.

MK

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Can RabbitMQ handle big messages?

Zabrane Mickael
Thanks for the Nanite pointer Michael.

Regards,
Zabrane

On Mar 12, 2012, at 9:28 AM, Michael Klishin wrote:

> Zabrane Mickael:
>
>> Before trying to rewrite the wheel, anyone faced this problem before and would like to
>> share some code (ie. splitting a big file are constructing it when reading from RabbitMQ queue)?
>
> You can use Nanite's file streaming implementation as example:
> https://github.com/ruby-amqp/nanite/blob/master/lib/nanite/streaming.rb
>
> Keep in mind that the project is old but the chunking part is just as relevant
> today.
>
> MK
>

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Can RabbitMQ handle big messages?

Emile Joubert
In reply to this post by Jerry Kuch-3
Hi Zabrane,

On 10/03/12 18:34, Jerry Kuch wrote:
> Hi, Zabrane:  In theory the AMQP protocol IIRC allows crazy large
> message payloads (2^64 bytes I believe).  In practice though, that's
> madness since you end up with potential copying and buffering along the
> way that could make a broker very unhealthy.

I have successfully processed messages as large as 2Gb using RabbitMQ,
where 2Gb was about 5% of the total RAM. If the ratio between message
size and total RAM stays low then you can send even larger messages, up
to the limit Jerry mentioned.


-Emile
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Can RabbitMQ handle big messages?

Irmo Manie
RabbitMQ should actually not be used for big file transfers or only
with great care and fragmenting the files into smaller separate
messages.
When running a single broker instance, you'd still be safe, but in a
clustered setup, very big messages will break the cluster.

Clustered nodes are connected via 1 tcp connection, which must also
transport a (erlang) heartbeat. If your big message takes more time to
transfer between nodes than the heartbeat timeout (anywhere between
~20-45 seconds if I'm correct), the cluster will break and your
message is lost.

The preferred architecture for file transfer over amqp is to just send
a message with a link to a downloadable resource and let the file
transfer be handle by specialized protocol like ftp :-)


- Irmo

On Mon, Mar 12, 2012 at 1:26 PM, Emile Joubert <[hidden email]> wrote:

> Hi Zabrane,
>
> On 10/03/12 18:34, Jerry Kuch wrote:
>> Hi, Zabrane:  In theory the AMQP protocol IIRC allows crazy large
>> message payloads (2^64 bytes I believe).  In practice though, that's
>> madness since you end up with potential copying and buffering along the
>> way that could make a broker very unhealthy.
>
> I have successfully processed messages as large as 2Gb using RabbitMQ,
> where 2Gb was about 5% of the total RAM. If the ratio between message
> size and total RAM stays low then you can send even larger messages, up
> to the limit Jerry mentioned.
>
>
> -Emile
> _______________________________________________
> rabbitmq-discuss mailing list
> [hidden email]
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Can RabbitMQ handle big messages?

Zabrane Mickael
> Clustered nodes are connected via 1 tcp connection, which must also
> transport a (erlang) heartbeat. If your big message takes more time to
> transfer between nodes than the heartbeat timeout (anywhere between
> ~20-45 seconds if I'm correct), the cluster will break and your
> message is lost.

I see.

> The preferred architecture for file transfer over amqp is to just send
> a message with a link to a downloadable resource and let the file
> transfer be handle by specialized protocol like ftp :-)


Nice idea. I'll try to implement that.

Regards,
Zabrane

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Can RabbitMQ handle big messages?

carlhoerberg
is it possible to configure a max message size limit?

On Mon, Mar 12, 2012 at 14:00, Zabrane Mickael <[hidden email]> wrote:

>> Clustered nodes are connected via 1 tcp connection, which must also
>> transport a (erlang) heartbeat. If your big message takes more time to
>> transfer between nodes than the heartbeat timeout (anywhere between
>> ~20-45 seconds if I'm correct), the cluster will break and your
>> message is lost.
>
> I see.
>
>> The preferred architecture for file transfer over amqp is to just send
>> a message with a link to a downloadable resource and let the file
>> transfer be handle by specialized protocol like ftp :-)
>
>
> Nice idea. I'll try to implement that.
>
> Regards,
> Zabrane
>
> _______________________________________________
> rabbitmq-discuss mailing list
> [hidden email]
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Can RabbitMQ handle big messages?

Emile Joubert
Hi Carl,

On 12/03/12 13:23, Carl Hörberg wrote:
> is it possible to configure a max message size limit?

No, but you can limit the maximum fragment size during connection tuning.

-Emile
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Can RabbitMQ handle big messages?

carlhoerberg
if Irmo is correct, wouldn't that be considered a bug?

or can fragment size tuning prevent that? if so, what might be a decent value?

On Mon, Mar 12, 2012 at 15:13, Emile Joubert <[hidden email]> wrote:
> Hi Carl,
>
> On 12/03/12 13:23, Carl Hörberg wrote:
>> is it possible to configure a max message size limit?
>
> No, but you can limit the maximum fragment size during connection tuning.
>
> -Emile
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Can RabbitMQ handle big messages?

Tony Garnock-Jones-6
In reply to this post by Irmo Manie
On 12 March 2012 08:51, Irmo Manie <[hidden email]> wrote:
Clustered nodes are connected via 1 tcp connection, which must also
transport a (erlang) heartbeat. If your big message takes more time to
transfer between nodes than the heartbeat timeout (anywhere between
~20-45 seconds if I'm correct), the cluster will break and your
message is lost.

Wow! Is that really the case? Erlang's distribution breaks if no heartbeats have been received *even if there's traffic coming in on the wire*? Sounds like an Erlang bug. Or perhaps there's some subtlety in the design I'm not seeing!

Regards,
  Tony
--
Tony Garnock-Jones
[hidden email]
http://homepages.kcbbs.gen.nz/tonyg/

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Can RabbitMQ handle big messages?

Zabrane Mickael
It's not a bug. You have to be aware of it to build reliable systems in case of multiple connected Erlang nodes.

Regards,
Zabrane

On Mar 12, 2012, at 4:31 PM, Tony Garnock-Jones wrote:

On 12 March 2012 08:51, Irmo Manie <[hidden email]> wrote:
Clustered nodes are connected via 1 tcp connection, which must also
transport a (erlang) heartbeat. If your big message takes more time to
transfer between nodes than the heartbeat timeout (anywhere between
~20-45 seconds if I'm correct), the cluster will break and your
message is lost.

Wow! Is that really the case? Erlang's distribution breaks if no heartbeats have been received *even if there's traffic coming in on the wire*? Sounds like an Erlang bug. Or perhaps there's some subtlety in the design I'm not seeing!

Regards,
  Tony
--
Tony Garnock-Jones
[hidden email]
http://homepages.kcbbs.gen.nz/tonyg/
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss



_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Can RabbitMQ handle big messages?

Jerry Kuch-3
It still seems slightly strange though, since if an Erlang node is
actively reading bytes off a connection, it seems like that could serve
as proxy for the heartbeat, which would only really be essential when
nothing else is going on...   But there could easily be some subtlety of
the design or implementation that's eluding me...

Jerry

----- Original Message -----
From: "Zabrane Mickael" <[hidden email]>
To: "Tony Garnock-Jones" <[hidden email]>
Cc: [hidden email]
Sent: Monday, March 12, 2012 8:42:52 AM
Subject: Re: [rabbitmq-discuss] Can RabbitMQ handle big messages?





It's not a bug. You have to be aware of it to build reliable systems in case of multiple connected Erlang nodes.



Regards,
Zabrane

On Mar 12, 2012, at 4:31 PM, Tony Garnock-Jones wrote:



On 12 March 2012 08:51, Irmo Manie < [hidden email] > wrote:


Clustered nodes are connected via 1 tcp connection, which must also
transport a (erlang) heartbeat. If your big message takes more time to
transfer between nodes than the heartbeat timeout (anywhere between
~20-45 seconds if I'm correct), the cluster will break and your
message is lost.


Wow! Is that really the case? Erlang's distribution breaks if no heartbeats have been received *even if there's traffic coming in on the wire*? Sounds like an Erlang bug. Or perhaps there's some subtlety in the design I'm not seeing!

Regards,
Tony
--
Tony Garnock-Jones
[hidden email]
http://homepages.kcbbs.gen.nz/tonyg/ 
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss 





_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Can RabbitMQ handle big messages?

Matthew Sackman-3
In reply to this post by Zabrane Mickael
On Mon, Mar 12, 2012 at 04:42:52PM +0100, Zabrane Mickael wrote:
> It's not a bug. You have to be aware of it to build reliable systems in case of multiple connected Erlang nodes.

For those of us struggling to follow this, if you're currently in the
act of receiving data from node X, why can't you assume node X is still
alive? I.e. what is wrong with treating arbitrary data from node X as
evidence it's still alive, in lieu of a heartbeat from node X?

Matthew
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Can RabbitMQ handle big messages?

Zabrane Mickael

On Mar 12, 2012, at 5:02 PM, Matthew Sackman wrote:

On Mon, Mar 12, 2012 at 04:42:52PM +0100, Zabrane Mickael wrote:
It's not a bug. You have to be aware of it to build reliable systems in case of multiple connected Erlang nodes.

For those of us struggling to follow this, if you're currently in the
act of receiving data from node X, why can't you assume node X is still
alive? I.e. what is wrong with treating arbitrary data from node X as
evidence it's still alive, in lieu of a heartbeat from node X?


Regards,
Zabrane


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Can RabbitMQ handle big messages?

Matthew Sackman-3
On Mon, Mar 12, 2012 at 05:23:59PM +0100, Zabrane Mickael wrote:
> http://learnyousomeerlang.com/distribunomicon

Mmm, the "Bandwidth is infinite" section mentions this bug in Erlang,
but says nothing about the justification for it:

"Worse than that, Erlang knows whether nodes are alive or not by sending
a thing called heartbeats. Heartbeats are small messages sent at a
regular interval between two nodes basically saying "I'm still alive,
keep on keepin' on!". They're like our Zombie survivors routinely
pinging each other with messages; "Bill, are you there?" And if Bill
never replies, then you might assume he's dead (our out of batteries)
and he won't get your future communications. Anyway, heartbeats are sent
over the same channel as regular messages.

"The problem is that a large message can thus hold heartbeats back. Too
many large messages keeping heartbeats at bay for too long and either of
the nodes will eventually assume the other is unresponsive and
disconnect from each other. That's bad. In any case, the good Erlang
design lesson to keep this from happenning is to keep your messages
small. Everything will be better that way."

Somewhat ironic that AMQP itself does understand that receiving any data
from a peer indicates the peer is alive, and furthermore has the
ability to multiplex messages so that a single large message doesn't
block other messages.

Matthew
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Can RabbitMQ handle big messages?

Tony Garnock-Jones-6
In reply to this post by Zabrane Mickael
On 12 March 2012 12:23, Zabrane Mickael <[hidden email]> wrote:
On Mar 12, 2012, at 5:02 PM, Matthew Sackman wrote:
For those of us struggling to follow this, if you're currently in the
act of receiving data from node X, why can't you assume node X is still
alive? I.e. what is wrong with treating arbitrary data from node X as
evidence it's still alive, in lieu of a heartbeat from node X?


Yes, that repeats the information that Irmo started this subthread with. It doesn't address Matthew's question at all, though.

Perhaps the erlang list is a better place for us to be asking about this, Matthew, since it's not directly about Rabbit - are you on that list? I'm not currently subscribed.

Tony
--
Tony Garnock-Jones
[hidden email]
http://homepages.kcbbs.gen.nz/tonyg/

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
12
Loading...