Disable writing to disk

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Disable writing to disk

Matthias Reik-2
Hi all,

we have been using RabbitMQ for quite a while and are happy users of it,
but now we got a new requirement to not write, the messages that are
coming through a specific queue, to disk.

Does RabbitMQ support such a use case? I know that with ZeroMQ you can
configure the per-queue "swap" and therefore set it to 0, but we would
prefer to not switch technology.

One alternative would of course be to create a RamDisk and write the
content to that one, but it was clear from this article
http://www.rabbitmq.com/blog/2011/01/20/rabbitmq-backing-stores-databases-and-disks/
that this is not a good solution (even though it might be a solution for
us).

If I understand it, RabbitMQ will only write things to disk if we start
to run low on memory (when we hit the high-watermark?). At the same time
we will also get into flow control, correct? Would be good to get some
confirmation on this.

Ideally we would like that we get into flow-control _before_ we run out
of memory (and therefore before we start writing things to disk). We
would even prefer that the server "crashes" instead of writing the
messages to disk (yeah, extreme approach). One "hacky" approach would of
course observe the memory consumption of the server and just kill it
once we get too close to our high watermark (or set the high watermark
to 1.0?).

So what are the possibilities to fulfill such a (surely unusual)
requirement? Any advice is very much appreciated.

Cheers
Maze

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Disable writing to disk

Emile Joubert

Hi Matthias,

On 14/05/13 16:23, Matthias Reik wrote:
> we have been using RabbitMQ for quite a while and are happy users of it,
> but now we got a new requirement to not write, the messages that are
> coming through a specific queue, to disk.
>
> Does RabbitMQ support such a use case?

Not at the moment.

> One alternative would of course be to create a RamDisk and write the
> content to that one, but it was clear from this article
> http://www.rabbitmq.com/blog/2011/01/20/rabbitmq-backing-stores-databases-and-disks/
> that this is not a good solution (even though it might be a solution for
> us).


A RAM-disk will guarantee that messages don't reach secondary storage,
but I agree this is not a satisfactory solution. It will also affect all
the queues, while you may need finer control.

An alternative option would be to set the vm_memory_high_watermark to a
very large factor (e.g. 100). The broker will then behave as if it had
alot more memory than it really has (and therefore it runs the risk of
crashing with an out-of-memory error). At the moment the broker may
start sending messages to disk when as little as 5% of RAM is in use.
Setting the vm_memory_high_watermark high enough that this threshold is
never reached should avoid disk writes.

Can you explain why you want to avoid messages in disk? We have been
asked about this feature before, but the request has never been
accompanied by a strong motivating argument.



-Emile





_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Disable writing to disk

Matthias Reik-2

On 2013-05-14 17:44 , Emile Joubert wrote:

> Hi Matthias,
>
> On 14/05/13 16:23, Matthias Reik wrote:
>> we have been using RabbitMQ for quite a while and are happy users of it,
>> but now we got a new requirement to not write, the messages that are
>> coming through a specific queue, to disk.
>>
>> Does RabbitMQ support such a use case?
> Not at the moment.
>
>> One alternative would of course be to create a RamDisk and write the
>> content to that one, but it was clear from this article
>> http://www.rabbitmq.com/blog/2011/01/20/rabbitmq-backing-stores-databases-and-disks/
>> that this is not a good solution (even though it might be a solution for
>> us).
>
> A RAM-disk will guarantee that messages don't reach secondary storage,
> but I agree this is not a satisfactory solution. It will also affect all
> the queues, while you may need finer control.
We have a RabbitMQ cluster, so we could make sure that the RamDisk would
be set up only on those servers that have those kind of queues.
>
> An alternative option would be to set the vm_memory_high_watermark to a
> very large factor (e.g. 100). The broker will then behave as if it had
> alot more memory than it really has (and therefore it runs the risk of
> crashing with an out-of-memory error). At the moment the broker may
> start sending messages to disk when as little as 5% of RAM is in use.
> Setting the vm_memory_high_watermark high enough that this threshold is
> never reached should avoid disk writes.
Cool. Thanks for the tips/info. For this cases the ZeroMQ approach sounds
quite attractive (at least on paper), that you can assign "swap" size on
a per
queue level, see http://www.zeromq.org/whitepapers:design-v06
>
> Can you explain why you want to avoid messages in disk? We have been
> asked about this feature before, but the request has never been
> accompanied by a strong motivating argument.
The requirement comes from legal to not infringe Copyright laws (in
different countries). Apparently (and I'm not a legal person, so don't
really ask me any details), it's ok to keep the content in memory, but
not to "store it to disk". It is a bit unknown what the "store to disk"
really means, and therefore I think it's preferred to be on the safe side
and make sure (with reasonable effort) that the message is never
written to disk.
>
>
>
> -Emile
>
>
>
>
>

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Disable writing to disk

michaelklishin

2013/5/14 Matthias Reik <[hidden email]>
We have a RabbitMQ cluster, so we could make sure that the RamDisk would
be set up only on those servers that have those kind of queues.

Matthias,

Using RAM nodes for this queue sounds like the least painful and crash prone
solution to me.
--
MK

http://github.com/michaelklishin
http://twitter.com/michaelklishin

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Disable writing to disk

Matthias Reik-2

On 2013-05-14 18:48 , Michael Klishin wrote:

2013/5/14 Matthias Reik <[hidden email]>
We have a RabbitMQ cluster, so we could make sure that the RamDisk would
be set up only on those servers that have those kind of queues.

Matthias,

Using RAM nodes for this queue sounds like the least painful and crash prone
solution to me.
But my understanding was that RAM node would not prevent the messages to touch
disk. But maybe my understanding of RAM nodes is wrong (would be happy to hear
that :-) )


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Disable writing to disk

Simon MacMullen-2
In reply to this post by michaelklishin
On 14/05/13 17:48, Michael Klishin wrote:
> Using RAM nodes for this queue sounds like the least painful and crash prone
> solution to me.

No no no.

Disc nodes vs RAM nodes *only* controls whether metadata (essentially
queue / exchange / binding declarations) goes to disc or RAM. A RAM node
can still store persistent messages and page messages out.

(Yeah, the names are confusing.Not sure what to do about that.)

Cheers, Simon

--
Simon MacMullen
RabbitMQ, Pivotal
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Disable writing to disk

Matthias Reik-2

On 2013-05-14 19:01 , Simon MacMullen wrote:

> On 14/05/13 17:48, Michael Klishin wrote:
>> Using RAM nodes for this queue sounds like the least painful and
>> crash prone
>> solution to me.
>
> No no no.
>
> Disc nodes vs RAM nodes *only* controls whether metadata (essentially
> queue / exchange / binding declarations) goes to disc or RAM. A RAM
> node can still store persistent messages and page messages out.
Puhhh, thanks for the confirmation... my view has been restored :-)
>
> (Yeah, the names are confusing.Not sure what to do about that.)
>
> Cheers, Simon
>

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Disable writing to disk

Emile Joubert
In reply to this post by Matthias Reik-2
On 14/05/13 17:33, Matthias Reik wrote:

> it's ok to keep the content in memory, but not to "store it to disk".

This presumably then precludes the use of swapfiles on your server as
well, because copyright material might end up on disk via this route.

I suggest you check whether any of the excepts made for transient copies
of copyright works already applies.




-Emile




_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Disable writing to disk

Matthias Reik-2
Thanks for your replies.


On 2013-05-15 9:54 , Emile Joubert wrote:
> On 14/05/13 17:33, Matthias Reik wrote:
>
>> it's ok to keep the content in memory, but not to "store it to disk".
> This presumably then precludes the use of swapfiles on your server as
> well, because copyright material might end up on disk via this route.
Correct
>
> I suggest you check whether any of the excepts made for transient copies
> of copyright works already applies.
In most countries this is not an issue, but for some it seems to be :-( But
as I said, I'm not a legal person :-(
>
>
>
>
> -Emile
>
>
>
>

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss