Backing up persisted messages

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Backing up persisted messages

Beth Jones
Hello

What's the best way to backup persistent messages?  I have a queue that contains messages that aren't being consumed and I want to create a back up of them.

Thanks in advance.

Beth

--
 The Wellcome Trust Sanger Institute is operated by Genome Research
 Limited, a charity registered in England with number 1021457 and a
 company registered in England with number 2742969, whose registered
 office is 215 Euston Road, London, NW1 2BE.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Backing up persisted messages

Tim Watson-6
It really depends on what you mean by "backup". What are you trying to achieve? Do you need resiliency (i.e., avoiding data loss), are you trying to conserve disk space, or is there some other requirement at play?

For resiliency, clustering + HA offers a good set of guarantees if you're on a reliable network. You can also use the shovel and/or federation plugins to replicate messages between RabbitMQ nodes. The only other alternative would be backing up the mnesia directory, but that would copy all the data for all queues (and a bunch of other stuff too).

I'd suggest looking at the shovel or federation plugins. You can replicate data and handle it differently depending on your needs, e.g., to archive messages you want to deal with much later on, you might consume the replicated messages and archive them to a persistent storage medium of your choice, to be "replayed" later.

http://www.rabbitmq.com/distributed.html is probably worth a read.

Cheers,
Tim

On 30 Jan 2014, at 16:39, Beth Jones wrote:

> Hello
>
> What's the best way to backup persistent messages?  I have a queue that contains messages that aren't being consumed and I want to create a back up of them.
>
> Thanks in advance.
>
> Beth
>
> --
> The Wellcome Trust Sanger Institute is operated by Genome Research
> Limited, a charity registered in England with number 1021457 and a
> company registered in England with number 2742969, whose registered
> office is 215 Euston Road, London, NW1 2BE.
> _______________________________________________
> rabbitmq-discuss mailing list
> [hidden email]
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Backing up persisted messages

Beth Jones
Thanks Tim.

Specifically I have a queue that is accumulating messages and I'm worried about those messages being deleted accidentally.  I want to be able to back up that queue on a regular basis (say daily or hourly) to a location on disk.  

I read about copying the mnesia directory.  Does that copy all the data for all of the queue on that node?  Is there something similar that is queue specific?

Beth



On 30 Jan 2014, at 17:04, Tim Watson wrote:

> It really depends on what you mean by "backup". What are you trying to achieve? Do you need resiliency (i.e., avoiding data loss), are you trying to conserve disk space, or is there some other requirement at play?
>
> For resiliency, clustering + HA offers a good set of guarantees if you're on a reliable network. You can also use the shovel and/or federation plugins to replicate messages between RabbitMQ nodes. The only other alternative would be backing up the mnesia directory, but that would copy all the data for all queues (and a bunch of other stuff too).
>
> I'd suggest looking at the shovel or federation plugins. You can replicate data and handle it differently depending on your needs, e.g., to archive messages you want to deal with much later on, you might consume the replicated messages and archive them to a persistent storage medium of your choice, to be "replayed" later.
>
> http://www.rabbitmq.com/distributed.html is probably worth a read.
>
> Cheers,
> Tim
>
> On 30 Jan 2014, at 16:39, Beth Jones wrote:
>
>> Hello
>>
>> What's the best way to backup persistent messages?  I have a queue that contains messages that aren't being consumed and I want to create a back up of them.
>>
>> Thanks in advance.
>>
>> Beth
>>
>> --
>> The Wellcome Trust Sanger Institute is operated by Genome Research
>> Limited, a charity registered in England with number 1021457 and a
>> company registered in England with number 2742969, whose registered
>> office is 215 Euston Road, London, NW1 2BE.
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> [hidden email]
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
> _______________________________________________
> rabbitmq-discuss mailing list
> [hidden email]
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss



--
 The Wellcome Trust Sanger Institute is operated by Genome Research
 Limited, a charity registered in England with number 1021457 and a
 company registered in England with number 2742969, whose registered
 office is 215 Euston Road, London, NW1 2BE.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Backing up persisted messages

Tim Watson-6
On 30 Jan 2014, at 17:58, Beth Jones wrote:
> Specifically I have a queue that is accumulating messages and I'm worried about those messages being deleted accidentally.  I want to be able to back up that queue on a regular basis (say daily or hourly) to a location on disk.  
>
> I read about copying the mnesia directory.  Does that copy all the data for all of the queue on that node?  Is there something similar that is queue specific?
>

The queue specific thing would be to cluster two or more RabbitMQ brokers and set a High Availability policy on the queue so that all its messages are replicated to the other node(s). If the primary node crashes, one of the replicas will take over and you won't loose any messages (that have already been written to disk). Take a look at http://www.rabbitmq.com/ha.html for more details.

If your network is potentially unreliable (i.e., network partitions are quite likely) then you might want to consider using a federated exchange instead. This will perform the replication for you (i.e., copy messages from the exchange to another node(s) as per your configuration/policy) and the receiving node will effectively have a backup of all the data sent to the original exchange. I'd go for HA if you've got a reliable network though, since for your use-case it is ideal.

Cheers,
Tim
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Backing up persisted messages

Beth Jones
The scenario I'm looking at, is someone accidentally / inadvertently / deliberately deleting the messages from the queue.  I want to be able to take a snapshot of the messages that are in a particular queue and put them somewhere else on disk outside of RabbitMQ.  Then if someone/something does delete the messages, I want to be able to restore them.

Having the messages replicated on multiple nodes goes someway towards mitigating the risk, but it doesn't stop someone accidentally clearing out multiple queues.

Beth


On 30 Jan 2014, at 18:34, Tim Watson wrote:

> On 30 Jan 2014, at 17:58, Beth Jones wrote:
>> Specifically I have a queue that is accumulating messages and I'm worried about those messages being deleted accidentally.  I want to be able to back up that queue on a regular basis (say daily or hourly) to a location on disk.  
>>
>> I read about copying the mnesia directory.  Does that copy all the data for all of the queue on that node?  Is there something similar that is queue specific?
>>
>
> The queue specific thing would be to cluster two or more RabbitMQ brokers and set a High Availability policy on the queue so that all its messages are replicated to the other node(s). If the primary node crashes, one of the replicas will take over and you won't loose any messages (that have already been written to disk). Take a look at http://www.rabbitmq.com/ha.html for more details.
>
> If your network is potentially unreliable (i.e., network partitions are quite likely) then you might want to consider using a federated exchange instead. This will perform the replication for you (i.e., copy messages from the exchange to another node(s) as per your configuration/policy) and the receiving node will effectively have a backup of all the data sent to the original exchange. I'd go for HA if you've got a reliable network though, since for your use-case it is ideal.
>
> Cheers,
> Tim
> _______________________________________________
> rabbitmq-discuss mailing list
> [hidden email]
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss



--
 The Wellcome Trust Sanger Institute is operated by Genome Research
 Limited, a charity registered in England with number 1021457 and a
 company registered in England with number 2742969, whose registered
 office is 215 Euston Road, London, NW1 2BE.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Backing up persisted messages

Alvaro Videla-2
Hi,

What if you replicate the messages to a node on different
administrative domain, using exchange federation?
http://www.rabbitmq.com/federation.html

That way you can prevent people from connecting there and deleting messages.

Regards,

Alvaro

On Fri, Jan 31, 2014 at 10:24 AM, Beth Jones <[hidden email]> wrote:

> The scenario I'm looking at, is someone accidentally / inadvertently / deliberately deleting the messages from the queue.  I want to be able to take a snapshot of the messages that are in a particular queue and put them somewhere else on disk outside of RabbitMQ.  Then if someone/something does delete the messages, I want to be able to restore them.
>
> Having the messages replicated on multiple nodes goes someway towards mitigating the risk, but it doesn't stop someone accidentally clearing out multiple queues.
>
> Beth
>
>
> On 30 Jan 2014, at 18:34, Tim Watson wrote:
>
>> On 30 Jan 2014, at 17:58, Beth Jones wrote:
>>> Specifically I have a queue that is accumulating messages and I'm worried about those messages being deleted accidentally.  I want to be able to back up that queue on a regular basis (say daily or hourly) to a location on disk.
>>>
>>> I read about copying the mnesia directory.  Does that copy all the data for all of the queue on that node?  Is there something similar that is queue specific?
>>>
>>
>> The queue specific thing would be to cluster two or more RabbitMQ brokers and set a High Availability policy on the queue so that all its messages are replicated to the other node(s). If the primary node crashes, one of the replicas will take over and you won't loose any messages (that have already been written to disk). Take a look at http://www.rabbitmq.com/ha.html for more details.
>>
>> If your network is potentially unreliable (i.e., network partitions are quite likely) then you might want to consider using a federated exchange instead. This will perform the replication for you (i.e., copy messages from the exchange to another node(s) as per your configuration/policy) and the receiving node will effectively have a backup of all the data sent to the original exchange. I'd go for HA if you've got a reliable network though, since for your use-case it is ideal.
>>
>> Cheers,
>> Tim
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> [hidden email]
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
>
> --
>  The Wellcome Trust Sanger Institute is operated by Genome Research
>  Limited, a charity registered in England with number 1021457 and a
>  company registered in England with number 2742969, whose registered
>  office is 215 Euston Road, London, NW1 2BE.
> _______________________________________________
> rabbitmq-discuss mailing list
> [hidden email]
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Backing up persisted messages

Beth Jones
I'll have a look at exchange federation - thanks for the suggestion.

I'm surprised that this isn't something that's supported out of the box - is it on the TODO list?

Beth


On 31 Jan 2014, at 09:32, Alvaro Videla wrote:

> Hi,
>
> What if you replicate the messages to a node on different
> administrative domain, using exchange federation?
> http://www.rabbitmq.com/federation.html
>
> That way you can prevent people from connecting there and deleting messages.
>
> Regards,
>
> Alvaro
>
> On Fri, Jan 31, 2014 at 10:24 AM, Beth Jones <[hidden email]> wrote:
>> The scenario I'm looking at, is someone accidentally / inadvertently / deliberately deleting the messages from the queue.  I want to be able to take a snapshot of the messages that are in a particular queue and put them somewhere else on disk outside of RabbitMQ.  Then if someone/something does delete the messages, I want to be able to restore them.
>>
>> Having the messages replicated on multiple nodes goes someway towards mitigating the risk, but it doesn't stop someone accidentally clearing out multiple queues.
>>
>> Beth
>>
>>
>> On 30 Jan 2014, at 18:34, Tim Watson wrote:
>>
>>> On 30 Jan 2014, at 17:58, Beth Jones wrote:
>>>> Specifically I have a queue that is accumulating messages and I'm worried about those messages being deleted accidentally.  I want to be able to back up that queue on a regular basis (say daily or hourly) to a location on disk.
>>>>
>>>> I read about copying the mnesia directory.  Does that copy all the data for all of the queue on that node?  Is there something similar that is queue specific?
>>>>
>>>
>>> The queue specific thing would be to cluster two or more RabbitMQ brokers and set a High Availability policy on the queue so that all its messages are replicated to the other node(s). If the primary node crashes, one of the replicas will take over and you won't loose any messages (that have already been written to disk). Take a look at http://www.rabbitmq.com/ha.html for more details.
>>>
>>> If your network is potentially unreliable (i.e., network partitions are quite likely) then you might want to consider using a federated exchange instead. This will perform the replication for you (i.e., copy messages from the exchange to another node(s) as per your configuration/policy) and the receiving node will effectively have a backup of all the data sent to the original exchange. I'd go for HA if you've got a reliable network though, since for your use-case it is ideal.
>>>
>>> Cheers,
>>> Tim
>>> _______________________________________________
>>> rabbitmq-discuss mailing list
>>> [hidden email]
>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>>
>>
>> --
>> The Wellcome Trust Sanger Institute is operated by Genome Research
>> Limited, a charity registered in England with number 1021457 and a
>> company registered in England with number 2742969, whose registered
>> office is 215 Euston Road, London, NW1 2BE.
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> [hidden email]
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> _______________________________________________
> rabbitmq-discuss mailing list
> [hidden email]
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss



--
 The Wellcome Trust Sanger Institute is operated by Genome Research
 Limited, a charity registered in England with number 1021457 and a
 company registered in England with number 2742969, whose registered
 office is 215 Euston Road, London, NW1 2BE.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Backing up persisted messages

Alvaro Videla-2
If what you want is to backup the Mnesia folder, then yes, that would
backup queue contents, but it won't be specific to one queue.

What should be on the TODO list?

On Fri, Jan 31, 2014 at 11:01 AM, Beth Jones <[hidden email]> wrote:

> I'll have a look at exchange federation - thanks for the suggestion.
>
> I'm surprised that this isn't something that's supported out of the box - is it on the TODO list?
>
> Beth
>
>
> On 31 Jan 2014, at 09:32, Alvaro Videla wrote:
>
>> Hi,
>>
>> What if you replicate the messages to a node on different
>> administrative domain, using exchange federation?
>> http://www.rabbitmq.com/federation.html
>>
>> That way you can prevent people from connecting there and deleting messages.
>>
>> Regards,
>>
>> Alvaro
>>
>> On Fri, Jan 31, 2014 at 10:24 AM, Beth Jones <[hidden email]> wrote:
>>> The scenario I'm looking at, is someone accidentally / inadvertently / deliberately deleting the messages from the queue.  I want to be able to take a snapshot of the messages that are in a particular queue and put them somewhere else on disk outside of RabbitMQ.  Then if someone/something does delete the messages, I want to be able to restore them.
>>>
>>> Having the messages replicated on multiple nodes goes someway towards mitigating the risk, but it doesn't stop someone accidentally clearing out multiple queues.
>>>
>>> Beth
>>>
>>>
>>> On 30 Jan 2014, at 18:34, Tim Watson wrote:
>>>
>>>> On 30 Jan 2014, at 17:58, Beth Jones wrote:
>>>>> Specifically I have a queue that is accumulating messages and I'm worried about those messages being deleted accidentally.  I want to be able to back up that queue on a regular basis (say daily or hourly) to a location on disk.
>>>>>
>>>>> I read about copying the mnesia directory.  Does that copy all the data for all of the queue on that node?  Is there something similar that is queue specific?
>>>>>
>>>>
>>>> The queue specific thing would be to cluster two or more RabbitMQ brokers and set a High Availability policy on the queue so that all its messages are replicated to the other node(s). If the primary node crashes, one of the replicas will take over and you won't loose any messages (that have already been written to disk). Take a look at http://www.rabbitmq.com/ha.html for more details.
>>>>
>>>> If your network is potentially unreliable (i.e., network partitions are quite likely) then you might want to consider using a federated exchange instead. This will perform the replication for you (i.e., copy messages from the exchange to another node(s) as per your configuration/policy) and the receiving node will effectively have a backup of all the data sent to the original exchange. I'd go for HA if you've got a reliable network though, since for your use-case it is ideal.
>>>>
>>>> Cheers,
>>>> Tim
>>>> _______________________________________________
>>>> rabbitmq-discuss mailing list
>>>> [hidden email]
>>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>>
>>>
>>>
>>> --
>>> The Wellcome Trust Sanger Institute is operated by Genome Research
>>> Limited, a charity registered in England with number 1021457 and a
>>> company registered in England with number 2742969, whose registered
>>> office is 215 Euston Road, London, NW1 2BE.
>>> _______________________________________________
>>> rabbitmq-discuss mailing list
>>> [hidden email]
>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> [hidden email]
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
>
> --
>  The Wellcome Trust Sanger Institute is operated by Genome Research
>  Limited, a charity registered in England with number 1021457 and a
>  company registered in England with number 2742969, whose registered
>  office is 215 Euston Road, London, NW1 2BE.
> _______________________________________________
> rabbitmq-discuss mailing list
> [hidden email]
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Backing up persisted messages

Beth Jones
I was wondering if the ability to back up a specific queue would ever make it onto the TODO list.

Beth

On 31 Jan 2014, at 10:06, Alvaro Videla wrote:

> If what you want is to backup the Mnesia folder, then yes, that would
> backup queue contents, but it won't be specific to one queue.
>
> What should be on the TODO list?
>
> On Fri, Jan 31, 2014 at 11:01 AM, Beth Jones <[hidden email]> wrote:
>> I'll have a look at exchange federation - thanks for the suggestion.
>>
>> I'm surprised that this isn't something that's supported out of the box - is it on the TODO list?
>>
>> Beth
>>
>>
>> On 31 Jan 2014, at 09:32, Alvaro Videla wrote:
>>
>>> Hi,
>>>
>>> What if you replicate the messages to a node on different
>>> administrative domain, using exchange federation?
>>> http://www.rabbitmq.com/federation.html
>>>
>>> That way you can prevent people from connecting there and deleting messages.
>>>
>>> Regards,
>>>
>>> Alvaro
>>>
>>> On Fri, Jan 31, 2014 at 10:24 AM, Beth Jones <[hidden email]> wrote:
>>>> The scenario I'm looking at, is someone accidentally / inadvertently / deliberately deleting the messages from the queue.  I want to be able to take a snapshot of the messages that are in a particular queue and put them somewhere else on disk outside of RabbitMQ.  Then if someone/something does delete the messages, I want to be able to restore them.
>>>>
>>>> Having the messages replicated on multiple nodes goes someway towards mitigating the risk, but it doesn't stop someone accidentally clearing out multiple queues.
>>>>
>>>> Beth
>>>>
>>>>
>>>> On 30 Jan 2014, at 18:34, Tim Watson wrote:
>>>>
>>>>> On 30 Jan 2014, at 17:58, Beth Jones wrote:
>>>>>> Specifically I have a queue that is accumulating messages and I'm worried about those messages being deleted accidentally.  I want to be able to back up that queue on a regular basis (say daily or hourly) to a location on disk.
>>>>>>
>>>>>> I read about copying the mnesia directory.  Does that copy all the data for all of the queue on that node?  Is there something similar that is queue specific?
>>>>>>
>>>>>
>>>>> The queue specific thing would be to cluster two or more RabbitMQ brokers and set a High Availability policy on the queue so that all its messages are replicated to the other node(s). If the primary node crashes, one of the replicas will take over and you won't loose any messages (that have already been written to disk). Take a look at http://www.rabbitmq.com/ha.html for more details.
>>>>>
>>>>> If your network is potentially unreliable (i.e., network partitions are quite likely) then you might want to consider using a federated exchange instead. This will perform the replication for you (i.e., copy messages from the exchange to another node(s) as per your configuration/policy) and the receiving node will effectively have a backup of all the data sent to the original exchange. I'd go for HA if you've got a reliable network though, since for your use-case it is ideal.
>>>>>
>>>>> Cheers,
>>>>> Tim
>>>>> _______________________________________________
>>>>> rabbitmq-discuss mailing list
>>>>> [hidden email]
>>>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>>>
>>>>
>>>>
>>>> --
>>>> The Wellcome Trust Sanger Institute is operated by Genome Research
>>>> Limited, a charity registered in England with number 1021457 and a
>>>> company registered in England with number 2742969, whose registered
>>>> office is 215 Euston Road, London, NW1 2BE.
>>>> _______________________________________________
>>>> rabbitmq-discuss mailing list
>>>> [hidden email]
>>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>> _______________________________________________
>>> rabbitmq-discuss mailing list
>>> [hidden email]
>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>>
>>
>> --
>> The Wellcome Trust Sanger Institute is operated by Genome Research
>> Limited, a charity registered in England with number 1021457 and a
>> company registered in England with number 2742969, whose registered
>> office is 215 Euston Road, London, NW1 2BE.
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> [hidden email]
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> _______________________________________________
> rabbitmq-discuss mailing list
> [hidden email]
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss



--
 The Wellcome Trust Sanger Institute is operated by Genome Research
 Limited, a charity registered in England with number 1021457 and a
 company registered in England with number 2742969, whose registered
 office is 215 Euston Road, London, NW1 2BE.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Backing up persisted messages

Matthias Radestock-3
In reply to this post by Alvaro Videla-2
On 31/01/14 09:32, Alvaro Videla wrote:
> What if you replicate the messages to a node on different
> administrative domain, using exchange federation?
> http://www.rabbitmq.com/federation.html
>
> That way you can prevent people from connecting there and deleting
> messages.

Furthermore, if you configure the queue(s) on the backup rabbit with
some per-queue message ttl, you effectively end up with a async rolling
backup.

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Backing up persisted messages

Duncan McIntyre
I can see that if a per-queue message ttl is set, messages in RAM get deleted after the timeout. 

But what happens if there are a lot of messages in the queue and they have been overflowed to disk? Is that garbage-collected, or does it just fill up?

How does this interact with message persistence?


On Fri, Jan 31, 2014 at 11:15 AM, Matthias Radestock <[hidden email]> wrote:
On 31/01/14 09:32, Alvaro Videla wrote:
What if you replicate the messages to a node on different
administrative domain, using exchange federation?
http://www.rabbitmq.com/federation.html

That way you can prevent people from connecting there and deleting
messages.

Furthermore, if you configure the queue(s) on the backup rabbit with some per-queue message ttl, you effectively end up with a async rolling backup.

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


Duncan

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Backing up persisted messages

Matthias Radestock-3
On 31/01/14 13:00, Duncan McIntyre wrote:
> I can see that if a per-queue message ttl is set, messages in RAM get
> deleted after the timeout.
>
> But what happens if there are a lot of messages in the queue and they
> have been overflowed to disk? Is that garbage-collected, or does it just
> fill up?
>
> How does this interact with message persistence?

It works as you'd expect; expired messages are (logically) removed from
the queue regardless of whether they have been paged out.

Disk space in the persistent message store is reclaimed by a garbage
collector, which has a guaranteed upper bound on the amount of garbage
left around (50% of the total).

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss