How does Messages will be persisted in RabbitMQ?

classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

How does Messages will be persisted in RabbitMQ?

sekhar434
HI,

I would like to know, how the messages are stored in RabbitMQ server. Whether it is file level or DB level persistence?

And please let clarify me what is DISC, RAM and MNESIA.

Thanks in advace.

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: How does Messages will be persisted in RabbitMQ?

Michael Klishin
Rajasekhar P:

> I would like to know, how the messages are stored in RabbitMQ server. Whether it is file level or DB level persistence?

RabbitMQ uses its own storage layer for messages, not an external database. Certain parts of it are pluggable
and can use K/V stores such as Tokyo Cabinet.

>
> And please let clarify me what is DISC, RAM and MNESIA

Mnesia is a distributed database that RabbitMQ uses to store information about users, exchanges, queues, bindings, etc. Messages or message positions in queues are not stored in Mnesia.

Mnesia can store data either in RAM or on disk, or both. You can learn about disk and RAM nodes
on [1]. If you want to learn more about Mnesia, see [2], but
you don't need to learn anything about it in order to use RabbitMQ.

1. http://www.rabbitmq.com/clustering.html
2. http://www.erlang.org/doc/apps/mnesia/Mnesia_chap1.html#id57488

MK




_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

signature.asc (506 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: How does Messages will be persisted in RabbitMQ?

sekhar434
Thanks MK.

As you have mentioned Messages are not stored in Mnesia.  In my local system I have configured RABBITMQ_BASE to \ect\RabbitMQBase\db\rabbit@node1-mnesia Mnesia Mnesia under this "db" folder created which is having "msg_store_persistent " and "msg_store_transient" sub folders. So, can you please let me know where can I check messages persistence.

Thanks in advace!!!


On Fri, Aug 30, 2013 at 3:58 PM, Michael Klishin <[hidden email]> wrote:
Rajasekhar P:

> I would like to know, how the messages are stored in RabbitMQ server. Whether it is file level or DB level persistence?

RabbitMQ uses its own storage layer for messages, not an external database. Certain parts of it are pluggable
and can use K/V stores such as Tokyo Cabinet.

>
> And please let clarify me what is DISC, RAM and MNESIA

Mnesia is a distributed database that RabbitMQ uses to store information about users, exchanges, queues, bindings, etc. Messages or message positions in queues are not stored in Mnesia.

Mnesia can store data either in RAM or on disk, or both. You can learn about disk and RAM nodes
on [1]. If you want to learn more about Mnesia, see [2], but
you don't need to learn anything about it in order to use RabbitMQ.

1. http://www.rabbitmq.com/clustering.html
2. http://www.erlang.org/doc/apps/mnesia/Mnesia_chap1.html#id57488

MK




_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss



_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: How does Messages will be persisted in RabbitMQ?

Michael Klishin
Rajasekhar P:

> In my local system I have configured RABBITMQ_BASE to \ect\RabbitMQBase\db\rabbit@node1-mnesia Mnesia Mnesia under this "db" folder created which is having "msg_store_persistent " and "msg_store_transient" sub folders. So, can you please let me know where can I check messages persistence.

Message stores keep files under the mnesia directory but Mnesia never touches those files.
What do you mean
by "check messages persistence"? Do you want to inspect messages on disk?

MK




_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

signature.asc (506 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: How does Messages will be persisted in RabbitMQ?

sekhar434
As you have mentioned messages are not stored on Mnesia. But, durable messages are recovered even the RabbitMQ restarted, so where exactly these messages are stored?


On Fri, Aug 30, 2013 at 4:22 PM, Michael Klishin <[hidden email]> wrote:
Rajasekhar P:

> In my local system I have configured RABBITMQ_BASE to \ect\RabbitMQBase\db\rabbit@node1-mnesia Mnesia Mnesia under this "db" folder created which is having "msg_store_persistent " and "msg_store_transient" sub folders. So, can you please let me know where can I check messages persistence.

Message stores keep files under the mnesia directory but Mnesia never touches those files.
What do you mean
by "check messages persistence"? Do you want to inspect messages on disk?

MK




_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss



_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: How does Messages will be persisted in RabbitMQ?

Michael Klishin
Rajasekhar P:

> As you have mentioned messages are not stored on Mnesia. But, durable messages are recovered even the RabbitMQ restarted, so where exactly these messages are stored?

They are stored in files in a custom format.

See also http://www.rabbitmq.com/blog/2011/01/20/rabbitmq-backing-stores-databases-and-disks/.

MK




_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

signature.asc (506 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: How does Messages will be persisted in RabbitMQ?

sekhar434
I have gone through the link, here backing stores are for writing the messages into disk. But, I couldn't understand the use of database which mentioned in the article.  Can you please explain me how can I use database here. Thank you!!!


On Fri, Aug 30, 2013 at 4:33 PM, Michael Klishin <[hidden email]> wrote:
Rajasekhar P:

> As you have mentioned messages are not stored on Mnesia. But, durable messages are recovered even the RabbitMQ restarted, so where exactly these messages are stored?

They are stored in files in a custom format.

See also http://www.rabbitmq.com/blog/2011/01/20/rabbitmq-backing-stores-databases-and-disks/.

MK




_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss



_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: How does Messages will be persisted in RabbitMQ?

Michael Klishin
Rajasekhar P:

> I have gone through the link, here backing stores are for writing the messages into disk. But, I couldn't understand the use of database which mentioned in the article.  Can you please explain me how can I use database here. Thank you!!!

You don't use it. RabbitMQ does. You simply publish messages as persistent
if you want them to be stored on disk. Backing stores is an implementation detail
(so is the use of Mnesia for storing user/vhost/queue/exchange/binding info.

MK




_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

signature.asc (506 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: How does Messages will be persisted in RabbitMQ?

sekhar434
Thanks MK and am new to this RabbitMQ, sorry if I am asking silly questions. 

So, you mean we shouldn't use any database where RabbitMQ does for us ,

    1> with internal database and 
    2> even it doesn't use even any file system. 

In this case how can we restore the database in case of server crash or some other issue.



On Fri, Aug 30, 2013 at 5:04 PM, Michael Klishin <[hidden email]> wrote:
Rajasekhar P:

> I have gone through the link, here backing stores are for writing the messages into disk. But, I couldn't understand the use of database which mentioned in the article.  Can you please explain me how can I use database here. Thank you!!!

You don't use it. RabbitMQ does. You simply publish messages as persistent
if you want them to be stored on disk. Backing stores is an implementation detail
(so is the use of Mnesia for storing user/vhost/queue/exchange/binding info.

MK




_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss



_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: How does Messages will be persisted in RabbitMQ?

Michael Klishin
Rajasekhar P:

> So, you mean we shouldn't use any database where RabbitMQ does for us ,
>
>     1> with internal database and
>     2> even it doesn't use even any file system.
>
> In this case how can we restore the database in case of server crash or some other issue.

RabbitMQ will recover durable queues/exchanges and persistent messages in those queues
on start. It will also perform database upgrades when you install a newer version.

To create a backup RabbitMQ database you need to stop the node and copy Mnesia directory,
the archive it how you see appropriate. To restore, replace Mnesia directory dir with a one
from backup.

You can try it locally by overriding RABBITMQ_MNESIA_DIR, see http://www.rabbitmq.com/relocate.html.

MK




_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

signature.asc (506 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: How does Messages will be persisted in RabbitMQ?

Tim Watson-6
What Michael said, plus just to clarify...

On 30 Aug 2013, at 12:49, Michael Klishin wrote:
>> So, you mean we shouldn't use any database where RabbitMQ does for us ,
>>

Not only shouldn't you, you *cannot* use an alternative database for the purpose of storing persistent messages.

A vanilla RabbitMQ install (with no plugins in use) uses three different kinds of databases, two of which are persisted to disk:

1. mnesia - used to store "schema" data, such as queue, exchange and binding definitions (as per Michael's earlier explanation), either to disk or in ram
2. the message store - used to store persistent messages to disk (and messages that require paging due to memory pressure)
3. the message store index - stores the position of messages in queues, in memory by default (but can use alternative mechanisms such as tokyo cabinet)

>>    2> even it doesn't use even any file system.
>>

It *does* use the file system. The internal database writes data to a number of files which are (incidentally) stored underneath the root mnesia directory, although they have nothing to do with mnesia (we just store them in the same directory). These files contain data encoded in an internal format which is not human readable and we do not provide any tools for inspecting the contents of these database files.

>> In this case how can we restore the database in case of server crash or some other issue.
>
> RabbitMQ will recover durable queues/exchanges and persistent messages in those queues
> on start. It will also perform database upgrades when you install a newer version.
>

Specifically, RabbitMQ will use the aforementioned files when recovering persistent messages stored in durable queues on startup.

Tim
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: How does Messages will be persisted in RabbitMQ?

David Tinker
If you need to be able to access Rabbit queues outside of RabbitMQ
(for looking at consumed messages or to replay messages for testing or
recovery) you could use QDB (http://qdb.io/). QDB is a standalone
message server that integrates with RabbitMQ. Its easy to configure it
to consume messages from a RabbitMQ queue and play them back etc.

On Fri, Aug 30, 2013 at 2:11 PM, Tim Watson <[hidden email]> wrote:

> What Michael said, plus just to clarify...
>
> On 30 Aug 2013, at 12:49, Michael Klishin wrote:
>>> So, you mean we shouldn't use any database where RabbitMQ does for us ,
>>>
>
> Not only shouldn't you, you *cannot* use an alternative database for the purpose of storing persistent messages.
>
> A vanilla RabbitMQ install (with no plugins in use) uses three different kinds of databases, two of which are persisted to disk:
>
> 1. mnesia - used to store "schema" data, such as queue, exchange and binding definitions (as per Michael's earlier explanation), either to disk or in ram
> 2. the message store - used to store persistent messages to disk (and messages that require paging due to memory pressure)
> 3. the message store index - stores the position of messages in queues, in memory by default (but can use alternative mechanisms such as tokyo cabinet)
>
>>>    2> even it doesn't use even any file system.
>>>
>
> It *does* use the file system. The internal database writes data to a number of files which are (incidentally) stored underneath the root mnesia directory, although they have nothing to do with mnesia (we just store them in the same directory). These files contain data encoded in an internal format which is not human readable and we do not provide any tools for inspecting the contents of these database files.
>
>>> In this case how can we restore the database in case of server crash or some other issue.
>>
>> RabbitMQ will recover durable queues/exchanges and persistent messages in those queues
>> on start. It will also perform database upgrades when you install a newer version.
>>
>
> Specifically, RabbitMQ will use the aforementioned files when recovering persistent messages stored in durable queues on startup.
>
> Tim
> _______________________________________________
> rabbitmq-discuss mailing list
> [hidden email]
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss



--
http://qdb.io/ Persistent Message Queues With Replay and #RabbitMQ Integration
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: How does Messages will be persisted in RabbitMQ?

sekhar434
In reply to this post by Michael Klishin
I have added RABBITMQ_BASE, RABBITMQ_MNESIA_DIR and RABBITMQ_MNESIA_BASE in envt var's. But, the folder RABBITMQ_MNESIA_DIR folder is not updated even after restarting the server. Do I need to setup any other things?


On Fri, Aug 30, 2013 at 5:19 PM, Michael Klishin <[hidden email]> wrote:
Rajasekhar P:

> So, you mean we shouldn't use any database where RabbitMQ does for us ,
>
>     1> with internal database and
>     2> even it doesn't use even any file system.
>
> In this case how can we restore the database in case of server crash or some other issue.

RabbitMQ will recover durable queues/exchanges and persistent messages in those queues
on start. It will also perform database upgrades when you install a newer version.

To create a backup RabbitMQ database you need to stop the node and copy Mnesia directory,
the archive it how you see appropriate. To restore, replace Mnesia directory dir with a one
from backup.

You can try it locally by overriding RABBITMQ_MNESIA_DIR, see http://www.rabbitmq.com/relocate.html.

MK




_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss



_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: How does Messages will be persisted in RabbitMQ?

Michael Klishin
Rajasekhar P:

> I have added RABBITMQ_BASE, RABBITMQ_MNESIA_DIR and RABBITMQ_MNESIA_BASE in envt var's. But, the folder RABBITMQ_MNESIA_DIR folder is not updated even after restarting the server. Do I need to setup any other things?

How did you do that? RabbitMQ process needs to have those env vars to be in set in
its environment. If you use RabbitMQ from packages [1] [2], it should be taken care of
by the package scripts.

1. http://www.rabbitmq.com/install-debian.html
2. http://www.rabbitmq.com/install-rpm.html

MK




_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

signature.asc (506 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: How does Messages will be persisted in RabbitMQ?

sekhar434
I am working on Windows, when I set RABBITMQ_BASE working to some directory (C:\RabbitMQBase). After creating the queue default folder like "db", "log", etc will created automatically. But, no file is created under RABBITMQ_MNESIA_DIR.


On Fri, Aug 30, 2013 at 6:21 PM, Michael Klishin <[hidden email]> wrote:
Rajasekhar P:

> I have added RABBITMQ_BASE, RABBITMQ_MNESIA_DIR and RABBITMQ_MNESIA_BASE in envt var's. But, the folder RABBITMQ_MNESIA_DIR folder is not updated even after restarting the server. Do I need to setup any other things?

How did you do that? RabbitMQ process needs to have those env vars to be in set in
its environment. If you use RabbitMQ from packages [1] [2], it should be taken care of
by the package scripts.

1. http://www.rabbitmq.com/install-debian.html
2. http://www.rabbitmq.com/install-rpm.html

MK




_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss



_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: How does Messages will be persisted in RabbitMQ?

sekhar434
generated folders are C:\RabbitMQBase\db\rabbit@PRAJASEKHARLT-mnesia\msg_store_persistent, C:\RabbitMQBase\db\rabbit@PRAJASEKHARLT-mnesia\msg_store_transient, C:\RabbitMQBase\log.. etc


On Fri, Aug 30, 2013 at 6:35 PM, Rajasekhar P <[hidden email]> wrote:
I am working on Windows, when I set RABBITMQ_BASE working to some directory (C:\RabbitMQBase). After creating the queue default folder like "db", "log", etc will created automatically. But, no file is created under RABBITMQ_MNESIA_DIR.


On Fri, Aug 30, 2013 at 6:21 PM, Michael Klishin <[hidden email]> wrote:
Rajasekhar P:

> I have added RABBITMQ_BASE, RABBITMQ_MNESIA_DIR and RABBITMQ_MNESIA_BASE in envt var's. But, the folder RABBITMQ_MNESIA_DIR folder is not updated even after restarting the server. Do I need to setup any other things?

How did you do that? RabbitMQ process needs to have those env vars to be in set in
its environment. If you use RabbitMQ from packages [1] [2], it should be taken care of
by the package scripts.

1. http://www.rabbitmq.com/install-debian.html
2. http://www.rabbitmq.com/install-rpm.html

MK




_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss




_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: How does Messages will be persisted in RabbitMQ?

Michael Klishin
Rajasekhar P:

> generated folders are C:\RabbitMQBase\db\rabbit@PRAJASEKHARLT-mnesia\msg_store_persistent, C:\RabbitMQBase\db\rabbit@PRAJASEKHARLT-mnesia\msg_store_transient, C:\RabbitMQBase\log.. etc

My guess is that if RABBITMQ_BASE is set, all other directory paths are calculated against it.

I was suggesting to set RABBITMQ_MNESIA_DIR only. In any case, your Mnesia directory
path is C:\RabbitMQBase\db\rabbit@PRAJASEKHARLT-mnesia\

MK




_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

signature.asc (506 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: How does Messages will be persisted in RabbitMQ?

Michael Klishin
Michael Klishin:

> My guess is that if RABBITMQ_BASE is set, all other directory paths are calculated against it.

That's not correct. If RABBITMQ_MNESIA_DIR is set, it will be used. If not, it will be calculated
against RABBITMQ_BASE path.

Using either method is fine for experimenting with backups.

MK




_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

signature.asc (506 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: How does Messages will be persisted in RabbitMQ?

sekhar434
Thank you. I have configured RABBITMQ_BASE only. It is created db and log folders like C:\RabbitMQBase\db, C:\RabbitMQBase\log and RabbitMQ.debug.1 files  are created. Could you please tell me which folder I have to take backup?




On Fri, Aug 30, 2013 at 7:12 PM, Michael Klishin <[hidden email]> wrote:
Michael Klishin:

> My guess is that if RABBITMQ_BASE is set, all other directory paths are calculated against it.

That's not correct. If RABBITMQ_MNESIA_DIR is set, it will be used. If not, it will be calculated
against RABBITMQ_BASE path.

Using either method is fine for experimenting with backups.

MK




_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss



_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: How does Messages will be persisted in RabbitMQ?

Michael Klishin
Rajasekhar P:

> Could you please tell me which folder I have to take backup?

C:\RabbitMQBase\db

MK




_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

signature.asc (506 bytes) Download Attachment
12