Quantcast

Use a mysql databse table as the provider for rabbitmq queue

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Use a mysql databse table as the provider for rabbitmq queue

Shadowalker
Hi,

I've just stumbled upon rabbitmq and I think it could very well help in a project that I'm currently working on. But there's one thing that I'd like to get some input about though :

My project consist of one, two or more applications that, though separated, need to keep some of their datas synchronized.

For example say I have two use cases that could happen at the same time :

First project has a app A and a app B : when app A insert some specific data in its db i need to get a meesage to appB to do the corresponding action on its own db
Second project has only app A and when I insert stuff in A's db it shouldn't send anything anywhere.

In both cases, I'll like to have app A and app B identical : I don't want to have to change app A's code jsut because app B is here.

So my question is : Is there anyway to plug the raabitmq consumer directly onto a specific table in app A's database so that every time  something happens appB can come and consume the data.

Cheers,
Shadowalker.

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Use a mysql databse table as the provider for rabbitmq queue

Emile Joubert
Hi,

On 17/10/12 09:58, Shadowalker wrote:
> plug the raabitmq consumer directly onto a specific table

I'm sure there are many different ways of doing this. A possible
solution is to create a database trigger for A's table that sends an
update message when a change occurs. Application B (or any other
interested party) can then consume these messages and know that they
might need to synchronise from the table.

-Emile
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Use a mysql databse table as the provider for rabbitmq queue

Tim Watson-6
In reply to this post by Shadowalker
Hi,

On 10/17/2012 09:58 AM, Shadowalker wrote:
Hi,

I've just stumbled upon rabbitmq and I think it could very well help in a project that I'm currently working on. But there's one thing that I'd like to get some input about though :

My project consist of one, two or more applications that, though separated, need to keep some of their datas synchronized.

For example say I have two use cases that could happen at the same time :

First project has a app A and a app B : when app A insert some specific data in its db i need to get a meesage to appB to do the corresponding action on its own db
Second project has only app A and when I insert stuff in A's db it shouldn't send anything anywhere.

In both cases, I'll like to have app A and app B identical : I don't want to have to change app A's code jsut because app B is here.

So my question is : Is there anyway to plug the raabitmq consumer directly onto a specific table in app A's database so that every time  something happens appB can come and consume the data.


Well, I'm not sure I would choose this approach personally, and I'll explain why. But as I'm rather attention deficit I'll cut to the chase first and say "yes - it is *possible* to do this - but you'll have to put in a lot of work to make it happen" and as I said, I certainly wouldn't recommend doing it. Now I'll explain why it's a bad idea IMHO and then I'll explain how you can do it if you insist on following this path.
 
First of all, you are creating a very tight coupling between the application and the database schema, which will be difficult to work with if the design changes (which most do) over time. Secondly, you are going to be running code *inside the database server* which is fraught with danger. Thirdly, it's going to be very difficult to test and/or debug when things go wrong. Our industry tends to favour integrating systems using messaging technology so much because doing so decouples applications from one another, at least to a certain extent.

The deceptively simple alternative to messaging, and it is often ineffective and costly to develop and maintain, is to share data using files and/or a shared database. Both applications share tables and/or schemas and either poll for changes using worker processes (or threads) on the one hand, or utilise notification features of the database system itself on the other (e,g., Microsoft SQL Server Notification Services). But this approach is full of problems, ranging from contention for reads/writes on shared tables, to timing issues where both applications make very subtle implicit assumptions about the order in which the various participants in the system are interacting with the database, leading to functional errors.

One reason that messaging based integration has been so successful is that it gets us *away* from these problems, and allows applications to simplify their model of interaction with the outside world. AMQP in particular allows an application developer to think in terms of asynchronously publishing events (for consumption elsewhere) or consuming data in a variety of ways, based purely on the usage pattern required.

Let's assume that you did implement this capability and whenever your publishing application writes to the database, a message gets sent to an exchange on a rabbit broker somewhere, which the other application is consuming via some queue. What happens if the machine or data centre the broker is running on crashes? What happens if the broker becomes overloaded and starts to assert tcp back-pressure on the publisher - which, remember, is running inside a database!? What happens if the database server has to be restarted? And each of these 'what happens' questions must be answered from the perspective of

1. the code running inside the database server
2. the application writing to the database
3. the application listening to the queue(s)
4. the RabbitMQ broker itself

That is an *awful* lot of moving parts, one of which is critical to your infrastructure (i.e., the database server) and largely not under you control (in so much as the semantics for running user defined code inside the database server and the error handling behaviour of such, are likely to be highly constrained). Even in an environment where you're running your user defined function in an external operating system process (e.g., Oracle UDFs written in java) the failure modes are terrifying. Even in an environment which gives you a very clear picture of the constrained runtime in which you're operating (e.g., custom .NET code running inside MS SQL Server as a UDF or stored procedure) there are questions about how stable and safe this approach is. And even in those environments, where stability and safety *can* be understood and controlled to some extent, there is still the question of semantics. If the publication fails, what does the application writing to the database see? Does the insert/update fail? Doing so would require making a synchronous call in the trigger/callback, which would block the database server and create contention for the table! Failing to make the call synchronous would lead to silent failures however! There are no happy endings here, I can assure you.

Now that I hope I've put you off this idea, here's how you can do it if you disagree. Write a user defined function in C (consulting http://dev.mysql.com/doc/refman/5.1/en/adding-udf.html or whatever resources you have available to assist in this) and in your user defined function, use the RabbitMQ native/C client (librabbitmq) to talk to rabbit. The details of doing all of this are left as an exercise to the daring. Do bare in mind that MySQL UDFs *must* be thread safe, that librabbitmq has no explicit support for threading whatsoever and that blocking in a udf is probably the first step on the road to bedlam.

Cheers,
Tim
Cheers,
Shadowalker.


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Use a mysql databse table as the provider for rabbitmq queue

Tim Watson-6
Hi

Please keep the list on CC - the info is useful for everyone. :)

On 10/17/2012 11:05 AM, Ryan R. wrote:
Hi,
Thanks for that long and quite detailed answer.

Based on what you're saying plugging the rabbitmq provider directly on the table would not be ideal and would most likely bring issues that I'd rather avoid.


That's certainly my view.

That said, could you provide some insight on a potentielly viable solution for my project.
Reminder :
I want to keep my applications separate in terms of DB and code (avoid triggers if possible)
The reason for this is that depending on the situations, I would deploy only one, two, or more applications out of the whole project ( which roughly includes about 5 separate applications, each of which needs to be fully usable withou the others as well as with, without changing code not db config)


From what I'm hearing, the primary concern here is that we want to be able to deploy any application in isolation and have it still work, regardless of whether or not the others are up an running. This is exactly the kind of scenario where messaging is supposed to help! If it was me, I would look at data first and foremost, and ask myself what information needs to be publicised. In each application I would publish that information, not via the database (which I would keep for internal use), but by publishing data to an exchange on the RabbitMQ broker. Publication is asynchronous and will not fail if there are no consumers - the data will simply get routed to a queue and sit there until someone eventually 'asks for it'. Then I would look at what external information my application(s) might want to consume. I would set up each application to consume only data it needs, again from the messaging broker not the database. Consuming can be asynchronous or synchronous - you can choose which - and worker threads or other similar tools can be used make this information available to the 'main application code' when required. A nice client library may do some or all of this 'worker/listener with callbacks' stuff for you.

Now lets consider that design for a moment. Each application publishes data (asynchronously) regardless of whether or not someone is around to consume it. If there are no consumers, the data sits around in the queue. If you don't want this, then you can have the data expire immediately or after some timeout - rabbit will allow you do configure all of these patterns quite easily. The consuming part introduces slightly more new paths on your code, as you've got to decide how to get received messages from the consuming callbacks, back into the application code, but this isn't too hard. You can configure rabbit so that each message is consumed by only one application, or a copy is sent to all consumers, or you can route messages to specific consumers based on routing keys. The possibilities are numerous, and many good examples are available in the tutorials on our website!!!

Now the only common thing that you've got to deal with in each application is plugging in the messaging sub system - i.e., each application needs to hook up to RabbitMQ using a client API and write a bit of glue code for publishing and consuming. Apart from that common factor, which if your applications are all written in the same language, can be delegated to a shared library, you've kept those applications completely decoupled.

I hope that helps you to get started thinking about ways in which rabbit could help with your project. I do recommend going through the tutorials to get an idea of some of what's possible, and do feel free to come back here and ask about anything that's unclear or just to discuss ideas with the rest of the community.

Cheers,
Tim

Cheers,
Ryan.

2012/10/17 Tim Watson <[hidden email]>
Hi,

On 10/17/2012 09:58 AM, Shadowalker wrote:
Hi,

I've just stumbled upon rabbitmq and I think it could very well help in a project that I'm currently working on. But there's one thing that I'd like to get some input about though :

My project consist of one, two or more applications that, though separated, need to keep some of their datas synchronized.

For example say I have two use cases that could happen at the same time :

First project has a app A and a app B : when app A insert some specific data in its db i need to get a meesage to appB to do the corresponding action on its own db
Second project has only app A and when I insert stuff in A's db it shouldn't send anything anywhere.

In both cases, I'll like to have app A and app B identical : I don't want to have to change app A's code jsut because app B is here.

So my question is : Is there anyway to plug the raabitmq consumer directly onto a specific table in app A's database so that every time  something happens appB can come and consume the data.


Well, I'm not sure I would choose this approach personally, and I'll explain why. But as I'm rather attention deficit I'll cut to the chase first and say "yes - it is *possible* to do this - but you'll have to put in a lot of work to make it happen" and as I said, I certainly wouldn't recommend doing it. Now I'll explain why it's a bad idea IMHO and then I'll explain how you can do it if you insist on following this path.
 
First of all, you are creating a very tight coupling between the application and the database schema, which will be difficult to work with if the design changes (which most do) over time. Secondly, you are going to be running code *inside the database server* which is fraught with danger. Thirdly, it's going to be very difficult to test and/or debug when things go wrong. Our industry tends to favour integrating systems using messaging technology so much because doing so decouples applications from one another, at least to a certain extent.

The deceptively simple alternative to messaging, and it is often ineffective and costly to develop and maintain, is to share data using files and/or a shared database. Both applications share tables and/or schemas and either poll for changes using worker processes (or threads) on the one hand, or utilise notification features of the database system itself on the other (e,g., Microsoft SQL Server Notification Services). But this approach is full of problems, ranging from contention for reads/writes on shared tables, to timing issues where both applications make very subtle implicit assumptions about the order in which the various participants in the system are interacting with the database, leading to functional errors.

One reason that messaging based integration has been so successful is that it gets us *away* from these problems, and allows applications to simplify their model of interaction with the outside world. AMQP in particular allows an application developer to think in terms of asynchronously publishing events (for consumption elsewhere) or consuming data in a variety of ways, based purely on the usage pattern required.

Let's assume that you did implement this capability and whenever your publishing application writes to the database, a message gets sent to an exchange on a rabbit broker somewhere, which the other application is consuming via some queue. What happens if the machine or data centre the broker is running on crashes? What happens if the broker becomes overloaded and starts to assert tcp back-pressure on the publisher - which, remember, is running inside a database!? What happens if the database server has to be restarted? And each of these 'what happens' questions must be answered from the perspective of

1. the code running inside the database server
2. the application writing to the database
3. the application listening to the queue(s)
4. the RabbitMQ broker itself

That is an *awful* lot of moving parts, one of which is critical to your infrastructure (i.e., the database server) and largely not under you control (in so much as the semantics for running user defined code inside the database server and the error handling behaviour of such, are likely to be highly constrained). Even in an environment where you're running your user defined function in an external operating system process (e.g., Oracle UDFs written in java) the failure modes are terrifying. Even in an environment which gives you a very clear picture of the constrained runtime in which you're operating (e.g., custom .NET code running inside MS SQL Server as a UDF or stored procedure) there are questions about how stable and safe this approach is. And even in those environments, where stability and safety *can* be understood and controlled to some extent, there is still the question of semantics. If the publication fails, what does the application writing to the database see? Does the insert/update fail? Doing so would require making a synchronous call in the trigger/callback, which would block the database server and create contention for the table! Failing to make the call synchronous would lead to silent failures however! There are no happy endings here, I can assure you.

Now that I hope I've put you off this idea, here's how you can do it if you disagree. Write a user defined function in C (consulting http://dev.mysql.com/doc/refman/5.1/en/adding-udf.html or whatever resources you have available to assist in this) and in your user defined function, use the RabbitMQ native/C client (librabbitmq) to talk to rabbit. The details of doing all of this are left as an exercise to the daring. Do bare in mind that MySQL UDFs *must* be thread safe, that librabbitmq has no explicit support for threading whatsoever and that blocking in a udf is probably the first step on the road to bedlam.

Cheers,
Tim
Cheers,
Shadowalker.


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss




_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Use a mysql databse table as the provider for rabbitmq queue

Shadowalker
I think I understand what you mean with the shared library.
However, in my case, RabbitMQ would only be installed if need be (meaning more than one of the apps are present, and two of those need to be synchronised for part of their data).

That said, using a shared library would require me to "include/import" said library when I need to, therefore making me change my app code depending of the situation I'm in.

And said library would only be required when there's a RabbitMQ available anyway.

Now a bit further in your message you talk about a listener library.
I'd like to know a bit more about this.
How would an external library be able to listen to anything happening within my app ?
Would it be listening on the DB queries ?

Cheers,
Ryan.

2012/10/17 Tim Watson <[hidden email]>
Hi

Please keep the list on CC - the info is useful for everyone. :)


On 10/17/2012 11:05 AM, Ryan R. wrote:
Hi,
Thanks for that long and quite detailed answer.

Based on what you're saying plugging the rabbitmq provider directly on the table would not be ideal and would most likely bring issues that I'd rather avoid.


That's certainly my view.


That said, could you provide some insight on a potentielly viable solution for my project.
Reminder :
I want to keep my applications separate in terms of DB and code (avoid triggers if possible)
The reason for this is that depending on the situations, I would deploy only one, two, or more applications out of the whole project ( which roughly includes about 5 separate applications, each of which needs to be fully usable withou the others as well as with, without changing code not db config)


From what I'm hearing, the primary concern here is that we want to be able to deploy any application in isolation and have it still work, regardless of whether or not the others are up an running. This is exactly the kind of scenario where messaging is supposed to help! If it was me, I would look at data first and foremost, and ask myself what information needs to be publicised. In each application I would publish that information, not via the database (which I would keep for internal use), but by publishing data to an exchange on the RabbitMQ broker. Publication is asynchronous and will not fail if there are no consumers - the data will simply get routed to a queue and sit there until someone eventually 'asks for it'. Then I would look at what external information my application(s) might want to consume. I would set up each application to consume only data it needs, again from the messaging broker not the database. Consuming can be asynchronous or synchronous - you can choose which - and worker threads or other similar tools can be used make this information available to the 'main application code' when required. A nice client library may do some or all of this 'worker/listener with callbacks' stuff for you.

Now lets consider that design for a moment. Each application publishes data (asynchronously) regardless of whether or not someone is around to consume it. If there are no consumers, the data sits around in the queue. If you don't want this, then you can have the data expire immediately or after some timeout - rabbit will allow you do configure all of these patterns quite easily. The consuming part introduces slightly more new paths on your code, as you've got to decide how to get received messages from the consuming callbacks, back into the application code, but this isn't too hard. You can configure rabbit so that each message is consumed by only one application, or a copy is sent to all consumers, or you can route messages to specific consumers based on routing keys. The possibilities are numerous, and many good examples are available in the tutorials on our website!!!

Now the only common thing that you've got to deal with in each application is plugging in the messaging sub system - i.e., each application needs to hook up to RabbitMQ using a client API and write a bit of glue code for publishing and consuming. Apart from that common factor, which if your applications are all written in the same language, can be delegated to a shared library, you've kept those applications completely decoupled.

I hope that helps you to get started thinking about ways in which rabbit could help with your project. I do recommend going through the tutorials to get an idea of some of what's possible, and do feel free to come back here and ask about anything that's unclear or just to discuss ideas with the rest of the community.

Cheers,
Tim


Cheers,
Ryan.

2012/10/17 Tim Watson <[hidden email]>
Hi,

On 10/17/2012 09:58 AM, Shadowalker wrote:
Hi,

I've just stumbled upon rabbitmq and I think it could very well help in a project that I'm currently working on. But there's one thing that I'd like to get some input about though :

My project consist of one, two or more applications that, though separated, need to keep some of their datas synchronized.

For example say I have two use cases that could happen at the same time :

First project has a app A and a app B : when app A insert some specific data in its db i need to get a meesage to appB to do the corresponding action on its own db
Second project has only app A and when I insert stuff in A's db it shouldn't send anything anywhere.

In both cases, I'll like to have app A and app B identical : I don't want to have to change app A's code jsut because app B is here.

So my question is : Is there anyway to plug the raabitmq consumer directly onto a specific table in app A's database so that every time  something happens appB can come and consume the data.


Well, I'm not sure I would choose this approach personally, and I'll explain why. But as I'm rather attention deficit I'll cut to the chase first and say "yes - it is *possible* to do this - but you'll have to put in a lot of work to make it happen" and as I said, I certainly wouldn't recommend doing it. Now I'll explain why it's a bad idea IMHO and then I'll explain how you can do it if you insist on following this path.
 
First of all, you are creating a very tight coupling between the application and the database schema, which will be difficult to work with if the design changes (which most do) over time. Secondly, you are going to be running code *inside the database server* which is fraught with danger. Thirdly, it's going to be very difficult to test and/or debug when things go wrong. Our industry tends to favour integrating systems using messaging technology so much because doing so decouples applications from one another, at least to a certain extent.

The deceptively simple alternative to messaging, and it is often ineffective and costly to develop and maintain, is to share data using files and/or a shared database. Both applications share tables and/or schemas and either poll for changes using worker processes (or threads) on the one hand, or utilise notification features of the database system itself on the other (e,g., Microsoft SQL Server Notification Services). But this approach is full of problems, ranging from contention for reads/writes on shared tables, to timing issues where both applications make very subtle implicit assumptions about the order in which the various participants in the system are interacting with the database, leading to functional errors.

One reason that messaging based integration has been so successful is that it gets us *away* from these problems, and allows applications to simplify their model of interaction with the outside world. AMQP in particular allows an application developer to think in terms of asynchronously publishing events (for consumption elsewhere) or consuming data in a variety of ways, based purely on the usage pattern required.

Let's assume that you did implement this capability and whenever your publishing application writes to the database, a message gets sent to an exchange on a rabbit broker somewhere, which the other application is consuming via some queue. What happens if the machine or data centre the broker is running on crashes? What happens if the broker becomes overloaded and starts to assert tcp back-pressure on the publisher - which, remember, is running inside a database!? What happens if the database server has to be restarted? And each of these 'what happens' questions must be answered from the perspective of

1. the code running inside the database server
2. the application writing to the database
3. the application listening to the queue(s)
4. the RabbitMQ broker itself

That is an *awful* lot of moving parts, one of which is critical to your infrastructure (i.e., the database server) and largely not under you control (in so much as the semantics for running user defined code inside the database server and the error handling behaviour of such, are likely to be highly constrained). Even in an environment where you're running your user defined function in an external operating system process (e.g., Oracle UDFs written in java) the failure modes are terrifying. Even in an environment which gives you a very clear picture of the constrained runtime in which you're operating (e.g., custom .NET code running inside MS SQL Server as a UDF or stored procedure) there are questions about how stable and safe this approach is. And even in those environments, where stability and safety *can* be understood and controlled to some extent, there is still the question of semantics. If the publication fails, what does the application writing to the database see? Does the insert/update fail? Doing so would require making a synchronous call in the trigger/callback, which would block the database server and create contention for the table! Failing to make the call synchronous would lead to silent failures however! There are no happy endings here, I can assure you.

Now that I hope I've put you off this idea, here's how you can do it if you disagree. Write a user defined function in C (consulting http://dev.mysql.com/doc/refman/5.1/en/adding-udf.html or whatever resources you have available to assist in this) and in your user defined function, use the RabbitMQ native/C client (librabbitmq) to talk to rabbit. The details of doing all of this are left as an exercise to the daring. Do bare in mind that MySQL UDFs *must* be thread safe, that librabbitmq has no explicit support for threading whatsoever and that blocking in a udf is probably the first step on the road to bedlam.

Cheers,
Tim
Cheers,
Shadowalker.


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss





_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Use a mysql databse table as the provider for rabbitmq queue

Tim Watson-6
On 10/17/2012 11:40 AM, Ryan R. wrote:
> I think I understand what you mean with the shared library.
> However, in my case, RabbitMQ would only be installed if need be
> (meaning more than one of the apps are present, and two of those need
> to be synchronised for part of their data).
>

That actually complicates the picture somewhat - is there a reason why
this is the case? In a typical integration architecture, the messaging
broker is deployed centrally (on the LAN somewhere) and clients choose
whether or not they want to connect to it from whatever machine they're
running on. To me, it is sounding like you're describing an architecture
where both applications reside on the same machine and assuming that the
broker will also need to be co-resident with them, which is not really
the case, though there's nothing to prohibit that either.

> That said, using a shared library would require me to "include/import"
> said library when I need to, therefore making me change my app code
> depending of the situation I'm in.
>

Well yes, if you're going to add messaging capabilities to your
applications that don't currently support it, then you are going to have
to write *some* code and integrate it into them! :)

> And said library would only be required when there's a RabbitMQ
> available anyway.
>

I think you're making your life more complicated than it needs to be by
thinking about whether the messaging broker is available vs. not. The
broker should *always* be available when applications residing on
different machines need to communicate with one another, regardless of
whether those applications are running or not. Again, it feels like
you're trying to deal with applications running on the same machine -
have I picked that up correctly? It might help if you explained your
architecture in a bit more detail, so I can understand exactly what
you're trying to achieve.

> Now a bit further in your message you talk about a listener library.
> I'd like to know a bit more about this.
> How would an external library be able to listen to anything happening
> within my app ?
> Would it be listening on the DB queries ?
>

No, not at all. Let's say you've got two applications, App1 and App2.
You'll write some library code that both applications share, that
probably looks something like this (with *wide* variations depending on
language/platform - I've just written pseudo code to keep things simple):

-------------------------------------

function init = do
     read_config_file_for_this_app
     open_connection_to_broker
     store_connection_somewhere_in_memory
end

function listen = do
     get_connection_from_memory
     read_message_from_broker
     pass_message_to_application_thread_somehow
     listen
end

function publish = do
     get_connection_from_memory
     send_message_to_broker
end

-------------------------------------

Now in your applications, you'll call the shared 'init' library function
when you're starting up to bootstrap the connection to the broker. When
your application is publishing data, it calls publish and if/when you
need to subscribe to data then you'll call 'listen'. The fact is that
'how to listen' for incoming messages really depends on how you're going
to use them. But the point is that the applications read from and write
to the messaging broker, and do so independently of database tables. You
*may* decide to do something like write a middle-man application that
periodically reads a database table and publishes each row to the
messaging broker so it can be read from a queue, or do that with a
worker thread instead of a separate application. I would *not* do
anything here with the database though. If applications need to share
data, then **they should send it to one another via message queues.** If
they need to persist data, they should persist their own data in their
own tables in the database, but they should **not use the database to
communicate with one another.** That is the key thing with using
messaging instead shared data(bases).

There is an overhead in sending (and in some cases, duplicating) data
between applications of course. This is *more* than compensated for by
the reduced coupling that comes from integrating using messaging
technology. This approach may not be suited to integrating applications
that are running on the same physical machine and are tightly and
deliberately coupled however. I can't really elaborate on the
suitability of messaging for your project without understanding a good
deal more about it I'm afraid.

I hope that clears a few things up at least! :)

Cheers,
Tim

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Use a mysql databse table as the provider for rabbitmq queue

Shadowalker
Ok I've definitely got what you're trying to say concerning the use of DBs and message broker.

To explain a bit more, I'll answer the few questiosn you've scattered across your reply :

Is it a distributed architecture ?
    
Definitely yes, each app will run on a different server

Why do I need to keep apps separate ?

Simply because my project can be viewed as a application suite composed of about 5 different apps. The end-user will be able to choose which of the apps he wants and I want to only provide what is absolutely necessary for the end-user based on what he asked for, nothing more nothing less. Therefore if the end user only wants one app out five I'm not going to provide him with a RabbitMQ, he will have no use for it. However should he asked for a set of apps that need to communicate, then I'll install a RabbitMQ to let them communicate.
Now like I said, I'm looking for a way to do this without changing anything in the apps' code.
What I'm looking for is a solution where I would have something like a "watcher" external app that would keep an eye on what's going on in each app and manage the content of each RabbitMQ queues accordingly.

That peculiar aspect of the apps is for my longterm evolution of my app : for my personal use, the simpler approach of the code for pushing a mesage in the queues embedded in the app would be fine. But since I'd like to make it evolve to something else, I'd rather think ahead and try to do so right now.

Hope these explanations help a little more.

Cheers,
Ryan.

2012/10/17 Tim Watson <[hidden email]>
On 10/17/2012 11:40 AM, Ryan R. wrote:
I think I understand what you mean with the shared library.
However, in my case, RabbitMQ would only be installed if need be (meaning more than one of the apps are present, and two of those need to be synchronised for part of their data).


That actually complicates the picture somewhat - is there a reason why this is the case? In a typical integration architecture, the messaging broker is deployed centrally (on the LAN somewhere) and clients choose whether or not they want to connect to it from whatever machine they're running on. To me, it is sounding like you're describing an architecture where both applications reside on the same machine and assuming that the broker will also need to be co-resident with them, which is not really the case, though there's nothing to prohibit that either.


That said, using a shared library would require me to "include/import" said library when I need to, therefore making me change my app code depending of the situation I'm in.


Well yes, if you're going to add messaging capabilities to your applications that don't currently support it, then you are going to have to write *some* code and integrate it into them! :)


And said library would only be required when there's a RabbitMQ available anyway.


I think you're making your life more complicated than it needs to be by thinking about whether the messaging broker is available vs. not. The broker should *always* be available when applications residing on different machines need to communicate with one another, regardless of whether those applications are running or not. Again, it feels like you're trying to deal with applications running on the same machine - have I picked that up correctly? It might help if you explained your architecture in a bit more detail, so I can understand exactly what you're trying to achieve.


Now a bit further in your message you talk about a listener library.
I'd like to know a bit more about this.
How would an external library be able to listen to anything happening within my app ?
Would it be listening on the DB queries ?


No, not at all. Let's say you've got two applications, App1 and App2. You'll write some library code that both applications share, that probably looks something like this (with *wide* variations depending on language/platform - I've just written pseudo code to keep things simple):

-------------------------------------

function init = do
    read_config_file_for_this_app
    open_connection_to_broker
    store_connection_somewhere_in_memory
end

function listen = do
    get_connection_from_memory
    read_message_from_broker
    pass_message_to_application_thread_somehow
    listen
end

function publish = do
    get_connection_from_memory
    send_message_to_broker
end

-------------------------------------

Now in your applications, you'll call the shared 'init' library function when you're starting up to bootstrap the connection to the broker. When your application is publishing data, it calls publish and if/when you need to subscribe to data then you'll call 'listen'. The fact is that 'how to listen' for incoming messages really depends on how you're going to use them. But the point is that the applications read from and write to the messaging broker, and do so independently of database tables. You *may* decide to do something like write a middle-man application that periodically reads a database table and publishes each row to the messaging broker so it can be read from a queue, or do that with a worker thread instead of a separate application. I would *not* do anything here with the database though. If applications need to share data, then **they should send it to one another via message queues.** If they need to persist data, they should persist their own data in their own tables in the database, but they should **not use the database to communicate with one another.** That is the key thing with using messaging instead shared data(bases).

There is an overhead in sending (and in some cases, duplicating) data between applications of course. This is *more* than compensated for by the reduced coupling that comes from integrating using messaging technology. This approach may not be suited to integrating applications that are running on the same physical machine and are tightly and deliberately coupled however. I can't really elaborate on the suitability of messaging for your project without understanding a good deal more about it I'm afraid.

I hope that clears a few things up at least! :)

Cheers,
Tim



_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Use a mysql databse table as the provider for rabbitmq queue

Tim Watson-6
Hi Ryan,

On 10/17/2012 02:03 PM, Ryan R. wrote:
Ok I've definitely got what you're trying to say concerning the use of DBs and message broker.


Good, and your post has helped me to understand what you're doing more clearly, so this should get easier now.

To explain a bit more, I'll answer the few questiosn you've scattered across your reply :

Is it a distributed architecture ?
    
Definitely yes, each app will run on a different server


Ok, so that's more the classic integration architecture for sure.

Why do I need to keep apps separate ?

Simply because my project can be viewed as a application suite composed of about 5 different apps. The end-user will be able to choose which of the apps he wants and I want to only provide what is absolutely necessary for the end-user based on what he asked for, nothing more nothing less. Therefore if the end user only wants one app out five I'm not going to provide him with a RabbitMQ, he will have no use for it. However should he asked for a set of apps that need to communicate, then I'll install a RabbitMQ to let them communicate.

Ok, well that is certainly do-able and I can understand the reasons why a particular customer site that doesn't need the middle-ware might as well skip it.

Now like I said, I'm looking for a way to do this without changing anything in the apps' code.
What I'm looking for is a solution where I would have something like a "watcher" external app that would keep an eye on what's going on in each app and manage the content of each RabbitMQ queues accordingly.


Right, and in fact that's probably quite a common pattern in integration architectures. What I would ask is this: how are these applications supposed to know about the incoming data? And how, for that matter, is the data supposed to get into the queues in the first place?

Now, one solution to this is to do what you say and write a new application, a copy of which resides alongside each main app in the system and basically does two things:

1. periodically read data from the database for application 1 and write it to a queue
2. constantly (in a worker thread, or periodically in the main one) read data from a queue and write it to the application database

This is fine, in theory, as long as you bare in mind that what you're basically doing is making the overall system into a sort of 'shared database where tables are replicated/synchronised via queues'. Think about it a while and it's clear that, even if the tables being read from and/or written to are different, that basically you end up with the same thing. Then ask yourself if you'd design a distributed database that uses a messaging system as its replication back-bone.

Now I don't want to completely put you off that approach. It certainly has some things going for it: you do not need to change the application code anywhere, and if the application doesn't need the messaging system then you just avoid installing the 'watcher' process (and broker) in the first place. Job done!

There are things to consider with this approach though, which will re-enforce my point about it being like a distributed database that isn't. Consider some rhetorical questions about the systems that may or may not be communicating:

- do they rely on incoming information in order to do their job?
- does incoming information lead (causally) to more outgoing information?
- does outgoing information rely on future incoming information?
- and for any of the above, does the order in which information is handled ever matter?

If the answer to any of those questions is 'yes', then it might be worth reconsidering the use of a separate 'watcher'. All of those scenarios can lead to races, deadlocks and all manner of other problems usually associated with concurrent application development, because a distributed system that uses asynchronous message passing is *inherently* concurrent in nature and races/deadlocks/etc are all *much* harder to spot (and fix) when they're distributed!

That peculiar aspect of the apps is for my longterm evolution of my app : for my personal use, the simpler approach of the code for pushing a mesage in the queues embedded in the app would be fine. But since I'd like to make it evolve to something else, I'd rather think ahead and try to do so right now.


Well, I'm unable to comment on the design choices for a technology I know nothing about, but I will offer this. If messaging is made into a specific (service) layer for an application, that application should usually be able to evolve without ever changing the messaging code. The event publication subsystem is, after all, a fairly static design point - you choose what to publish and when (which can, of course, be made configurable in any language/platform) and then just leave the publication code alone. The event listening subsystem is likewise fairly static. You start a worker thread (or external process) and register callbacks that get run when a message arrives. Just as with the publication subsystem, you can pass whatever callbacks you like and these can be chosen at build time *or* at runtime, possibly driven by configuration settings if that is required. Finally, the remaining problem is what to do when there *is no broker* because the application is standalone. I think this is fairly simple - just set a flag (or configuration value) when starting you application and let the event handling subsystems (both the listener and publisher) treat all function calls as a no-op when there is no broker to communicate with.

The great advantage of this approach, IMO, is precisely the one you seem to want - that the applications can be evolved over time to publish new data and/or respond to new (or existing) messages in varying ways. Providing you have the infrastructure to make decisions about what to do in callbacks based on configuration settings, you *might* even be able to add new behaviours to your applications without writing any code! ;)

But..... There is the up-front cost of embedding the messaging technology into your applications in the first place. Personally, I would not make that cost the primary driver behind the design decision though. In your shoes, I would consider carefully how the interactions between these applications work and understand the distribution and information sharing model they need to conform to first. Once that is clear, then you can decide whether or not the timing, ordering and/or existential questions about the system architecture as a whole really matter or not. If they don't, then you might as well choose the external 'watcher' as this minimises the impact on your code base and simplifies your deployments. If, on the other hand, they do matter, then you should *very* carefully consider whether the decoupling that the 'watcher' offers is likely to be a hindrance or a help. If enough of those causal relationships in the information sharing model require coupling of some kind, then interacting directly with a messaging system (where you can choose whether or not you care about acknowledgements/receipts, batching, transactions/confirms, and other such features) might well prove to be architecturally important.

HTH and makes some degree of sense!

Cheers,
Tim

Hope these explanations help a little more.

Cheers,
Ryan.

2012/10/17 Tim Watson <[hidden email]>
On 10/17/2012 11:40 AM, Ryan R. wrote:
I think I understand what you mean with the shared library.
However, in my case, RabbitMQ would only be installed if need be (meaning more than one of the apps are present, and two of those need to be synchronised for part of their data).


That actually complicates the picture somewhat - is there a reason why this is the case? In a typical integration architecture, the messaging broker is deployed centrally (on the LAN somewhere) and clients choose whether or not they want to connect to it from whatever machine they're running on. To me, it is sounding like you're describing an architecture where both applications reside on the same machine and assuming that the broker will also need to be co-resident with them, which is not really the case, though there's nothing to prohibit that either.


That said, using a shared library would require me to "include/import" said library when I need to, therefore making me change my app code depending of the situation I'm in.


Well yes, if you're going to add messaging capabilities to your applications that don't currently support it, then you are going to have to write *some* code and integrate it into them! :)


And said library would only be required when there's a RabbitMQ available anyway.


I think you're making your life more complicated than it needs to be by thinking about whether the messaging broker is available vs. not. The broker should *always* be available when applications residing on different machines need to communicate with one another, regardless of whether those applications are running or not. Again, it feels like you're trying to deal with applications running on the same machine - have I picked that up correctly? It might help if you explained your architecture in a bit more detail, so I can understand exactly what you're trying to achieve.


Now a bit further in your message you talk about a listener library.
I'd like to know a bit more about this.
How would an external library be able to listen to anything happening within my app ?
Would it be listening on the DB queries ?


No, not at all. Let's say you've got two applications, App1 and App2. You'll write some library code that both applications share, that probably looks something like this (with *wide* variations depending on language/platform - I've just written pseudo code to keep things simple):

-------------------------------------

function init = do
    read_config_file_for_this_app
    open_connection_to_broker
    store_connection_somewhere_in_memory
end

function listen = do
    get_connection_from_memory
    read_message_from_broker
    pass_message_to_application_thread_somehow
    listen
end

function publish = do
    get_connection_from_memory
    send_message_to_broker
end

-------------------------------------

Now in your applications, you'll call the shared 'init' library function when you're starting up to bootstrap the connection to the broker. When your application is publishing data, it calls publish and if/when you need to subscribe to data then you'll call 'listen'. The fact is that 'how to listen' for incoming messages really depends on how you're going to use them. But the point is that the applications read from and write to the messaging broker, and do so independently of database tables. You *may* decide to do something like write a middle-man application that periodically reads a database table and publishes each row to the messaging broker so it can be read from a queue, or do that with a worker thread instead of a separate application. I would *not* do anything here with the database though. If applications need to share data, then **they should send it to one another via message queues.** If they need to persist data, they should persist their own data in their own tables in the database, but they should **not use the database to communicate with one another.** That is the key thing with using messaging instead shared data(bases).

There is an overhead in sending (and in some cases, duplicating) data between applications of course. This is *more* than compensated for by the reduced coupling that comes from integrating using messaging technology. This approach may not be suited to integrating applications that are running on the same physical machine and are tightly and deliberately coupled however. I can't really elaborate on the suitability of messaging for your project without understanding a good deal more about it I'm afraid.

I hope that clears a few things up at least! :)

Cheers,
Tim




_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Loading...