Exchange and queue naming conventions

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Exchange and queue naming conventions

Bill Moseley
Are there any good conventions to follow when naming queues, exchanges, and bindings?  As I'm adding more queues I'm starting to think some type of naming convention would be very smart.  Just curious what other's do.

For now I'm only using a direct exchange type.  When should I be declaring additional exchanges vs. using the same (or even the default) exchange?

A pattern I'm seeing is I have an app-specific producer than sends a message to a generic consumer (used by multiple apps) but then that consumer sends a message to another app-specific consumer.  What I'm doing is passing the return queue name in the reply_to attribute.  I'm not passing in the exchange name -- so I'm just using the same exchange for every binding.

I also declare the exchange, queue, and binding in every component (the producer and both consumers).  I assume that's typical as it allows starting the components in any order.  Is that appropriate?

Finally, I have been naming queue and bindings with the same name.  Is there a reason that should not be done?

Thanks,

--
Bill Moseley
[hidden email]

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Exchange and queue naming conventions

David Wragg-4
Hi Bill,

Bill Moseley <[hidden email]> writes:
> Are there any good conventions to follow when naming queues, exchanges, and
> bindings?  As I'm adding more queues I'm starting to think some type of
> naming convention would be very smart.  Just curious what other's do.

I don't think there is any one right way.  When choosing names for
queues, exchanges, and keys, the use of dotted paths is widespread.  But
your questions are more about usage than naming.

> For now I'm only using a direct exchange type.  When should I be declaring
> additional exchanges vs. using the same (or even the default)
> exchange?

It depends on your message exchange patterns.

Generally, you should arrange your application to take full advantage of
the facilities AMQP presents.  If your application ends up dispatching
or filtering messages in a way that could be delegated to RabbitMQ, then
you should do.

Also, consider where you might later want to add new producers and
consumers - even if just for debugging purposes.  The ability to easily
add additional parties into message exchanges is one of the major
advantages of a messaging-based approach as opposed to point-to-point
mechanisms such as web services.

> A pattern I'm seeing is I have an app-specific producer than sends a message
> to a generic consumer (used by multiple apps) but then that consumer sends a
> message to another app-specific consumer.  What I'm doing is passing the
> return queue name in the reply_to attribute.  I'm not passing in the
> exchange name -- so I'm just using the same exchange for every
> binding.

For a request-response pattern, this seems fine.

One variation is to make the exchange a topic exchange rather than a
direct exchange.  This allows you to intercept all replies with a
wildcard binding (e.g. for debugging).

> I also declare the exchange, queue, and binding in every component (the
> producer and both consumers).  I assume that's typical as it allows starting
> the components in any order.  Is that appropriate?

Yes, this kind of thing is perfectly normal with AMQP.

Typically, both the producers and consumers will declare the exchanges
of interest.  But only consumers will declare and bind the queues,
because they know which messages they wish to receive.

> Finally, I have been naming queue and bindings with the same name.  Is there
> a reason that should not be done?

By naming bindings, I assume you are referring to the binding key.
There's nothing wrong with what you describe, except that this sounds
very close to what the AMQP nameless exchange does implicitly.  So
perhaps you could take advantage of that, and so avoid the need to
declare an exchange and explicitly bind your queues.


David

--
David Wragg
Staff Engineer, RabbitMQ
SpringSource, a division of VMware
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Exchange and queue naming conventions

Bill Moseley


On Mon, Jan 24, 2011 at 10:14 AM, David Wragg <[hidden email]> wrote:
Hi Bill,

Bill Moseley <[hidden email]> writes:
> Are there any good conventions to follow when naming queues, exchanges, and
> bindings?  As I'm adding more queues I'm starting to think some type of
> naming convention would be very smart.  Just curious what other's do.

I don't think there is any one right way.  When choosing names for
queues, exchanges, and keys, the use of dotted paths is widespread.  But
your questions are more about usage than naming.

Hi David,

I do think it makes sense to use the dotted notation, and then use topic queues as that makes for very flexible routing options.  I was wondering if anyone had worked out some "best practices" over time that would be useful as we start to add more tasks.

I'm trying to break up some of our existing processes that are very tightly coupled - the workers know quite a bit about the producers.  For example, a worker might be configured to handle just one application in "QA" mode, and another in "production" mode.  And likewise, the producer (in our current system) must know where to send the message ("I'm in QA mode so I need to send to the QA worker").

So, as an example I'm wondering about schemes where an app might use a routing key of "app1.qa.rotate_image" which would allow us to configure consumers that handles all image rotation for all apps and all modes, or separate consumers based on the mode.

BTW -- in that case can I configure binding for three queues:  *.qa.rotate_image. *.production.rotate_image, and then a third for all except "qa" a and "production"?


Finally, I'm still a bit confused when to create new exchanges (assuming all the same type).  Just trying to get some basic guidelines.  Is there any overhead with using more exchanges vs. just more bindings on the same exchange?  Is is common to use separate exchanges for grouping consumers (i.e. an exchange for report queues and another exchange for image manipulation queues)?

Thanks very much for your help,



--
Bill Moseley
[hidden email]

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Exchange and queue naming conventions

michaelklishin
2011/1/25 Bill Moseley <[hidden email]>
I do think it makes sense to use the dotted notation, and then use topic queues as that makes for very flexible routing options.  I was wondering if anyone had worked out some "best practices" over time that would be useful as we start to add more tasks.

1+ year and several apps later I see the following pattern developing on my team:

1. Exchanges are declared by producers.
2. Queues are declared and bound by consumers.
3. In vast majority of cases, consumers are "smart" and producers are very dumb: they just put messages in well-known places. Consumers
    are "smart" in a sense that they almost always keep some persistent state (in Redis, for example), while producers are just monitored in case they crash.
4. Exchanges are named using the following pattern $prefix.$applications_group.$app.$environment. For example, megacorp.mail_delivery.shaper.qa or megacorp.search.online_indexer.production. It works well with multiple installations and if you want to route some data to your customers/partners/archive.

Also, it becomes increasingly obvious that many exchanges and bindings would be unnecessary if we begin to use default exchange more.

 
BTW -- in that case can I configure binding for three queues:  *.qa.rotate_image. *.production.rotate_image, and then a third for all except "qa" a and "production"?

"Except" case is not handled by AMQP very well (not in 0.9.1 I am most familiar with, anyway). Maybe *.other.rotate_image would be good enough?
 

Finally, I'm still a bit confused when to create new exchanges (assuming all the same type).  Just trying to get some basic guidelines.  Is there any overhead with using more exchanges vs. just more bindings on the same exchange?  Is is common to use separate exchanges for grouping consumers (i.e. an exchange for report queues and another exchange for image manipulation queues)?


Unless you have thousands of them on a small box, it probably does not make much difference. For support cases, dealing with apps that use 150 exchanges may be significantly more difficult than if they were using just 3, though.

Hope this helps.
--
MK

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Exchange and queue naming conventions

Jeff Hinrichs
On Tue, Jan 25, 2011 at 1:50 AM, Michael Klishin <[hidden email]> wrote:
2011/1/25 Bill Moseley <[hidden email]>
I do think it makes sense to use the dotted notation, and then use topic queues as that makes for very flexible routing options.  I was wondering if anyone had worked out some "best practices" over time that would be useful as we start to add more tasks.

1+ year and several apps later I see the following pattern developing on my team:

1. Exchanges are declared by producers.
2. Queues are declared and bound by consumers.
3. In vast majority of cases, consumers are "smart" and producers are very dumb: they just put messages in well-known places. Consumers
    are "smart" in a sense that they almost always keep some persistent state (in Redis, for example), while producers are just monitored in case they crash.
4. Exchanges are named using the following pattern $prefix.$applications_group.$app.$environment. For example, megacorp.mail_delivery.shaper.qa or megacorp.search.online_indexer.production. It works well with multiple installations and if you want to route some data to your customers/partners/archive.

Also, it becomes increasingly obvious that many exchanges and bindings would be unnecessary if we begin to use default exchange more.

 
BTW -- in that case can I configure binding for three queues:  *.qa.rotate_image. *.production.rotate_image, and then a third for all except "qa" a and "production"?

"Except" case is not handled by AMQP very well (not in 0.9.1 I am most familiar with, anyway). Maybe *.other.rotate_image would be good enough?
 

Finally, I'm still a bit confused when to create new exchanges (assuming all the same type).  Just trying to get some basic guidelines.  Is there any overhead with using more exchanges vs. just more bindings on the same exchange?  Is is common to use separate exchanges for grouping consumers (i.e. an exchange for report queues and another exchange for image manipulation queues)?


Unless you have thousands of them on a small box, it probably does not make much difference. For support cases, dealing with apps that use 150 exchanges may be significantly more difficult than if they were using just 3, though.

Hope this helps.
--
MK


That was very helpful, thank you.  I am very interested if others concur or have differing wisdom?

Best,

Jeff 

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Exchange and queue naming conventions

tlempart
In reply to this post by michaelklishin
Hi,

Your discussion was very helpful for me. In particular, information about who is responsible for creating what. However, I see one problem in case of two independent applications, where the first one constains producer (PA) and the second one contains consumer (CA). At the beginning there are no exchanges/queues/bindings on the rabbitmq, so if the PA starts first, than the producer creates exchange and starts sending messages. Because consumer is not ready (CA has not yet started), than the queue and binding is not created and the messages are lost. Is there any simple way/pattern to solve such case, I mean to suspend sending messages by producer, until the consumer creates own infrastructure (queue and binding)?
Reply | Threaded
Open this post in threaded view
|

Re: Exchange and queue naming conventions

Jerry Kuch
You might consider having producers use the mandatory flag.  That way if it's not possible for Rabbit to put the message into at least one queue, you'll get it sent back to you via a basic.return and the producer can rethink its life choices accordingly.

Best regards,
Jerry

On Thu, Aug 30, 2012 at 6:53 AM, tlempart <[hidden email]> wrote:
Hi,

Your discussion was very helpful for me. In particular, information about
who is responsible for creating what. However, I see one problem in case of
two independent applications, where the first one constains producer (PA)
and the second one contains consumer (CA). At the beginning there are no
exchanges/queues/bindings on the rabbitmq, so if the PA starts first, than
the producer creates exchange and starts sending messages. Because consumer
is not ready (CA has not yet started), than the queue and binding is not
created and the messages are lost. Is there any simple way/pattern to solve
such case, I mean to suspend sending messages by producer, until the
consumer creates own infrastructure (queue and binding)?



--
View this message in context: http://rabbitmq.1065348.n5.nabble.com/Exchange-and-queue-naming-conventions-tp7254p21702.html
Sent from the RabbitMQ mailing list archive at Nabble.com.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Exchange and queue naming conventions

Matthias Radestock-3
On 30/08/12 23:43, Jerry Kuch wrote:
> You might consider having producers use the mandatory flag.  That way if
> it's not possible for Rabbit to put the message into at least one queue,
> you'll get it sent back to you via a basic.return and the producer can
> rethink its life choices accordingly.

Or *both* producer and consumer could declare the required exchanges,
queues and bindings.

Obviously that only works if they both a) have the required knowledge,
and b) are sufficiently trusted.

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Exchange and queue naming conventions

tlempart
In reply to this post by Jerry Kuch
Thank you Jerry,

  the mandatory flag looks like the perfect solution to my problem.