queue selection based on a 'feature list'

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

queue selection based on a 'feature list'

Lawrence Freil
Hello,

I'm attempting to create a distributed message processing system in which
messages posted requires a list of features and can only be posted to
queues which have all the required features. In addition it needs to act as
a direct exchange in that only one queue should receive a message. In
looking over both the headers and topic exchanges it seems the wildcards
are specified the wrong way around for this. For example, I have servers
which can support optional features of w,x,y and z. When I publish a
message it may require x and y. I don't care if the server it runs on
supports w and z. So what I really need is the message to support the
'pattern'' and the queues to support the 'string'. Is there a way to
accomplish this with the current exchanges?

Thanks.


--
Lawrence Freil

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: queue selection based on a 'feature list'

Simon MacMullen-2
On 30/10/13 13:54, Lawrence Freil wrote:
> Is there a way to accomplish this with the current exchanges?

I'm not sure there is. I can't think of one anyway.

I don't think it would be too hard to implement as an exchange type plugin.

Cheers, Simon

--
Simon MacMullen
RabbitMQ, Pivotal
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: queue selection based on a 'feature list'

Alvaro Videla-2
In reply to this post by Lawrence Freil
On Wed, Oct 30, 2013 at 2:54 PM, Lawrence Freil <[hidden email]> wrote:
In addition it needs to act as a direct exchange in that only one queue should receive a message

BTW, on the direct exchange, every queue that is bound with a particular routing key will receive the message. An exchange that routes messages to one queue is the default exchange (aka anon exchange). 

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: queue selection based on a 'feature list'

Alvaro Videla-2
Adding back rabbitmq-discuss to the loop.


On Wed, Oct 30, 2013 at 3:26 PM, Lawrence Freil <[hidden email]> wrote:
Alvaro,

I should have stated 'consumer' instead of 'queue'. The docs (and my testing indicates that it does work this way) states:

Direct exchanges are often used to distribute tasks between multiple workers (instances of the same application) in a round robin manner. When doing so, it is important to understand that, in AMQP 0-9-1, messages are load balanced between consumers and not between queues.

The exchanges will route messages to queues based on the routing key used when publishing the message and the routing key used to bind the queue(s) to the exchange. Depending on the exchange is how that routing key will be used during routing. So one message could end up in more than one queue.

Once the message reached one queue (note that the message could have been delivered to more than one queue), then the messages on a particular queue will be round-robin'ed across consumers.

For more details take a look here: http://www.rabbitmq.com/tutorials/amqp-concepts.html

Regards,

Alvaro

 





On Oct 30 2013, Alvaro Videla wrote:

On Wed, Oct 30, 2013 at 2:54 PM, Lawrence Freil <[hidden email]> wrote:

In addition it needs to act as a direct exchange in that only one queue
should receive a message


BTW, on the direct exchange, every queue that is bound with a particular
routing key will receive the message. An exchange that routes messages to
one queue is the default exchange (aka anon exchange).


--
Lawrence Freil
[hidden email]
<a href="tel:770-619-1884" value="+17706191884" target="_blank">770-619-1884


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: queue selection based on a 'feature list'

Emile Joubert
In reply to this post by Lawrence Freil
On 30/10/13 13:54, Lawrence Freil wrote:

> Is there a way to accomplish this with the current exchanges?

There may be a way to approximate this with a topic exchange if you
associate features with consumers rather than queues. You could create a
queue for each combination of features and bind so that it received
messages that require exactly those features.

E.g. you could declare queues

____, w___, _x___, __y_, ___z,
wx__, ... , w__z, ..., wxyz.

and bind them:

____ with binding key _._._._
w___ with binding key w._._._
_x__ with binding key _.x._._
__y_ with binding key _._.y._
___z with binding key _._._.z
...
wx__ with binding key w.x._._
w__z with binding key w._._.z
...
wxyz with binding key w.x.y.z

Consumers can then consume from all queues that contain messages that
demand a subset of the features offered by that consumer, e.g. a
consumer that offers

x and y consumes from ____ and _x__ and __y_ and _xy_
only z  consumes from ____ and ___z
w,x,y,z consumes from all queues

Then if you publish a message with routing key _.x.y._ it will arrive at
queue _xy_ and be sent to a consumer that is guaranteed to offer x and y.


If the number of features is large or rapidly changing then this will
become unmanageable. If the features are strictly associated with queues
rather than consumers then it also won't work.



-Emile






_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: queue selection based on a 'feature list'

Lawrence Freil
Emile,

Thanks for the reply.

I had considered this approach and could use this for initial testing,
however the feature set that I am looking for is somewhat large and could
expand, each time doubling the combinations. I think the end result will be
creating an exchange plugin very similar to the topic exchange but where
the '*' and '#' can also be used in the message routing keys. Using this
wouldn't break any existing topic exchange usage but would allow messages
to be crafted with 'I don't care', and if a trailing '#' was always used in
the message, adding new features to the servers wouldn't break existing
producers.

On Oct 30 2013, Emile Joubert wrote:

>On 30/10/13 13:54, Lawrence Freil wrote:
>
>> Is there a way to accomplish this with the current exchanges?
>
>There may be a way to approximate this with a topic exchange if you
>associate features with consumers rather than queues. You could create a
>queue for each combination of features and bind so that it received
>messages that require exactly those features.
>
>E.g. you could declare queues
>
>____, w___, _x___, __y_, ___z,
>wx__, ... , w__z, ..., wxyz.
>
>and bind them:
>
>____ with binding key _._._._
>w___ with binding key w._._._
>_x__ with binding key _.x._._
>__y_ with binding key _._.y._
>___z with binding key _._._.z
>...
>wx__ with binding key w.x._._
>w__z with binding key w._._.z
>...
>wxyz with binding key w.x.y.z
>
>Consumers can then consume from all queues that contain messages that
>demand a subset of the features offered by that consumer, e.g. a
>consumer that offers
>
>x and y consumes from ____ and _x__ and __y_ and _xy_
>only z  consumes from ____ and ___z
>w,x,y,z consumes from all queues
>
>Then if you publish a message with routing key _.x.y._ it will arrive at
>queue _xy_ and be sent to a consumer that is guaranteed to offer x and y.
>
>
>If the number of features is large or rapidly changing then this will
>become unmanageable. If the features are strictly associated with queues
>rather than consumers then it also won't work.
>
>
>
>-Emile
>
>
>
>
>
>

--
Lawrence Freil
[hidden email]
770-619-1884
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

queue selection based on a 'feature list'

Laing, Michael P.
In reply to this post by Lawrence Freil
Perhaps this approach will help. For example:

Using a key pattern of 'w.x.y.z':

A server supports: 'w.x.y.-'

A message needs: '-.x.y.-'

Bind with the complement of the server pattern: '*.*.*.z'

Publish with the complement of the message pattern: 'w.-.-.z'

If server capabilities overlap there is still a routing problem...

ml




On Wed, Oct 30, 2013 at 9:54 AM, Lawrence Freil <<a href="javascript:_e({}, &#39;cvml&#39;, &#39;lef@apago.com&#39;);" target="_blank">lef@...> wrote:
Hello,

I'm attempting to create a distributed message processing system in which messages posted requires a list of features and can only be posted to queues which have all the required features. In addition it needs to act as a direct exchange in that only one queue should receive a message. In looking over both the headers and topic exchanges it seems the wildcards are specified the wrong way around for this. For example, I have servers which can support optional features of w,x,y and z. When I publish a message it may require x and y. I don't care if the server it runs on supports w and z. So what I really need is the message to support the 'pattern'' and the queues to support the 'string'. Is there a way to accomplish this with the current exchanges?

Thanks.


--
Lawrence Freil

_______________________________________________
rabbitmq-discuss mailing list
<a href="javascript:_e({}, &#39;cvml&#39;, &#39;rabbitmq-discuss@lists.rabbitmq.com&#39;);" target="_blank">rabbitmq-discuss@lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: queue selection based on a 'feature list'

Chris-3
In reply to this post by Lawrence Freil
I've had cases where allowing the publisher to specify wildcards would help too.  I think it's a very useful custom exchange and would gladly use it if RabbitMQ or anyone else ever published it as an extension.

-Chris


On Wed, Oct 30, 2013 at 11:29 AM, Lawrence Freil <[hidden email]> wrote:
Emile,

Thanks for the reply.

I had considered this approach and could use this for initial testing, however the feature set that I am looking for is somewhat large and could expand, each time doubling the combinations. I think the end result will be creating an exchange plugin very similar to the topic exchange but where the '*' and '#' can also be used in the message routing keys. Using this wouldn't break any existing topic exchange usage but would allow messages to be crafted with 'I don't care', and if a trailing '#' was always used in the message, adding new features to the servers wouldn't break existing producers.


On Oct 30 2013, Emile Joubert wrote:

On 30/10/13 13:54, Lawrence Freil wrote:

Is there a way to accomplish this with the current exchanges?

There may be a way to approximate this with a topic exchange if you
associate features with consumers rather than queues. You could create a
queue for each combination of features and bind so that it received
messages that require exactly those features.

E.g. you could declare queues

____, w___, _x___, __y_, ___z,
wx__, ... , w__z, ..., wxyz.

and bind them:

____ with binding key _._._._
w___ with binding key w._._._
_x__ with binding key _.x._._
__y_ with binding key _._.y._
___z with binding key _._._.z
...
wx__ with binding key w.x._._
w__z with binding key w._._.z
...
wxyz with binding key w.x.y.z

Consumers can then consume from all queues that contain messages that
demand a subset of the features offered by that consumer, e.g. a
consumer that offers

x and y consumes from ____ and _x__ and __y_ and _xy_
only z  consumes from ____ and ___z
w,x,y,z consumes from all queues

Then if you publish a message with routing key _.x.y._ it will arrive at
queue _xy_ and be sent to a consumer that is guaranteed to offer x and y.


If the number of features is large or rapidly changing then this will
become unmanageable. If the features are strictly associated with queues
rather than consumers then it also won't work.



-Emile







--
Lawrence Freil
[hidden email]
<a href="tel:770-619-1884" value="+17706191884" target="_blank">770-619-1884
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: queue selection based on a 'feature list'

Alvaro Videla-2
Hi,

I've created a "reverse topic exchange" plugin to accomplish uses cases like yours.


Feedback is most welcome.

Regards,

Alvaro


On Wed, Oct 30, 2013 at 5:28 PM, Chris <[hidden email]> wrote:
I've had cases where allowing the publisher to specify wildcards would help too.  I think it's a very useful custom exchange and would gladly use it if RabbitMQ or anyone else ever published it as an extension.

-Chris


On Wed, Oct 30, 2013 at 11:29 AM, Lawrence Freil <[hidden email]> wrote:
Emile,

Thanks for the reply.

I had considered this approach and could use this for initial testing, however the feature set that I am looking for is somewhat large and could expand, each time doubling the combinations. I think the end result will be creating an exchange plugin very similar to the topic exchange but where the '*' and '#' can also be used in the message routing keys. Using this wouldn't break any existing topic exchange usage but would allow messages to be crafted with 'I don't care', and if a trailing '#' was always used in the message, adding new features to the servers wouldn't break existing producers.


On Oct 30 2013, Emile Joubert wrote:

On 30/10/13 13:54, Lawrence Freil wrote:

Is there a way to accomplish this with the current exchanges?

There may be a way to approximate this with a topic exchange if you
associate features with consumers rather than queues. You could create a
queue for each combination of features and bind so that it received
messages that require exactly those features.

E.g. you could declare queues

____, w___, _x___, __y_, ___z,
wx__, ... , w__z, ..., wxyz.

and bind them:

____ with binding key _._._._
w___ with binding key w._._._
_x__ with binding key _.x._._
__y_ with binding key _._.y._
___z with binding key _._._.z
...
wx__ with binding key w.x._._
w__z with binding key w._._.z
...
wxyz with binding key w.x.y.z

Consumers can then consume from all queues that contain messages that
demand a subset of the features offered by that consumer, e.g. a
consumer that offers

x and y consumes from ____ and _x__ and __y_ and _xy_
only z  consumes from ____ and ___z
w,x,y,z consumes from all queues

Then if you publish a message with routing key _.x.y._ it will arrive at
queue _xy_ and be sent to a consumer that is guaranteed to offer x and y.


If the number of features is large or rapidly changing then this will
become unmanageable. If the features are strictly associated with queues
rather than consumers then it also won't work.



-Emile







--
Lawrence Freil
[hidden email]
<a href="tel:770-619-1884" value="+17706191884" target="_blank">770-619-1884
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss



_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss