3.3.x dynamic shovel, delivery-mode?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

3.3.x dynamic shovel, delivery-mode?

Angelo Turetta
I like very much the new dynamic shovels in 3.3, but I'm missing one key feature: how can I force the delivery-mode of messages in transit?

The old static shovels allowed me to specify:

{publish_properties, [{delivery_mode, 2}]},

While the new api used to define dynamic ones seems not to allow any publish_properties, or at least there is no docs about it.

Is it possible to make messages persistent with the help of dynamic shovels? Any chance that default parameters specified for static shovels would apply to dynamic ones?

Thank you
Angelo Turetta

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: 3.3.x dynamic shovel, delivery-mode?

Simon MacMullen-2
On 11/06/14 15:35, Angelo Turetta wrote:
> I like very much the new dynamic shovels in 3.3, but I'm missing one key
> feature: how can I force the delivery-mode of messages in transit?

At the moment you can't.

> The old static shovels allowed me to specify:
>
> {publish_properties, [{delivery_mode, 2}]},
>
> While the new api used to define dynamic ones seems not to allow any
> publish_properties, or at least there is no docs about it.

No, there's no equivalent.

One goal for the dynamic shovel was to be a bit simpler than the static
shovel, which does make it a bit less flexible.

> Is it possible to make messages persistent with the help of dynamic
> shovels? Any chance that default parameters specified for static shovels
> would apply to dynamic ones?

Hmm, that's an interesting idea. Alternatively, switching delivery_mode
might be a common enough thing to be worth special casing, the way we
already do for exchange and routing key. Hmm.

Cheers, Simon

--
Simon MacMullen
RabbitMQ, Pivotal
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: 3.3.x dynamic shovel, delivery-mode?

Laing, Michael P.
+1 on that. 

We will not be able to use dynamic shovels in our proxies and product clusters until we can set delivery_mode, in our case to 1 as we never persist messages in the core clusters.

Actually we set user_id too.

This usage is actually long-lived rather than dynamic in the usual sense, but it is very inconvenient to have to restart to change configuration, particularly since everything else needed to set up a new vhost in our environment can be done dynamically.

ml


On Thu, Jun 12, 2014 at 11:29 AM, Simon MacMullen <[hidden email]> wrote:
On 11/06/14 15:35, Angelo Turetta wrote:
I like very much the new dynamic shovels in 3.3, but I'm missing one key
feature: how can I force the delivery-mode of messages in transit?

At the moment you can't.


The old static shovels allowed me to specify:

{publish_properties, [{delivery_mode, 2}]},

While the new api used to define dynamic ones seems not to allow any
publish_properties, or at least there is no docs about it.

No, there's no equivalent.

One goal for the dynamic shovel was to be a bit simpler than the static shovel, which does make it a bit less flexible.


Is it possible to make messages persistent with the help of dynamic
shovels? Any chance that default parameters specified for static shovels
would apply to dynamic ones?

Hmm, that's an interesting idea. Alternatively, switching delivery_mode might be a common enough thing to be worth special casing, the way we already do for exchange and routing key. Hmm.

Cheers, Simon

--
Simon MacMullen
RabbitMQ, Pivotal
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: 3.3.x dynamic shovel, delivery-mode?

Simon MacMullen-2
On 12/06/2014 5:19PM, Laing, Michael wrote:
> We will not be able to use dynamic shovels in our proxies and product
> clusters until we can set delivery_mode, in our case to 1 as we never
> persist messages in the core clusters.
>
> Actually we set user_id too.

Thanks, that's useful to know.

Cheers, Simon

--
Simon MacMullen
RabbitMQ, Pivotal
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: 3.3.x dynamic shovel, delivery-mode?

mc717990
We also set the following in our shovels.  Looking at the dynamic support, some of these appear to be able to be set.

delivery_mode - we override right now as well but I'm probably going to disable this and let whatever originally was published just go through. Course, looks like Michael needs this

reconnect_delay - we set to 30 seconds (default seems to be 5) as we were getting spammed during data center disconnects with the shorter retries.  It appears configuring this dynamically is supported from what I can tell of the source code.
prefetch_count - we set this to 1500 which seemed to be a pretty decent number for our systems.  If I'm looking at the right setting (the rabbitmq_shovel.app.src) this is defaulted to 1000.

headers (currently) - we set a source on a header specifying where the message came from.  Because though that overrides any message published headers, we're looking to use the cluster_id property instead.  Though, just digging through the source, I see "AddHeaders" from <<"add-forward-headers">>.  Looks like the dynamic shovel finally adds that support without replacing existing headers!  Sadly, it's not in the static shovel :(  I may play with adding this myself for some erlang programming learning.






On Fri, Jun 13, 2014 at 4:20 AM, Simon MacMullen <[hidden email]> wrote:
On 12/06/2014 5:19PM, Laing, Michael wrote:
We will not be able to use dynamic shovels in our proxies and product
clusters until we can set delivery_mode, in our case to 1 as we never
persist messages in the core clusters.

Actually we set user_id too.

Thanks, that's useful to know.


Cheers, Simon

--
Simon MacMullen
RabbitMQ, Pivotal
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss



--
Jason McIntosh
https://github.com/jasonmcintosh/
573-424-7612

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss