Messages lost with RabbitMQ Federation

[Roman Gaufman]

I have 2 RabbitMQ Nodes. One node is on a US server and one is on a UK server. They are both connected to the same VPN.

I'm consuming the messages from the local nodes in both in the US and UK with:

AMQP.start('amqp://guest:guest@localhost:25672') do |connection, open_ok|
  channel  = AMQP::Channel.new(connection)
  exchange = AMQP::Exchange.new(channel, "x-federation", "xanview", :durable => true,
    :arguments => {"upstream-set" => "my-upstreams", "type" => "topic", "durable" => "true"})
  channel.queue("testqueue_uk", :durable => true) do |queue| # testqueue_us in the US
    queue.bind(exchange, :routing_key => '#')
    queue.subscribe do |metadata, payload|
      i+=1
      puts "#{i}: Recieved message: #{payload.inspect}."
    end
  end
end

I'm producing messages locally to the local nodes in both the US and UK with:

EventMachine.run do
  AMQP.connect('amqp://guest:guest@localhost:25672') do |connection, open_ok|
    channel = AMQP::Channel.new(connection)
    exchange = AMQP::Exchange.new(channel, "x-federation", "xanview", :durable => true, :arguments => {"upstream-set" => "my-upstreams", "type" => "topic", "durable" => "true"})
    EventMachine.add_periodic_timer(0.1) do
      $i+=1
      puts "#{$i}: publishing msg"
      $exchange.publish("#{$i} this is a test message from the US", :routing_key => "some.topic", :persistent => true )
    end
  end
end

This all seems to work reliably until I try to kill the connection: I shutdown the VPN and see messages being queued in queues called: "federation: xanview -> node2" (on the UK server) and "federation: xanview -> node1" (on the US server) as expected.

However when I restart the VPN, the UK server has received 43737 messages while the US only received 43708 messages. Where did 29 messages disappear?

Both node configs look like this with the only exception of the US connecting to federation on 10.8.0.22 and the UK connecting to federation on 10.8.0.33:

[
  {rabbit, [
    {tcp_listeners, [25672]}
  ]},
  {rabbitmq_mochiweb, [
    {listeners,        [{mgmt, [{port, 55555}]}]},
    {default_listener, [{port, 55550}]},
    {contexts,         [{rabbit_mgmt, mgmt}]}
  ]},

{rabbitmq_federation,
   [ {exchanges, [[{exchange,     "xanview"},
                   {virtual_host, "/"},
                   {type,         "topic"},
                   {durable,      true},
                   {auto_delete,  false},
                   {internal,     false},
                   {upstream_set, "my-upstreams"}]
                 ]},
     {upstream_sets, [{"my-upstreams", [[{connection, "upstream-server"}]]}]},
     {connections, [{"upstream-server", [{host,            "10.8.0.22"},
                                         {protocol,        "amqp"},
                                         {port,            25672},
                                         {virtual_host,    "/"},
                                         {username,        "guest"},
                                         {password,        "guest"},
                                         {mechanism,       default},
                                         {prefetch_count,  1000},
                                         {reconnect_delay, 5},
                                         {ha_policy,       "all"}
                                        ]}
                   ]},
     {local_username, "guest"},
     {local_nodename, "node2"}
   ]
  }

].

Any ideas why messages are lost? What could be causing it?

Roman

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Simon MacMullen-2]

On 05/07/12 12:08, Roman Gaufman wrote:
> However when I restart the VPN, the UK server has received 43737
> messages while the US only received 43708 messages. Where did 29
> messages disappear?

Is it instead possible that 29 messages have been duplicated? Remember
that when you kill an AMQP connection where A is consuming from B, A can
have sent acknowledgements which B has not yet received. In this case
when the federation link comes back up again, those messages will be
retransmitted. You will be able to see this as there will be a
redelivered flag set in the federation headers. (But don't just throw
away redelivered messages - they might not have reached A when the
connection went down.)

I suggest you modify your tests to send a fixed number of messages and
see what happens.

Cheers, Simon

--
Simon MacMullen
RabbitMQ, VMware
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Roman Gaufman | Xanview]

Ah! - very possible, changing my tests :) - I tried to reproduce briefly but wasn't able to - I will leave it running for a few hours randomly killing the connection and report back.

On 5 July 2012 14:09, Simon MacMullen <[hidden email]> wrote:

On 05/07/12 12:08, Roman Gaufman wrote:

However when I restart the VPN, the UK server has received 43737
messages while the US only received 43708 messages. Where did 29
messages disappear?

Is it instead possible that 29 messages have been duplicated? Remember that when you kill an AMQP connection where A is consuming from B, A can have sent acknowledgements which B has not yet received. In this case when the federation link comes back up again, those messages will be retransmitted. You will be able to see this as there will be a redelivered flag set in the federation headers. (But don't just throw away redelivered messages - they might not have reached A when the connection went down.)

I suggest you modify your tests to send a fixed number of messages and see what happens.

Cheers, Simon

--
Simon MacMullen
RabbitMQ, VMware

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[rabbithole]

Hi
What VPN are you using?
Is it hardware VPN both ends?
Or software?

Regards
RH