Handle Stale Connections on Clients

[cogitate]

Hi Mathias et al:

Given:
-- java rabbitmq client 2.8.2
-- rabbitmq broker 2.8.2
-- wrote an amqp client library (java) that has the following semantics:
[1] setting up connection properties
 [1.1] username, password (authentication)
 [1.2] host, port ( endpoint )
 [1.3] # of threads(concurrent users), # of connections(conn.), conn. timeout, receive timeout
 * # of usable clients effectively = threads x connections

[2] A manager sets up a pool for fair-sharing of resources per thread.
 [2.1] sets up a connection pool
 [2.2] sets up request/reply channels
 [2.3] sets up re-usable reply-queues

[3] A client class derived from the manager
 [3.1] this pre-configured (with properties in [2])
 [3.2] specify exchange(context), routing-key(service), payload(a message)
 

when a broker is shutdown and restarted, the stack i get is :
[ERROR:clean connection shutdown; reason: Attempt to use closed channel]
com.rabbitmq.client.AlreadyClosedException: clean connection shutdown; reason: Attempt to use closed channel
        at com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:190)
        at com.rabbitmq.client.impl.AMQChannel.rpc(AMQChannel.java:223)
        at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:209)
        at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:118)
        at com.rabbitmq.client.impl.ChannelN.exchangeDeclarePassive(ChannelN.java:660)
        at com.rabbitmq.client.impl.ChannelN.exchangeDeclarePassive(ChannelN.java:61)
        at com.xx.s2.amqp.AmqpClient.doSend(AmqpClient.java:187)
        at com.xx.s2.amqp.AmqpClient.callService(AmqpClient.java:80)
        at com.xx.s2.amqp.StaleConnectionTest.callLoopTest(StaleConnectionTest.java:132)
        at com.xx.s2.amqp.StaleConnectionTest.loopTest(StaleConnectionTest.java:86)

at this point, i have to shutdown the client using the library and restart it. is there a way to handle this exception and re-establish the connection? any utilities that will allow me to do this "automatically"?

thanks and regards,
-monish

[Jerry Kuch]

Hi, Monish:

If you're using the naked Java client, you'll have to implement your own reconnection logic.  If you're using Spring AMQP there's some helper code that wraps your connections with some retry logic for you.

Best regards,
Jerry

On Tue, Jul 10, 2012 at 9:47 AM, cogitate <[hidden email]> wrote:

Hi Mathias et al:

Given:
-- java rabbitmq client 2.8.2
-- rabbitmq broker 2.8.2
-- wrote an amqp client library (java) that has the following semantics:
[1] setting up connection properties
 [1.1] username, password (authentication)
 [1.2] host, port ( endpoint )
 [1.3] # of threads(concurrent users), # of connections(conn.), conn.
timeout, receive timeout
 * # of usable clients effectively = threads x connections

[2] A manager sets up a pool for fair-sharing of resources per thread.
 [2.1] sets up a connection pool
 [2.2] sets up request/reply channels
 [2.3] sets up re-usable reply-queues

[3] A client class derived from the manager
 [3.1] this pre-configured (with properties in [2])
 [3.2] specify exchange(context), routing-key(service), payload(a message)


when a broker is shutdown and restarted, the stack i get is :
[ERROR:clean connection shutdown; reason: Attempt to use closed channel]
com.rabbitmq.client.AlreadyClosedException: clean connection shutdown;
reason: Attempt to use closed channel
        at com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:190)
        at com.rabbitmq.client.impl.AMQChannel.rpc(AMQChannel.java:223)
        at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:209)
        at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:118)
        at
com.rabbitmq.client.impl.ChannelN.exchangeDeclarePassive(ChannelN.java:660)
        at
com.rabbitmq.client.impl.ChannelN.exchangeDeclarePassive(ChannelN.java:61)
        at com.xx.s2.amqp.AmqpClient.doSend(AmqpClient.java:187)
        at com.xx.s2.amqp.AmqpClient.callService(AmqpClient.java:80)
        at
com.xx.s2.amqp.StaleConnectionTest.callLoopTest(StaleConnectionTest.java:132)
        at com.xx.s2.amqp.StaleConnectionTest.loopTest(StaleConnectionTest.java:86)

at this point, i have to shutdown the client using the library and restart
it. is there a way to handle this exception and re-establish the connection?
any utilities that will allow me to do this "automatically"?

thanks and regards,
-monish

--
View this message in context: http://rabbitmq.1065348.n5.nabble.com/Handle-Stale-Connections-on-Clients-tp20687.html
Sent from the RabbitMQ mailing list archive at Nabble.com.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[cogitate]

Jerry Kuch-2 wrote

Hi, Monish:

If you're using the naked Java client, you'll have to implement your own
reconnection logic.  If you're using Spring AMQP there's some helper code
that wraps your connections with some retry logic for you.

Best regards,
Jerry

thanks much Jerry for such a prompt reply. I would love to re-use spring-amqp and use the wrapped connection( it seems CachedConnectionFactory + SimpleListenerContainer ).
what i am wondering for a simple amqp client that simply wants to re-connect when a broker shutsdown, is there any way i can re-initialize all my reply-queues( i have uuids associated with them), channels (wrapped channel objects associated with connections) without having to re-write the code with spring-amqp?

sorry to be presume that you would know the answer to this.....but a simple logic/example would be great of how to go about it.
the spring-amqp project seems to be quite complicated as it seems to have logic for transactions ( also it's not entirely clear to me if i need to write a listener container or just register a shutdownlistener etc).

thanks and regards,
-monish

[Gary Russell]

See http://static.springsource.org/spring-amqp/docs/1.1.1.RELEASE/reference/html/#d0e999

It shouldn't take much heavy lifting to use spring-amqp instead of the raw apis, and it will take care of reconnecting, setting up queues etc, via the ConnectionListener callback in the RabbitAdmin class.

On the receiving side, you don't have to write any special listeners etc to use spring-amqp; you can just write your code as a POJO and the container uses an adapter to call your code. On the outbound side just use a RabbitTemplate; in most cases you can use high-level APIs, while still having access to the low level APIs (channel) if you ever need them.

If you can't move to spring-amqp, you might be able to leverage some of the code from RabbitAdmin.

Gary

On 07/11/2012 02:16 PM, cogitate wrote:

Jerry Kuch-2 wrote

Hi, Monish:

If you're using the naked Java client, you'll have to implement your own
reconnection logic.  If you're using Spring AMQP there's some helper code
that wraps your connections with some retry logic for you.

Best regards,
Jerry

thanks much Jerry for such a prompt reply. I would love to re-use
spring-amqp and use the wrapped connection( it seems CachedConnectionFactory
+ SimpleListenerContainer ). 
what i am wondering for a simple amqp client that simply wants to re-connect
when a broker shutsdown, is there any way i can re-initialize all my
reply-queues( i have uuids associated with them), channels (wrapped channel
objects associated with connections) without having to re-write the code
with spring-amqp?

sorry to be presume that you would know the answer to this.....but a simple
logic/example would be great of how to go about it. 
the spring-amqp project seems to be quite complicated as it seems to have
logic for transactions ( also it's not entirely clear to me if i need to
write a listener container or just register a shutdownlistener etc).

thanks and regards,
-monish



--
View this message in context: http://rabbitmq.1065348.n5.nabble.com/Handle-Stale-Connections-on-Clients-tp20687p20721.html
Sent from the RabbitMQ mailing list archive at Nabble.com.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[cogitate]

Thanks Gary!
i also use spring-amqp and amqp inbound listener in spring-integration. however, my issue is with reply-queue implementation of spring-amqp. i need pre-made reply-queues that can be re-used (setup by configuration) even for RPC semantics. i believe spring-amqp for RPC uses anonymous reply-queues and also does consumer.cancel(benchmarks showed a 5ms delay for this on 24 core cpu with broker and client running on the same box), which turns out to have performance impact. thus, this rules out use of RabbitTemplate ( i wish it had a way to use pre-made reply-queues that could be re-used).

thus, my wrappers around raw rabbitmq apis and even if i use CachingConnectionFactory, i'd need to register a callback listener to re-register my channels and reply-queues.

will definitely look at RabbitAdmin to learn how to do this.

thanks and regards,
-monish

Gary Russell wrote

See http://static.springsource.org/spring-amqp/docs/1.1.1.RELEASE/reference/html/#d0e999

It shouldn't take much heavy lifting to use spring-amqp instead of the raw apis, and it will take care of reconnecting, setting up queues etc, via the ConnectionListener callback in the RabbitAdmin class.

On the receiving side, you don't have to write any special listeners etc to use spring-amqp; you can just write your code as a POJO and the container uses an adapter to call your code. On the outbound side just use a RabbitTemplate; in most cases you can use high-level APIs, while still having access to the low level APIs (channel) if you ever need them.

If you can't move to spring-amqp, you might be able to leverage some of the code from RabbitAdmin.

Gary

[Gary Russell]

Sorry for the delay.

The current spring-amqp (1.1.1) now supports a 'fixed' reply queue for request/reply scenarios.

See

http://static.springsource.org/spring-amqp/docs/1.1.1.RELEASE/reference/html/#request-reply

We added this specifically for the performance impact you cite.

Gary

On 07/11/2012 05:38 PM, cogitate wrote:

Thanks Gary! 
i also use spring-amqp and amqp inbound listener in spring-integration.
however, my issue is with reply-queue implementation of spring-amqp. i need
pre-made reply-queues that can be re-used (setup by configuration) even for
RPC semantics. i believe spring-amqp for RPC uses anonymous reply-queues and
also does consumer.cancel(benchmarks showed a 5ms delay for this on 24 core
cpu with broker and client running on the same box), which turns out to have
performance impact. thus, this rules out use of RabbitTemplate ( i wish it
had a way to use pre-made reply-queues that could be re-used).

thus, my wrappers around raw rabbitmq apis and even if i use
CachingConnectionFactory, i'd need to register a callback listener to
re-register my channels and reply-queues.

will definitely look at RabbitAdmin to learn how to do this.

thanks and regards,
-monish


Gary Russell wrote

See
http://static.springsource.org/spring-amqp/docs/1.1.1.RELEASE/reference/html/#d0e999

It shouldn't take much heavy lifting to use spring-amqp instead of the raw
apis, and it will take care of reconnecting, setting up queues etc, via
the ConnectionListener callback in the RabbitAdmin class.

On the receiving side, you don't have to write any special listeners etc
to use spring-amqp; you can just write your code as a POJO and the
container uses an adapter to call your code. On the outbound side just use
a RabbitTemplate; in most cases you can use high-level APIs, while still
having access to the low level APIs (channel) if you ever need them.

If you can't move to spring-amqp, you might be able to leverage some of
the code from RabbitAdmin.

Gary

--
View this message in context: http://rabbitmq.1065348.n5.nabble.com/Handle-Stale-Connections-on-Clients-tp20687p20725.html
Sent from the RabbitMQ mailing list archive at Nabble.com.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[cogitate]

Thanks Gary again for pointers!

** ah, so all i have to do is pool rabbit-template objects and i'll get the benefit of re-connecting?
** do you think we could have a hard reconnect after every "x" minutes so that if the brokers are behind a VIP and some are brought down and brought back up, the VIP will do fair-load-balancing?

regards,
-monish

Gary Russell wrote

Sorry for the delay.

The current spring-amqp (1.1.1) now supports a 'fixed' reply queue for request/reply scenarios.

See

http://static.springsource.org/spring-amqp/docs/1.1.1.RELEASE/reference/html/#request-reply

We added this specifically for the performance impact you cite.

Gary

[Gary Russell]

You'll have to catch the exception on the  *sendAndReceive() but, yes,
it will reconnect if you try again.

There's no out-of-the-box mechanism to recycle the connection; you could
add a connection listener to get a reference to the connection and close
it from time to time, but this will likely impact in-flight messages.

Gary

On 07/12/2012 01:03 PM, cogitate wrote:

> Thanks Gary again for pointers!
>
> ** ah, so all i have to do is pool rabbit-template objects and i'll get the
> benefit of re-connecting?
> ** do you think we could have a hard reconnect after every "x" minutes so
> that if the brokers are behind a VIP and some are brought down and brought
> back up, the VIP will do fair-load-balancing?
>
> regards,
> -monish
>
>
> Gary Russell wrote
>> Sorry for the delay.
>>
>> The current spring-amqp (1.1.1) now supports a 'fixed' reply queue for
>> request/reply scenarios.
>>
>> See
>>
>> http://static.springsource.org/spring-amqp/docs/1.1.1.RELEASE/reference/html/#request-reply
>>
>> We added this specifically for the performance impact you cite.
>>
>> Gary
>>
>
>
>
> --
> View this message in context: http://rabbitmq.1065348.n5.nabble.com/Handle-Stale-Connections-on-Clients-tp20687p20761.html
> Sent from the RabbitMQ mailing list archive at Nabble.com.
> _______________________________________________
> rabbitmq-discuss mailing list
> [hidden email]
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[cogitate]

hi gary:
 using spring-amqp 1.1.1.RELEASE it seems i have to change the version of spring-rabbit to 1.0.0.RELEASE

         <dependency>
            <groupId>org.springframework.amqp</groupId>
            <artifactId>spring-amqp</artifactId>
            <version>1.1.1.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.amqp</groupId>
            <artifactId>spring-rabbit</artifactId>
           
            <version>1.0.0.RELEASE</version>
        </dependency>

the problem is this forms the core to a spring-integration webapp for which i need spring-rabbit:1.1.0.RELEASE which has support for dead-letter-exchange.

even when i upgrade spring-rabbit:1.1.1.RELEASE i get the same error as with 1.1.0.RELEASE of spring-rabbit.

the error is :
Caused by: org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'replies' is defined
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:527)


what would you recommend we do?

thanks and regards,
-monish

[Matthias Radestock-3]

On 06/08/12 23:39, cogitate wrote:
>   using spring-amqp 1.1.1.RELEASE it seems i have to change the version of
> spring-rabbit to 1.0.0.RELEASE [...]

I see you have asked the same question on the spring-amqp forum, which
is the right place for it so please continue the discussion there.

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss