[rabbitmq_c] Infinite wait in login_with_properties function

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

[rabbitmq_c] Infinite wait in login_with_properties function

Haster
Hi there!

Today I have caught a bug that my program could not stop.
I used a gdb and found that application was on recv method. Call stack:

#0  0x0000003435c0dc81 in recv () from /lib64/libpthread.so.0
#1  0x00002aaaad84ead1 in ?? () from /import/home/yury.alioshinov/linux_i686/5102/lib/librabbitmq_c.so.1.1.1
#2  0x00002aaaad84be83 in ?? () from /import/home/yury.alioshinov/linux_i686/5102/lib/librabbitmq_c.so.1.1.1
#3  0x00002aaaad84c19e in ?? () from /import/home/yury.alioshinov/linux_i686/5102/lib/librabbitmq_c.so.1.1.1
#4  0x00002aaaad84c72c in amqp_simple_wait_method ()
   from /import/home/yury.alioshinov/linux_i686/5102/lib/librabbitmq_c.so.1.1.1
#5  0x00002aaaad84cab6 in ?? () from /import/home/yury.alioshinov/linux_i686/5102/lib/librabbitmq_c.so.1.1.1
#6  0x00002aaaad84ce8a in amqp_login_with_properties ()
   from /import/home/yury.alioshinov/linux_i686/5102/lib/librabbitmq_c.so.1.1.1
#7  0x00002aaaad626313 in RabbitMQ::RabbitMQConnectionImpl::connect (this=0xfaf3a00, hostName=...,
    portNumber=<value optimized out>, userName=..., userPwd=..., heartbeat=0, vhost=...)
    at rabbitmq_source/rabbitmq_connector/RabbitMQConnectionImpl.cpp:115
#8  0x00002aaaad60d900 in rabbitmq_ns::consumer_impl::execute (this=0xb4579d8) at rabbitmq_source/rabbitmq_impl.cpp:823
#9  0x00002b25bdde1aa3 in threads::thread_proc (thr_ptr=0xb4579d8) at has_common_source/source/cpp/threads.cpp:225
#10 0x0000003435c0673d in start_thread () from /lib64/libpthread.so.0
#11 0x00000034350d40cd in clone () from /lib64/libc.so.6

So, method login_with_properties is blocked, and there is some usecase where program execute amqp_send_header in amqp_login_inner without errors but can't get data from amqp_simple_wait_method.

Maybe rabbitmq_c needs nonblock login method?
Reply | Threaded
Open this post in threaded view
|

Re: [rabbitmq_c] Infinite wait in login_with_properties function

alan.antonuk
How frequently does this happen?

Is it something you can reliably reproduce?

-Alan


On Mon, Dec 23, 2013 at 12:12 AM, Haster <[hidden email]> wrote:
Hi there!

Today I have caught a bug that my program could not stop.
I used a gdb and found that application was on recv method. Call stack:

#0  0x0000003435c0dc81 in recv () from /lib64/libpthread.so.0
#1  0x00002aaaad84ead1 in ?? () from
/import/home/yury.alioshinov/linux_i686/5102/lib/librabbitmq_c.so.1.1.1
#2  0x00002aaaad84be83 in ?? () from
/import/home/yury.alioshinov/linux_i686/5102/lib/librabbitmq_c.so.1.1.1
#3  0x00002aaaad84c19e in ?? () from
/import/home/yury.alioshinov/linux_i686/5102/lib/librabbitmq_c.so.1.1.1
#4  0x00002aaaad84c72c in amqp_simple_wait_method ()
   from
/import/home/yury.alioshinov/linux_i686/5102/lib/librabbitmq_c.so.1.1.1
#5  0x00002aaaad84cab6 in ?? () from
/import/home/yury.alioshinov/linux_i686/5102/lib/librabbitmq_c.so.1.1.1
#6  0x00002aaaad84ce8a in amqp_login_with_properties ()
   from
/import/home/yury.alioshinov/linux_i686/5102/lib/librabbitmq_c.so.1.1.1
#7  0x00002aaaad626313 in RabbitMQ::RabbitMQConnectionImpl::connect
(this=0xfaf3a00, hostName=...,
    portNumber=<value optimized out>, userName=..., userPwd=...,
heartbeat=0, vhost=...)
    at rabbitmq_source/rabbitmq_connector/RabbitMQConnectionImpl.cpp:115
#8  0x00002aaaad60d900 in rabbitmq_ns::consumer_impl::execute
(this=0xb4579d8) at rabbitmq_source/rabbitmq_impl.cpp:823
#9  0x00002b25bdde1aa3 in threads::thread_proc (thr_ptr=0xb4579d8) at
has_common_source/source/cpp/threads.cpp:225
#10 0x0000003435c0673d in start_thread () from /lib64/libpthread.so.0
#11 0x00000034350d40cd in clone () from /lib64/libc.so.6

So, method login_with_properties is blocked, and there is some usecase where
program execute amqp_send_header in amqp_login_inner without errors but
can't get data from amqp_simple_wait_method.

Maybe rabbitmq_c needs nonblock login method?




--
View this message in context: http://rabbitmq.1065348.n5.nabble.com/rabbitmq-c-Infinite-wait-in-login-with-properties-function-tp32275.html
Sent from the RabbitMQ mailing list archive at Nabble.com.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: [rabbitmq_c] Infinite wait in login_with_properties function

Haster
Hi, Alan.

I saw this behavior only one time. But I didn't try to reproduce it.

I don't think, that network problems could happen frequently in that place, but
if they happen... there will be infinite loop
Reply | Threaded
Open this post in threaded view
|

Re: [rabbitmq_c] Infinite wait in login_with_properties function

alan.antonuk
To be clear rabbitmq-c shouldn't infinite loop inside of amqp_login*() when network issues occur. Most likely what will happen is that the recv() call will have a long timeout before returning failure, which will be bubbled up and returned as an error from the amqp_login*() function.

-Alan


On Wed, Dec 25, 2013 at 10:05 PM, Haster <[hidden email]> wrote:
Hi, Alan.

I saw this behavior only one time. But I didn't try to reproduce it.

I don't think, that network problems could happen frequently in that place,
but
if they happen... there will be infinite loop



--
View this message in context: http://rabbitmq.1065348.n5.nabble.com/rabbitmq-c-Infinite-wait-in-login-with-properties-function-tp32275p32306.html
Sent from the RabbitMQ mailing list archive at Nabble.com.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: [rabbitmq_c] Infinite wait in login_with_properties function

Haster
Alan,
But is it possible to limit such timeout or use nonblocked version of recv?
And can you add this functionality to rabbitmq_c library?
Reply | Threaded
Open this post in threaded view
|

Re: [rabbitmq_c] Infinite wait in login_with_properties function

alan.antonuk
rabbitmq-c does have limited support for doing non-blocking operations on sockets, I do not surface it in the amqp_login() functions currently, as the the implementation of such a thing is not straight-forward to get right in all cases. If there's enough interest I would consider adding this, though I really haven't seen a huge demand for such a thing.

-Alan




On Mon, Apr 28, 2014 at 3:20 AM, Haster <[hidden email]> wrote:
Alan,
But is it possible to limit such timeout or use nonblocked version of recv?
And can you add this functionality to rabbitmq_c library?



--
View this message in context: http://rabbitmq.1065348.n5.nabble.com/rabbitmq-c-Infinite-wait-in-login-with-properties-function-tp32275p35118.html
Sent from the RabbitMQ mailing list archive at Nabble.com.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: [rabbitmq_c] Infinite wait in login_with_properties function

Haster
Alan,
I've investigated this problem and found that problem occurs when socket get descriptor > 1024.
Because we are using select + fdset for wait data on socket, sometimes it works incorrect (I have heard that select can control only descriptors with number < 1024).

I see that you also use select in revc_with_timeout function.
Maybe it is better to use poll or epoll in Linux/Unix?
Reply | Threaded
Open this post in threaded view
|

Re: [rabbitmq_c] Infinite wait in login_with_properties function

alan.antonuk
Use of poll(2) instead of select(2) was added since v0.5.0. Try pulling down the master branch and see if that works for your use case.

-Alan

On Tue May 13 2014 at 6:34:10 AM, Haster <[hidden email]> wrote:
Alan,
I've investigated this problem and found that problem occurs when socket get
descriptor > 1024.
Because we are using select + fdset for wait data on socket, sometimes it
works incorrect (I have heard that select can control only descriptors with
number < 1024).

I see that you also use select in revc_with_timeout function.
Maybe it is better to use poll or epoll in Linux/Unix?



--
View this message in context: http://rabbitmq.1065348.n5.nabble.com/rabbitmq-c-Infinite-wait-in-login-with-properties-function-tp32275p35492.html
Sent from the RabbitMQ mailing list archive at Nabble.com.
_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

_______________________________________________
rabbitmq-discuss mailing list
[hidden email]
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Reply | Threaded
Open this post in threaded view
|

Re: [rabbitmq_c] Infinite wait in login_with_properties function

Haster
Alan, thanks for patch.
It works well