publish messages non-stop/heartbeat issue

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

publish messages non-stop/heartbeat issue

This post has NOT been accepted by the mailing list yet.
Hi, i'm wondering what is the best strategy to publish messages to rabbit exchange in regards to opening new connections/channels. My use-case, i need to download files from FTP and once they're downloaded i need to publish some data to exchange. Downloading should work all the time and approximately around 40K files per 24hours should be downloaded, i.e. 40K messages will be published. My flow is like this:

1. Open connection and channel
2. Get X files from FTP
3. Publish messages to rabbit
4. Connection sleep
5. Repeat 2-4

Do i open only one connection and use as long as it stays open? How about channels, only one or i open new channel for every new message that i publish?

I'm using 'pika' client. What is better connection adapter for this use case, blocking or select connection? Is there any example or best practice how to implement re-connection strategy?

I also saw some problems on Windows(on Unix it doesn't occur) when rabbit server due to heartbeat timeout is closing the connection. I wonder how client handle heartbeats sent from server when it has channel that is only used for publishing and why it wasn't able to 'intercept' the heartbeat. Am i missing something?