ElasticSearch and RabbitMQ: disconnected after inactivity

Elastic Stack Elasticsearch
1

Hi all,

I have a setup with two node RabbitMQ cluster and ElasticSearch server with
RabbitMQ river that acts as consumer of RabbitMQ queue.
I use it proof of concept purposes to test if this configuration works,
what is the load that can take etc.
I have the following problem: after some inactivity - no data is to ES over
RabbitMQ - I am not able to send data to ES again.
How I reproduce the situation: restart ES, the river connects to RabbitMQ
and it is ready to receive messages. I wait for say 60-90 minutes. When I
look at RabbitMQ Web UI it looks like there is connection between RabbitMQ
and ES - there is a registered consumer. When I try to put message in the
queue the river disconnects and I am not able to send messages to ES
anymore. Netstat shows that there is open connection between the river and
RabbitMQ. The only solution is to restart ES.

My questions are:
Is this a normal behavior?
Is there anything like timeout, heartbeat that can cause this behavior?
Do I have to simulate heartbeat to keep the river connected?
Can I reconnect the river without restarting ES server?

Are there any tools how to debug this?

Thanks in advance

best regards,
nickolay kolev

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

2013/7/8 Николай Колев lem00na@gmail.com

Is there anything like timeout, heartbeat that can cause this behavior?

Your OS or firewall seem to be closing TCP connections it considers idle.

Do I have to simulate heartbeat to keep the river connected?

RabbitMQ Java client lets you set a heartbeat value (the default value used
unless you tune it is 30 minutes).

I'm not sure if the ES river may be disabling that setting, intentionally
or not (it should not).

Not sure why it does not try to reconnect either.

MK

http://twitter.com/michaelklishin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

We don't disable heartbeat so I assume it's using default value.
We can expose it as an option though. Issue opened: https://github.com/elasticsearch/elasticsearch-river-rabbitmq/issues/29

@nickolay: which versions of elasticsearch and rabbitMQ river are you using?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 8 juil. 2013 à 13:04, Michael Klishin michael.s.klishin@gmail.com a écrit :

2013/7/8 Николай Колев lem00na@gmail.com
Is there anything like timeout, heartbeat that can cause this behavior?

Your OS or firewall seem to be closing TCP connections it considers idle.

Do I have to simulate heartbeat to keep the river connected?

RabbitMQ Java client lets you set a heartbeat value (the default value used unless you tune it is 30 minutes).

I'm not sure if the ES river may be disabling that setting, intentionally or not (it should not).

Not sure why it does not try to reconnect either.

MK

michaelklishin (Michael Klishin) · GitHub
http://twitter.com/michaelklishin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Hi Michael, David,

Michael: I have a conversation with the system admins and they confirmed
that there is a rule that closes inactive for one hour sessions.

David: I will create river with that option and will test; will write back
about the results
about versions:
RabbitMQ 2.8.6
ElasticSearch: 0.90.2

Thanks both for the support

best regards,
nickolay kolev

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

I was asking for rabbitmq river version. I assume it's 1.6.0, right?
Could you check that the latest version on master branch works as you are expecting? See commit: Expose heartbeat option · elastic/elasticsearch-river-rabbitmq@e3ed899 · GitHub

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 8 juil. 2013 à 14:27, Николай Колев lem00na@gmail.com a écrit :

Hi Michael, David,

Michael: I have a conversation with the system admins and they confirmed that there is a rule that closes inactive for one hour sessions.

David: I will create river with that option and will test; will write back about the results
about versions:
RabbitMQ 2.8.6
Elasticsearch: 0.90.2

Thanks both for the support

best regards,
nickolay kolev

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

Hi David,

Sorry for misunderstanding,
yes the version is 1.6.0
will test and come later with results

best regards,
nickolay kolev

08 юли 2013, понеделник, 15:48:29 UTC+3, David Pilato написа:

I was asking for rabbitmq river version. I assume it's 1.6.0, right?
Could you check that the latest version on master branch works as you are
expecting? See commit:
Expose heartbeat option · elastic/elasticsearch-river-rabbitmq@e3ed899 · GitHub

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr
| @scrutmydocs https://twitter.com/scrutmydocs

Le 8 juil. 2013 à 14:27, Николай Колев <lem...@gmail.com <javascript:>> a
écrit :

Hi Michael, David,

Michael: I have a conversation with the system admins and they confirmed
that there is a rule that closes inactive for one hour sessions.

David: I will create river with that option and will test; will write back
about the results
about versions:
RabbitMQ 2.8.6
Elasticsearch: 0.90.2

Thanks both for the support

best regards,
nickolay kolev

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

Hi David,

I can confirm that the patch works.
I have compiled and installed latest revision and I works as expected.
Thanks for the great work.

best regards,
nickolay kolev

08 юли 2013, понеделник, 15:48:29 UTC+3, David Pilato написа:

I was asking for rabbitmq river version. I assume it's 1.6.0, right?
Could you check that the latest version on master branch works as you are
expecting? See commit:
Expose heartbeat option · elastic/elasticsearch-river-rabbitmq@e3ed899 · GitHub

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr
| @scrutmydocs https://twitter.com/scrutmydocs

Le 8 juil. 2013 à 14:27, Николай Колев <lem...@gmail.com <javascript:>> a
écrit :

Hi Michael, David,

Michael: I have a conversation with the system admins and they confirmed
that there is a rule that closes inactive for one hour sessions.

David: I will create river with that option and will test; will write back
about the results
about versions:
RabbitMQ 2.8.6
Elasticsearch: 0.90.2

Thanks both for the support

best regards,
nickolay kolev

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.