Dropped HTTP Connections when Indexing

Elastic Stack Elasticsearch
1

I'm trying to scale my indexing for the first time, and I'm running into
connections problems. I reach a scale where cURL connections from my
indexers start getting cURL7 errors ( connect failed ). It looks like ES
just stops accepting all HTTP connections for a period of time. I cannot
find the root cause.

I'm running on an Amazon C3.4XL. The processors are not maxed, memory is
not maxed, IO is not showing issues. I'm not seeing problems in the ES
log, but I'm not sure I have logging fully enabled. I've tried increasing
the thread_pool for the indexer, and that doesn't help. I'm not seeing any
rejected connections there. I'm at a loss.

The closest I can get is a guess using data from Bigdesk. When the number
HTTP channels starts exceeding the number of transport channels, I start to
see the problem emerge. I have no idea if this is related, but it's the
only metric I've traced that seems correlated.

Thoughts?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0fd75b96-3c00-47f1-9a59-3c9707f38734%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Hey,

are you using HTTP keep alive connections? If not consider switching to
them, as reopening a new TCP connection not only results in high latencies
but also removes file handle resources from the elasticsearch process (the
number of open files). if your client/language does not support this, you
should either use nginx or at least try to use bulk operations in order to
create less TCP connections, however this is just a workaround.

--Alex

On Sat, Jul 19, 2014 at 5:23 AM, Brian Jones tbrianjones@gmail.com wrote:

I'm trying to scale my indexing for the first time, and I'm running into
connections problems. I reach a scale where cURL connections from my
indexers start getting cURL7 errors ( connect failed ). It looks like ES
just stops accepting all HTTP connections for a period of time. I cannot
find the root cause.

I'm running on an Amazon C3.4XL. The processors are not maxed, memory is
not maxed, IO is not showing issues. I'm not seeing problems in the ES
log, but I'm not sure I have logging fully enabled. I've tried increasing
the thread_pool for the indexer, and that doesn't help. I'm not seeing any
rejected connections there. I'm at a loss.

The closest I can get is a guess using data from Bigdesk. When the number
HTTP channels starts exceeding the number of transport channels, I start to
see the problem emerge. I have no idea if this is related, but it's the
only metric I've traced that seems correlated.

Thoughts?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0fd75b96-3c00-47f1-9a59-3c9707f38734%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/0fd75b96-3c00-47f1-9a59-3c9707f38734%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-QZCWND%2B0FuYNEZdjK_5izSaZUMFmB_TyBTTuFSyZB3g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.