ES Keeps Falling Over

We've really been enjoying elasticsearch, where we were having some
unpleasantness with Sphinx. At any rate, we're continuously inserting
batches of documents through a job queue, using the python bindings
(https://github.com/aparo/pyes), and after a couple of hours, the cluster
reliably falls apart. We've got 6 m1.larges running the thing with 300
shards (we want to have room to grow if need be).

I've noticed that it periodically complains about the number of open file
descriptors, despite it being set to 64k on each machine. From the logs, it
seems that eventually shard recovery fails, and then it goes downhill from
there. Restarting the whole cluster brings it back into the green, but I
have a feeling that's not meant to be a reguarly-run ops issue.

When inserting documents, we're using the bulk API, and then waiting for
green status. Rather, that's what we're requesting of the python API. I'm
not entirely convinced that there's not something going on in that library,
but I've not been able to track anything obvious down in there.

Gist with mapping, health, machine
configuration: https://gist.github.com/0c175c2b0ed3d84138b6

Is this symptomatic of a common problem? Or a known problem? I imagine it's
something I've not set up correctly :-/

Can you use the nodes info API to check if the file descriptor settings has
actually been applied to the process? Node stats will show you the current
open file handles. You can use bigdesk to see that visually.

Also, why 300 shards? Can you share a bit more your model? See some info
here:
Redirecting to Google Groups,
and here:
https://speakerdeck.com/u/kimchy/p/elasticsearch-big-data-search-analytics.

On Wed, Jun 6, 2012 at 6:11 PM, Dan Lecocq dan@seomoz.org wrote:

We've really been enjoying elasticsearch, where we were having some
unpleasantness with Sphinx. At any rate, we're continuously inserting
batches of documents through a job queue, using the python bindings (
GitHub - aparo/pyes: Python connector for ElasticSearch - the pythonic way to use ElasticSearch), and after a couple of hours, the cluster
reliably falls apart. We've got 6 m1.larges running the thing with 300
shards (we want to have room to grow if need be).

I've noticed that it periodically complains about the number of open file
descriptors, despite it being set to 64k on each machine. From the logs, it
seems that eventually shard recovery fails, and then it goes downhill from
there. Restarting the whole cluster brings it back into the green, but I
have a feeling that's not meant to be a reguarly-run ops issue.

When inserting documents, we're using the bulk API, and then waiting for
green status. Rather, that's what we're requesting of the python API. I'm
not entirely convinced that there's not something going on in that library,
but I've not been able to track anything obvious down in there.

Gist with mapping, health, machine configuration:
https://gist.github.com/0c175c2b0ed3d84138b6

Is this symptomatic of a common problem? Or a known problem? I imagine
it's something I've not set up correctly :-/

Yes, I have such experience with the max open file. You'd better check the
Java process's max open file by:
car /proc/pidofjava/limit|grep file
in Linux or something like in other OS.

Alan

Sent from my Windows Phone

From: Shay Banon
Sent: 6/11/2012 5:29 AM
To: elasticsearch@googlegroups.com
Subject: Re: ES Keeps Falling Over

Can you use the nodes info API to check if the file descriptor settings has
actually been applied to the process? Node stats will show you the current
open file handles. You can use bigdesk to see that visually.

Also, why 300 shards? Can you share a bit more your model? See some info
here:
Redirecting to Google Groups,
and here:
https://speakerdeck.com/u/kimchy/p/elasticsearch-big-data-search-analytics.

On Wed, Jun 6, 2012 at 6:11 PM, Dan Lecocq dan@seomoz.org wrote:

We've really been enjoying elasticsearch, where we were having some
unpleasantness with Sphinx. At any rate, we're continuously inserting
batches of documents through a job queue, using the python bindings (
GitHub - aparo/pyes: Python connector for ElasticSearch - the pythonic way to use ElasticSearch), and after a couple of hours, the cluster
reliably falls apart. We've got 6 m1.larges running the thing with 300
shards (we want to have room to grow if need be).

I've noticed that it periodically complains about the number of open file
descriptors, despite it being set to 64k on each machine. From the logs, it
seems that eventually shard recovery fails, and then it goes downhill from
there. Restarting the whole cluster brings it back into the green, but I
have a feeling that's not meant to be a reguarly-run ops issue.

When inserting documents, we're using the bulk API, and then waiting for
green status. Rather, that's what we're requesting of the python API. I'm
not entirely convinced that there's not something going on in that library,
but I've not been able to track anything obvious down in there.

Gist with mapping, health, machine configuration:
https://gist.github.com/0c175c2b0ed3d84138b6

Is this symptomatic of a common problem? Or a known problem? I imagine
it's something I've not set up correctly :-/

Thanks for replies;

@Shay - I've seen my open fd count spike at 64k, and though I've only just
started checking the _nodes/stats endpdoint, since ES is the only thing
running on those boxes, I'm relatively certain those fds were owned by ES.
WRT the shards, I initially didn't have much of a sense of how many was too
many, and wanted to have room to grow, which bring me to our data flow.
We're hoping to insert on the order of maybe a few million documents (not
saving content) per day, and keeping the last 90 days of history in the
index (using _ttl).

@LiMac - I have verified that my 64k limit has been applied to these. It's
something I hit earlier, when I had it set to 32k, and had to up it (to no
avail, it would seem).

  • D

In light of learning about aliasing, and since our data is very conducive
to time-based slicing, I'm now planning on using many fewer shards over
more indexes, and then aliasing them as an administrative task.

It seems to me that this will also make scaling up easier, so that when I
need to add more nodes, I can create a new index with sufficiently many
shards, set that as the write index, and then introduce the new nodes? The
added flexibility will be nice.

On Sunday, June 10, 2012 4:08:27 PM UTC-7, Dan Lecocq wrote:

Thanks for replies;

@Shay - I've seen my open fd count spike at 64k, and though I've only just
started checking the _nodes/stats endpdoint, since ES is the only thing
running on those boxes, I'm relatively certain those fds were owned by ES.
WRT the shards, I initially didn't have much of a sense of how many was too
many, and wanted to have room to grow, which bring me to our data flow.
We're hoping to insert on the order of maybe a few million documents (not
saving content) per day, and keeping the last 90 days of history in the
index (using _ttl).

@LiMac - I have verified that my 64k limit has been applied to these. It's
something I hit earlier, when I had it set to 32k, and had to up it (to no
avail, it would seem).

  • D