We are currently running version 2.4.0 in our production cluster. After node restarts I've noticed we have around 36 shards which seem to be stuck initializing.
We have been having problems with nodes crashing due to OOM errors. Overtime (and many node restarts later: -- This is separate issue I'm trying to address) we have ended up with a bunch of initializing shards which never seem to finish initializing. I'll update this with more info/logs when they become available.
I was wondering if there is anything I can do to address this issue other than delete the index / restart the node?
At this point the cluster is red. In this state, can this have a slowing impact on overall performance? Could this slow down indexing b/c the cluster is also busying trying to initialize shards?
Is there a way to determine if the shard has become corrupted and just won't ever initialize?
Running the cat/_recovery api tells me the following:
index shard time type stage source_host target_host repository snapshot files files_percent bytes bytes_percent total_files total_bytes translog translog_percent total_translog
xyz-index 0 596067535 store init 10.10.10.10 10.10.10.10 n/a n/a 0 0.0% 0 0.0% 0 0 0 -1.0% -1
Prior to this index/shard not initializing I verified it contained 57 docs and was about 23KB so I'm thinking it should have initialized pretty quickly.
I think you have too many shards, and possibly also too many indices. The fact that you do not seem to use any replica shards probably does not help either. Would recommend you read this blog post.
Please don't add replicas on this cluster though. You are already way over sharded and you need to reduce that first before you start adding replicas in.
We have approximately ~17TB of data in our cluster. I was looking at a 6 month sample of data index breakdowns for our cluster. I'm seeing 90% of the indexes are pretty small in size. They can range anywhere from 50KB up to 48MB in size. We also have 3 groups of indices which make up the most of the data. They can range from 11-14GB in size. For the smaller indices, they are time based broken up by hour. The group of larger indexes are also time based but broken up by days.
It sounds like recommendation is to decrease the number of shards in order to collapse the smaller indices into larger indices. Do you have a recommendation for doing that with version 2 of ES? Would we just need to reindex the data? Any thoughts are appreciated!
Hey thanks for all of the quick responses from everyone. Very much appreciated!. Ok, let me know if you want me to open a new topic since this a different question, but I was curious about something related to this cluster. Recently I updated the bootstrap.mlock setting from false to true for half of the nodes (10). After about 1-2 weeks later, I'm seeing the nodes (with that setting enabled) are starting to use some swap space as it's slowly climbing.
-- If the cluster is under a lot of stress due to too many indices/shards (as we described above), is it possible this setting will be ignored, or do you think there something else going on that would cause this setting to be ignored. If I understand correctly, enabling this setting will prevent the ES processes from swapping correct? I'm a little confused because I'm seeing something different. Again I realize our cluster is not in an ideal state. Thanks again.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
