# Disappearing Data and Unassigned Shards
**URL:** https://discuss.elastic.co/t/disappearing-data-and-unassigned-shards/9892
**Category:** Elasticsearch
**Created:** [November 29, 2012, 5:28pm UTC](https://discuss.elastic.co/t/disappearing-data-and-unassigned-shards/9892 "2012-11-29T17:28:23Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@Dan\_Lecocq](https://discuss.elastic.co/u/Dan_Lecocq)
#### Post date: [November 29, 2012, 5:28pm UTC](https://discuss.elastic.co/t/disappearing-data-and-unassigned-shards/9892/1 "2012-11-29T17:28:23Z")
It's become a recurring problem that our ES cluster crashes, and then when
we bring it back up, shards are missing. We keep tweaking and tuning
parameters, but it's mostly grasping at straws and it continues to
happen. Currently we're running 0.19.9 on a 16-node cluster of m1.xlarge
instances, with RAID0 across the four ephemeral drives (giving each node
1.6TB storage). The heap size for each machine is about half (8GB) of the
available memory (15GB).
We're generally happy with the performance, except for, you know, keeping
it running and alive :-/ Nothing in the logs seems to be an indicator.
Originally we were having some memory issues, but it turns out the heap
size environment variable wasn't getting properly set and that has since
been fixed. We ran into issues about the system mlock limit being set too
low, but that has also since been fixed. What seems to usually happen is
that a node will be unable to ping the master node, and then suddenly a few
more nodes will bite the dust, too. Sometimes bouncing elasticsearch on the
troubled boxes suffices, and sometimes we have to bounce the entire cluster.
Once everything comes back, \_cluster/health reports that there are
unassigned shards, and they just sit there like that indefinitely. I've
been trawling through the data directory, hoping to find some clues. For
the shards that sit around unassigned, there's actually no corresponding
index files on /any/ machine:
```
# Comes back '0' for all of our machines for missing shards
pssh -i --host=fresh-search-{11..26} -l ec2-user 'find $ESDIR
```
-wholename _2012-11-28/7_ -type f | wc -l'
# Comes back as non-zero on each machine that has a copy of the shard
for OK shards
pssh -i --host=fresh-search-{11..26} -l ec2-user 'find $ESDIR
-wholename _2012-11-28/8_ -type f | wc -l'
Suffice it to say, this is both disconcerting and frustrating. Any ideas?
--
---
### Author:  [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)
#### Post date: [November 30, 2012, 3:13pm UTC](https://discuss.elastic.co/t/disappearing-data-and-unassigned-shards/9892/2 "2012-11-30T15:13:47Z")
Hello Dan,
What's the number of shards and replicas?
And what's your configuration regarding recovery? I'm thinking about the
ones listed here:
> **[Elastic — The Search AI Company](https://www.elastic.co)**
>
> Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ...
Also, in the logs do you see something about dangling indices that will be
deleted?
## Best regards, Radu
[http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene
On Thu, Nov 29, 2012 at 7:28 PM, Dan Lecocq [dan@seomoz.org](mailto:dan@seomoz.org) wrote:
> It's become a recurring problem that our ES cluster crashes, and then when
> we bring it back up, shards are missing. We keep tweaking and tuning
> parameters, but it's mostly grasping at straws and it continues to
> happen. Currently we're running 0.19.9 on a 16-node cluster of m1.xlarge
> instances, with RAID0 across the four ephemeral drives (giving each node
> 1.6TB storage). The heap size for each machine is about half (8GB) of the
> available memory (15GB).
>
> We're generally happy with the performance, except for, you know, keeping
> it running and alive :-/ Nothing in the logs seems to be an indicator.
> Originally we were having some memory issues, but it turns out the heap
> size environment variable wasn't getting properly set and that has since
> been fixed. We ran into issues about the system mlock limit being set too
> low, but that has also since been fixed. What seems to usually happen is
> that a node will be unable to ping the master node, and then suddenly a few
> more nodes will bite the dust, too. Sometimes bouncing elasticsearch on the
> troubled boxes suffices, and sometimes we have to bounce the entire cluster.
>
> Once everything comes back, \_cluster/health reports that there are
> unassigned shards, and they just sit there like that indefinitely. I've
> been trawling through the data directory, hoping to find some clues. For
> the shards that sit around unassigned, there's actually no corresponding
> index files on /any/ machine:
>
> ```
> # Comes back '0' for all of our machines for missing shards
> pssh -i --host=fresh-search-{11..26} -l ec2-user 'find $ESDIR
>
> ```
>
> -wholename _2012-11-28/7_ -type f | wc -l'
> # Comes back as non-zero on each machine that has a copy of the shard
> for OK shards
> pssh -i --host=fresh-search-{11..26} -l ec2-user 'find $ESDIR
> -wholename _2012-11-28/8_ -type f | wc -l'
>
> Suffice it to say, this is both disconcerting and frustrating. Any ideas?
>
> --
--
---
### Author:  [@Dan\_Lecocq](https://discuss.elastic.co/u/Dan_Lecocq)
#### Post date: [December 3, 2012, 10:55pm UTC](https://discuss.elastic.co/t/disappearing-data-and-unassigned-shards/9892/3 "2012-12-03T22:55:49Z")
Hi Radu,
Thanks for your response. We've mostly been using 12 shards with 2
replicas, and we're currently using the local gateway with a
recover\_after\_time of 2 minutes:
```
gateway.type: local
gateway.recover_after_time: 2m
```
And yes, I hadn't noticed it before, but I am seeing a few lines about
dangling indexes. In particular, that it 'exists on local file system, but
not in cluster metadata' and that it's scheduled to be deleted in two
hours. Why would something like that happen?
On Friday, November 30, 2012 7:13:47 AM UTC-8, Radu Gheorghe wrote:
> Hello Dan,
>
> What's the number of shards and replicas?
>
> And what's your configuration regarding recovery? I'm thinking about the
> ones listed here:
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/modules/gateway/)
>
> Also, in the logs do you see something about dangling indices that will be
> deleted?
>
> ## Best regards, Radu
>
> [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene
>
> On Thu, Nov 29, 2012 at 7:28 PM, Dan Lecocq \<[d...@seomoz.org](mailto:d...@seomoz.org) \
>
> > wrote:
>
> > It's become a recurring problem that our ES cluster crashes, and then
> > when we bring it back up, shards are missing. We keep tweaking and tuning
> > parameters, but it's mostly grasping at straws and it continues to
> > happen. Currently we're running 0.19.9 on a 16-node cluster of m1.xlarge
> > instances, with RAID0 across the four ephemeral drives (giving each node
> > 1.6TB storage). The heap size for each machine is about half (8GB) of the
> > available memory (15GB).
> >
> > We're generally happy with the performance, except for, you know, keeping
> > it running and alive :-/ Nothing in the logs seems to be an indicator.
> > Originally we were having some memory issues, but it turns out the heap
> > size environment variable wasn't getting properly set and that has since
> > been fixed. We ran into issues about the system mlock limit being set too
> > low, but that has also since been fixed. What seems to usually happen is
> > that a node will be unable to ping the master node, and then suddenly a few
> > more nodes will bite the dust, too. Sometimes bouncing elasticsearch on the
> > troubled boxes suffices, and sometimes we have to bounce the entire cluster.
> >
> > Once everything comes back, \_cluster/health reports that there are
> > unassigned shards, and they just sit there like that indefinitely. I've
> > been trawling through the data directory, hoping to find some clues. For
> > the shards that sit around unassigned, there's actually no corresponding
> > index files on /any/ machine:
> >
> > ```
> > # Comes back '0' for all of our machines for missing shards
> > pssh -i --host=fresh-search-{11..26} -l ec2-user 'find $ESDIR
> >
> > ```
> >
> > -wholename _2012-11-28/7_ -type f | wc -l'
> > # Comes back as non-zero on each machine that has a copy of the shard
> > for OK shards
> > pssh -i --host=fresh-search-{11..26} -l ec2-user 'find $ESDIR
> > -wholename _2012-11-28/8_ -type f | wc -l'
> >
> > Suffice it to say, this is both disconcerting and frustrating. Any ideas?
> >
> > --
--
---
### Author:  [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)
#### Post date: [December 4, 2012, 1:00pm UTC](https://discuss.elastic.co/t/disappearing-data-and-unassigned-shards/9892/4 "2012-12-04T13:00:37Z")
Hello Dan,
I would assume that in your case, on a full cluster restart, recovery
begins after 2 minutes, since the gateway.recover\_after\_nodes setting
defaults to 1. At that point, I would assume many nodes might not be
started, so the cluster is not aware of the existence of some indices.
Afterwards, when other nodes join, the missing indices are reported as
dangling. You can avoid that by setting a high value of
gateway.recover\_after\_nodes, to make sure you have all indices in that
number of nodes before you begin recovery. You might also want to increase
gateway.recover\_after\_time, to give time for other nodes to start. And
gateway.expected\_nodes should be the total number of nodes in your cluster,
so 16.
Since 0.19.8, dangling indices should be automatically imported by default
(gateway.local.auto\_import\_dangled: yes), but obviously in your case it
isn't applied. So I'd suggest you explicitly specify that in your config,
to prevent such indices from being deleted in future.
Anyway, I would tweak the recovery settings and see if the problem still
appears.
## Best regards, Radu
[http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene
On Tue, Dec 4, 2012 at 12:55 AM, Dan Lecocq [dan@seomoz.org](mailto:dan@seomoz.org) wrote:
> Hi Radu,
>
> Thanks for your response. We've mostly been using 12 shards with 2
> replicas, and we're currently using the local gateway with a
> recover\_after\_time of 2 minutes:
>
> ```
> gateway.type: local
> gateway.recover_after_time: 2m
>
> ```
>
> And yes, I hadn't noticed it before, but I am seeing a few lines about
> dangling indexes. In particular, that it 'exists on local file system, but
> not in cluster metadata' and that it's scheduled to be deleted in two
> hours. Why would something like that happen?
>
> On Friday, November 30, 2012 7:13:47 AM UTC-8, Radu Gheorghe wrote:
>
> > Hello Dan,
> >
> > What's the number of shards and replicas?
> >
> > And what's your configuration regarding recovery? I'm thinking about the
> > ones listed here:
> > [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/ **guide/reference/modules/** gateway/)[http://www.elasticsearch.org/guide/reference/modules/gateway/](http://www.elasticsearch.org/guide/reference/modules/gateway/)
> >
> > Also, in the logs do you see something about dangling indices that will
> > be deleted?
> >
> > ## Best regards, Radu
> >
> > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene
> >
> > On Thu, Nov 29, 2012 at 7:28 PM, Dan Lecocq [d...@seomoz.org](mailto:d...@seomoz.org) wrote:
> >
> > > It's become a recurring problem that our ES cluster crashes, and then
> > > when we bring it back up, shards are missing. We keep tweaking and tuning
> > > parameters, but it's mostly grasping at straws and it continues to
> > > happen. Currently we're running 0.19.9 on a 16-node cluster of m1.xlarge
> > > instances, with RAID0 across the four ephemeral drives (giving each node
> > > 1.6TB storage). The heap size for each machine is about half (8GB) of the
> > > available memory (15GB).
> > >
> > > We're generally happy with the performance, except for, you know,
> > > keeping it running and alive :-/ Nothing in the logs seems to be an
> > > indicator. Originally we were having some memory issues, but it turns out
> > > the heap size environment variable wasn't getting properly set and that has
> > > since been fixed. We ran into issues about the system mlock limit being set
> > > too low, but that has also since been fixed. What seems to usually happen
> > > is that a node will be unable to ping the master node, and then suddenly a
> > > few more nodes will bite the dust, too. Sometimes bouncing elasticsearch on
> > > the troubled boxes suffices, and sometimes we have to bounce the entire
> > > cluster.
> > >
> > > Once everything comes back, \_cluster/health reports that there are
> > > unassigned shards, and they just sit there like that indefinitely. I've
> > > been trawling through the data directory, hoping to find some clues. For
> > > the shards that sit around unassigned, there's actually no corresponding
> > > index files on /any/ machine:
> > >
> > > ```
> > > # Comes back '0' for all of our machines for missing shards
> > > pssh -i --host=fresh-search-{11..26} -l ec2-user 'find $ESDIR
> > >
> > > ```
> > >
> > > -wholename _2012-11-28/7_ -type f | wc -l'
> > > # Comes back as non-zero on each machine that has a copy of the
> > > shard for OK shards
> > > pssh -i --host=fresh-search-{11..26} -l ec2-user 'find $ESDIR
> > > -wholename _2012-11-28/8_ -type f | wc -l'
> > >
> > > Suffice it to say, this is both disconcerting and frustrating. Any ideas?
> > >
> > > --
> >
> > --
--
---
### Author:  [@Dan\_Lecocq](https://discuss.elastic.co/u/Dan_Lecocq)
#### Post date: [December 4, 2012, 9:32pm UTC](https://discuss.elastic.co/t/disappearing-data-and-unassigned-shards/9892/5 "2012-12-04T21:32:32Z")
Thanks,
I changed all those settings, and I'm keeping my fingers crossed!
On Tuesday, December 4, 2012 5:00:37 AM UTC-8, Radu Gheorghe wrote:
> Hello Dan,
>
> I would assume that in your case, on a full cluster restart, recovery
> begins after 2 minutes, since the gateway.recover\_after\_nodes setting
> defaults to 1. At that point, I would assume many nodes might not be
> started, so the cluster is not aware of the existence of some indices.
>
> Afterwards, when other nodes join, the missing indices are reported as
> dangling. You can avoid that by setting a high value of
> gateway.recover\_after\_nodes, to make sure you have all indices in that
> number of nodes before you begin recovery. You might also want to increase
> gateway.recover\_after\_time, to give time for other nodes to start. And
> gateway.expected\_nodes should be the total number of nodes in your cluster,
> so 16.
>
> Since 0.19.8, dangling indices should be automatically imported by default
> (gateway.local.auto\_import\_dangled: yes), but obviously in your case it
> isn't applied. So I'd suggest you explicitly specify that in your config,
> to prevent such indices from being deleted in future.
>
> Anyway, I would tweak the recovery settings and see if the problem still
> appears.
>
> ## Best regards, Radu
>
> [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene
>
> On Tue, Dec 4, 2012 at 12:55 AM, Dan Lecocq \<[d...@seomoz.org](mailto:d...@seomoz.org) \
>
> > wrote:
>
> > Hi Radu,
> >
> > Thanks for your response. We've mostly been using 12 shards with 2
> > replicas, and we're currently using the local gateway with a
> > recover\_after\_time of 2 minutes:
> >
> > ```
> > gateway.type: local
> > gateway.recover_after_time: 2m
> >
> > ```
> >
> > And yes, I hadn't noticed it before, but I am seeing a few lines about
> > dangling indexes. In particular, that it 'exists on local file system, but
> > not in cluster metadata' and that it's scheduled to be deleted in two
> > hours. Why would something like that happen?
> >
> > On Friday, November 30, 2012 7:13:47 AM UTC-8, Radu Gheorghe wrote:
> >
> > > Hello Dan,
> > >
> > > What's the number of shards and replicas?
> > >
> > > And what's your configuration regarding recovery? I'm thinking about the
> > > ones listed here:
> > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/ **guide/reference/modules/** gateway/)[http://www.elasticsearch.org/guide/reference/modules/gateway/](http://www.elasticsearch.org/guide/reference/modules/gateway/)
> > >
> > > Also, in the logs do you see something about dangling indices that will
> > > be deleted?
> > >
> > > ## Best regards, Radu
> > >
> > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene
> > >
> > > On Thu, Nov 29, 2012 at 7:28 PM, Dan Lecocq [d...@seomoz.org](mailto:d...@seomoz.org) wrote:
> > >
> > > > It's become a recurring problem that our ES cluster crashes, and then
> > > > when we bring it back up, shards are missing. We keep tweaking and tuning
> > > > parameters, but it's mostly grasping at straws and it continues to
> > > > happen. Currently we're running 0.19.9 on a 16-node cluster of m1.xlarge
> > > > instances, with RAID0 across the four ephemeral drives (giving each node
> > > > 1.6TB storage). The heap size for each machine is about half (8GB) of the
> > > > available memory (15GB).
> > > >
> > > > We're generally happy with the performance, except for, you know,
> > > > keeping it running and alive :-/ Nothing in the logs seems to be an
> > > > indicator. Originally we were having some memory issues, but it turns out
> > > > the heap size environment variable wasn't getting properly set and that has
> > > > since been fixed. We ran into issues about the system mlock limit being set
> > > > too low, but that has also since been fixed. What seems to usually happen
> > > > is that a node will be unable to ping the master node, and then suddenly a
> > > > few more nodes will bite the dust, too. Sometimes bouncing elasticsearch on
> > > > the troubled boxes suffices, and sometimes we have to bounce the entire
> > > > cluster.
> > > >
> > > > Once everything comes back, \_cluster/health reports that there are
> > > > unassigned shards, and they just sit there like that indefinitely. I've
> > > > been trawling through the data directory, hoping to find some clues. For
> > > > the shards that sit around unassigned, there's actually no corresponding
> > > > index files on /any/ machine:
> > > >
> > > > ```
> > > > # Comes back '0' for all of our machines for missing shards
> > > > pssh -i --host=fresh-search-{11..26} -l ec2-user 'find $ESDIR
> > > >
> > > > ```
> > > >
> > > > -wholename _2012-11-28/7_ -type f | wc -l'
> > > > # Comes back as non-zero on each machine that has a copy of the
> > > > shard for OK shards
> > > > pssh -i --host=fresh-search-{11..26} -l ec2-user 'find $ESDIR
> > > > -wholename _2012-11-28/8_ -type f | wc -l'
> > > >
> > > > Suffice it to say, this is both disconcerting and frustrating. Any
> > > > ideas?
> > > >
> > > > --
> > >
> > > --
--
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 3:01am UTC](https://discuss.elastic.co/t/disappearing-data-and-unassigned-shards/9892/6 "2017-07-06T03:01:30Z")