# Elasticsearch Missing Data **URL:** \ **Category:** Elasticsearch\ **Created:** [January 8, 2014, 7:10pm UTC](https://discuss.elastic.co/t/elasticsearch-missing-data/15150 "2014-01-08T19:10:37Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Eric\_Luellen](https://avatars.discourse-cdn.com/v4/letter/e/ce73a5/32.png) [@Eric\_Luellen](https://discuss.elastic.co/u/Eric_Luellen)\ **Post date:** [January 8, 2014, 7:10pm UTC](https://discuss.elastic.co/t/elasticsearch-missing-data/15150/1 "2014-01-08T19:10:37Z")
Hello, I've had my elasticsearch instance running for about a week with no issues, but last night it stopped working. When I went to look in Kibana, it stops logging around 20:45 on 1/7/14. I then restarted the service on both both elasticsearch servers and it started logging again and back pulled some logs from 07:10 that morning, even though I restarted the service around 10:00. So my questions are: 1. Why did it stop working? I don't see any obvious errors. 2. When I restarted it, why didn't it go back and pull all of the data and not just some of it? I see that there are no unassigned shards. curl -XGET '[http://localhost:9200/\_cluster/health?pretty=true](http://localhost:9200/_cluster/health?pretty=true)' { "cluster\_name" : "my-elasticsearch", "status" : "green", "timed\_out" : false, "number\_of\_nodes" : 3, "number\_of\_data\_nodes" : 2, "active\_primary\_shards" : 40, "active\_shards" : 80, "relocating\_shards" : 0, "initializing\_shards" : 0, "unassigned\_shards" : 0 Are there any additional queries or logs I can look at to see what is going on? On a slight side note, when I restarted my 2nd elasticsearch server it isn't reading from the /etc/elasticsearch.yml file like it should. It isn't creating the node name correctly or putting the data files in the spot I have configured. I'm using CentOS and doing everything via /etc/init.d/elasticsearch on both servers and the elasticsearch1 server reads everything correctly but elasticsearch2 does not. Thanks for your help. Eric -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/fc191ee4-b312-4c52-89d9-de04c4309b65%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/fc191ee4-b312-4c52-89d9-de04c4309b65%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [January 9, 2014, 8:30am UTC](https://discuss.elastic.co/t/elasticsearch-missing-data/15150/2 "2014-01-09T08:30:40Z")
Hey, a couple of things: 1. Did you check the log files? Most likely in /var/log/elasticsearch if you use the packages. Is there anything suspicious at the time of your outage? Please check your master node as well, if you have one (not sure if it is a master or client node from the cluster health). 2. Why should elasticsearch pull your data? Any special configuration you didnt mention? Or what exactly do you mean here? 3. Happy to debug your issue with the init script. The elasticsearch.yml file should be in /etc/elasticsearch/ and not in /etc - anything manually moved around? Can you still reproduce it? --Alex On Wed, Jan 8, 2014 at 8:10 PM, Eric Luellen [eric.luellen@gmail.com](mailto:eric.luellen@gmail.com) wrote: > Hello, > > I've had my elasticsearch instance running for about a week with no > issues, but last night it stopped working. When I went to look in Kibana, > it stops logging around 20:45 on 1/7/14. I then restarted the service on > both both elasticsearch servers and it started logging again and back > pulled some logs from 07:10 that morning, even though I restarted the > service around 10:00. So my questions are: > > 1. Why did it stop working? I don't see any obvious errors. > 2. When I restarted it, why didn't it go back and pull all of the data and > not just some of it? I see that there are no unassigned shards. > > curl -XGET '[http://localhost:9200/\_cluster/health?pretty=true](http://localhost:9200/_cluster/health?pretty=true)' > { > "cluster\_name" : "my-elasticsearch", > "status" : "green", > "timed\_out" : false, > "number\_of\_nodes" : 3, > "number\_of\_data\_nodes" : 2, > "active\_primary\_shards" : 40, > "active\_shards" : 80, > "relocating\_shards" : 0, > "initializing\_shards" : 0, > "unassigned\_shards" : 0 > > Are there any additional queries or logs I can look at to see what is > going on? > > On a slight side note, when I restarted my 2nd elasticsearch server it > isn't reading from the /etc/elasticsearch.yml file like it should. It isn't > creating the node name correctly or putting the data files in the spot I > have configured. I'm using CentOS and doing everything via > /etc/init.d/elasticsearch on both servers and the elasticsearch1 server > reads everything correctly but elasticsearch2 does not. > > Thanks for your help. > Eric > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/fc191ee4-b312-4c52-89d9-de04c4309b65%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/fc191ee4-b312-4c52-89d9-de04c4309b65%40googlegroups.com) > . > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM8EOWdC5esVkfZ5hogocQkgreJBQUbF2zE7s-gGCt4NdQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM8EOWdC5esVkfZ5hogocQkgreJBQUbF2zE7s-gGCt4NdQ%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Eric\_Luellen](https://avatars.discourse-cdn.com/v4/letter/e/ce73a5/32.png) [@Eric\_Luellen](https://discuss.elastic.co/u/Eric_Luellen)\ **Post date:** [January 9, 2014, 6:01pm UTC](https://discuss.elastic.co/t/elasticsearch-missing-data/15150/3 "2014-01-09T18:01:24Z")
Alexander, 1. The only odd log entry was at 19:00 on 1/7/14, which was about 1 hr. before logs stopped. These logs are on the master and She-Hulk is the only other node. [2014-01-07 19:00:02,947][DEBUG][indices.recovery] [Elasticsearch Server1] [logstash-2014.01.08][0] recovery completed from [She-Hulk][\_MtrVsSmQIaM-BErhEtg9w][inet[/10.1.11.111:9300]], took[333ms] phase1: recovered\_files [1] with total\_size of [71b], took [68ms], throttling\_wait [0s] : reusing\_files [0] with total\_size of [0b] phase2: start took [13ms] : recovered [17] transaction log operations, took [12ms] phase3: recovered [0] transaction log operations, took [164ms] [2014-01-07 19:00:03,375][DEBUG][indices.recovery] [Elasticsearch Server1] [logstash-2014.01.08][2] recovery completed from [She-Hulk][\_MtrVsSmQIaM-BErhEtg9w][inet[/10.1.11.111:9300]], took[502ms] phase1: recovered\_files [1] with total\_size of [71b], took [30ms], throttling\_wait [0s] : reusing\_files [0] with total\_size of [0b] phase2: start took [6ms] : recovered [6] transaction log operations, took [38ms] phase3: recovered [13] transaction log operations, took [20ms] [2014-01-07 19:00:06,898][INFO][cluster.metadata] [Elasticsearch Server1] [logstash-2014.01.08] update\_mapping [logs] (dynamic) Also, on She-Hulk I got an error stating that the master\_left at 20:52 because it wasn't pingable, but not sure why. 2.I am not sure. I was thinking that the shard should still be there but just unassigned and once it came back up, it'd start processing it. 3. On both my master and my 2ndary, the config is in /etc/elasticsearch/elasticsearch.yml and it is ran by /etc/init.d/elasticsearch. On the master, it works fine and make the correct node name, cluster name, data directory, etc. It is an identical setup on the 2ndary but it only grabs the cluster name. Everything else defaults to some other location.On the secondary, the only data location is in /var/lib/elasticsearch/node-name. In the config I tell it to go to /etc/elasticsearch/data. On the master it is in the correct location of /etc/elasticsearch/data. So overall, I guess the first issue was something weird happened to my server and not much I can do about that. I'm more interested in the 3rd question now since I still don't know why it's not reading that full config file but obviously part of it since it's part of my cluster. On Thursday, January 9, 2014 3:30:40 AM UTC-5, Alexander Reelsen wrote: > Hey, > > a couple of things: > > 1. Did you check the log files? Most likely in /var/log/elasticsearch if > you use the packages. Is there anything suspicious at the time of your > outage? Please check your master node as well, if you have one (not sure if > it is a master or client node from the cluster health). > 2. Why should elasticsearch pull your data? Any special configuration you > didnt mention? Or what exactly do you mean here? > 3. Happy to debug your issue with the init script. The elasticsearch.yml > file should be in /etc/elasticsearch/ and not in /etc - anything manually > moved around? Can you still reproduce it? > > --Alex > > On Wed, Jan 8, 2014 at 8:10 PM, Eric Luellen \<[eric.l...@gmail.com](mailto:eric.l...@gmail.com)\ > > > wrote: > > > Hello, > > > > I've had my elasticsearch instance running for about a week with no > > issues, but last night it stopped working. When I went to look in Kibana, > > it stops logging around 20:45 on 1/7/14. I then restarted the service on > > both both elasticsearch servers and it started logging again and back > > pulled some logs from 07:10 that morning, even though I restarted the > > service around 10:00. So my questions are: > > > > 1. Why did it stop working? I don't see any obvious errors. > > 2. When I restarted it, why didn't it go back and pull all of the data > > and not just some of it? I see that there are no unassigned shards. > > > > curl -XGET '[http://localhost:9200/\_cluster/health?pretty=true](http://localhost:9200/_cluster/health?pretty=true)' > > { > > "cluster\_name" : "my-elasticsearch", > > "status" : "green", > > "timed\_out" : false, > > "number\_of\_nodes" : 3, > > "number\_of\_data\_nodes" : 2, > > "active\_primary\_shards" : 40, > > "active\_shards" : 80, > > "relocating\_shards" : 0, > > "initializing\_shards" : 0, > > "unassigned\_shards" : 0 > > > > Are there any additional queries or logs I can look at to see what is > > going on? > > > > On a slight side note, when I restarted my 2nd elasticsearch server it > > isn't reading from the /etc/elasticsearch.yml file like it should. It isn't > > creating the node name correctly or putting the data files in the spot I > > have configured. I'm using CentOS and doing everything via > > /etc/init.d/elasticsearch on both servers and the elasticsearch1 server > > reads everything correctly but elasticsearch2 does not. > > > > Thanks for your help. > > Eric > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/fc191ee4-b312-4c52-89d9-de04c4309b65%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/fc191ee4-b312-4c52-89d9-de04c4309b65%40googlegroups.com) > > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/d7a0967b-1e86-4b95-a28f-d703362c992a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d7a0967b-1e86-4b95-a28f-d703362c992a%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [January 14, 2014, 11:22am UTC](https://discuss.elastic.co/t/elasticsearch-missing-data/15150/4 "2014-01-14T11:22:49Z")
Hey, regarding the config file... wondering if your naming or your indentation maybe is wrong somewhere? Can you copy the config files and make sure their structure is the same? --Alex On Thu, Jan 9, 2014 at 7:01 PM, Eric Luellen [eric.luellen@gmail.com](mailto:eric.luellen@gmail.com) wrote: > Alexander, > > 1. The only odd log entry was at 19:00 on 1/7/14, which was about 1 hr. > before logs stopped. These logs are on the master and She-Hulk is the only > other node. > > [2014-01-07 19:00:02,947][DEBUG][indices.recovery] [Elasticsearch > Server1] [logstash-2014.01.08][0] recovery completed from > [She-Hulk][\_MtrVsSmQIaM-BErhEtg9w][inet[/10.1.11.111:9300]], took[333ms] > phase1: recovered\_files [1] with total\_size of [71b], took [68ms], > throttling\_wait [0s] > : reusing\_files [0] with total\_size of [0b] > phase2: start took [13ms] > : recovered [17] transaction log operations, took [12ms] > phase3: recovered [0] transaction log operations, took [164ms] > [2014-01-07 19:00:03,375][DEBUG][indices.recovery] [Elasticsearch > Server1] [logstash-2014.01.08][2] recovery completed from > [She-Hulk][\_MtrVsSmQIaM-BErhEtg9w][inet[/10.1.11.111:9300]], took[502ms] > phase1: recovered\_files [1] with total\_size of [71b], took [30ms], > throttling\_wait [0s] > : reusing\_files [0] with total\_size of [0b] > phase2: start took [6ms] > : recovered [6] transaction log operations, took [38ms] > phase3: recovered [13] transaction log operations, took [20ms] > [2014-01-07 19:00:06,898][INFO][cluster.metadata] [Elasticsearch > Server1] [logstash-2014.01.08] update\_mapping [logs] (dynamic) > > Also, on She-Hulk I got an error stating that the master\_left at 20:52 > because it wasn't pingable, but not sure why. > > 2.I am not sure. I was thinking that the shard should still be there but > just unassigned and once it came back up, it'd start processing it. > 3. On both my master and my 2ndary, the config is in > /etc/elasticsearch/elasticsearch.yml and it is ran by > /etc/init.d/elasticsearch. On the master, it works fine and make the > correct node name, cluster name, data directory, etc. It is an identical > setup on the 2ndary but it only grabs the cluster name. Everything else > defaults to some other location.On the secondary, the only data location is > in /var/lib/elasticsearch/node-name. In the config I tell it to go to > /etc/elasticsearch/data. On the master it is in the correct location of > /etc/elasticsearch/data. > > So overall, I guess the first issue was something weird happened to my > server and not much I can do about that. I'm more interested in the 3rd > question now since I still don't know why it's not reading that full config > file but obviously part of it since it's part of my cluster. > > On Thursday, January 9, 2014 3:30:40 AM UTC-5, Alexander Reelsen wrote: > > > Hey, > > > > a couple of things: > > > > 1. Did you check the log files? Most likely in /var/log/elasticsearch if > > you use the packages. Is there anything suspicious at the time of your > > outage? Please check your master node as well, if you have one (not sure if > > it is a master or client node from the cluster health). > > 2. Why should elasticsearch pull your data? Any special configuration you > > didnt mention? Or what exactly do you mean here? > > 3. Happy to debug your issue with the init script. The elasticsearch.yml > > file should be in /etc/elasticsearch/ and not in /etc - anything manually > > moved around? Can you still reproduce it? > > > > --Alex > > > > On Wed, Jan 8, 2014 at 8:10 PM, Eric Luellen [eric.l...@gmail.com](mailto:eric.l...@gmail.com) wrote: > > > > > Hello, > > > > > > I've had my elasticsearch instance running for about a week with no > > > issues, but last night it stopped working. When I went to look in Kibana, > > > it stops logging around 20:45 on 1/7/14. I then restarted the service on > > > both both elasticsearch servers and it started logging again and back > > > pulled some logs from 07:10 that morning, even though I restarted the > > > service around 10:00. So my questions are: > > > > > > 1. Why did it stop working? I don't see any obvious errors. > > > 2. When I restarted it, why didn't it go back and pull all of the data > > > and not just some of it? I see that there are no unassigned shards. > > > > > > curl -XGET '[http://localhost:9200/\_cluster/health?pretty=true](http://localhost:9200/_cluster/health?pretty=true)' > > > { > > > "cluster\_name" : "my-elasticsearch", > > > "status" : "green", > > > "timed\_out" : false, > > > "number\_of\_nodes" : 3, > > > "number\_of\_data\_nodes" : 2, > > > "active\_primary\_shards" : 40, > > > "active\_shards" : 80, > > > "relocating\_shards" : 0, > > > "initializing\_shards" : 0, > > > "unassigned\_shards" : 0 > > > > > > Are there any additional queries or logs I can look at to see what is > > > going on? > > > > > > On a slight side note, when I restarted my 2nd elasticsearch server it > > > isn't reading from the /etc/elasticsearch.yml file like it should. It isn't > > > creating the node name correctly or putting the data files in the spot I > > > have configured. I'm using CentOS and doing everything via > > > /etc/init.d/elasticsearch on both servers and the elasticsearch1 server > > > reads everything correctly but elasticsearch2 does not. > > > > > > Thanks for your help. > > > Eric > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/fc191ee4-b312-4c52-89d9-de04c4309b65% > > > [40googlegroups.com](http://40googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/d7a0967b-1e86-4b95-a28f-d703362c992a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d7a0967b-1e86-4b95-a28f-d703362c992a%40googlegroups.com) > > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM8KiWAfFuh88oQyBWcj8yK7g%2BBSOS6WqxTMZGdSQKWBcQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM8KiWAfFuh88oQyBWcj8yK7g%2BBSOS6WqxTMZGdSQKWBcQ%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:56am UTC](https://discuss.elastic.co/t/elasticsearch-missing-data/15150/5 "2017-07-06T01:56:57Z")