Elasticsearch crash

Elastic Stack Elasticsearch
1

So it appears my Elasticsearch node crashed due to lucene memory issues. I have increased the memory and restarted Elasticsearch, but my logs are no longer showing in Kibana though they are in Elasticsearch.

It looks like the old logs are in /var/lib/elasticsearch/nodes/0/... while the new logs are in .../nodes/1/... is there a way to fix this or have kibana look in both node areas for data?

Thanks

2

It means that you probably started a new node on the machine while the other node was still running (or something made elasticsearch think that it was still running).

Are you using an old version of elasticsearch by any chance?

3

I am using 2.4, boss's orders lol I went into kibana and seen an error stating it couldn't connect to elasticsearch or something like that. So I checked on the elasticsearch service and seen that it was running, so I attempted to restart it. It seemed to hang during the restart so I stopped the restart process and checked out the logs. That's when I found out the lucene memory issue. I force stopped elasticsearch and restarted it, but then all the old logs were not visible any longer, but were still under the /nodes/0/ directory while the new ones went to /nodes/1/.

4

It went there because when you start 2 nodes on the same machine each node needs to write in his own directory.

Version 5+ prevents that from happening.

5

Ah. So I could fix this by just moving the stuff in /node/0/ to /node/1/ correct or how should I go about seeing all the data again?

6

If you do that (remove data 0 and rename data 1 as data 0) you will get back data that has been indexed recently but you will miss all the data that lived in data 0.

You can't merge both dirs.

I'd recommend reindexing what you indexed in data 1.

7

So you are saying my only option is to reindex the data from 0 to 1 if I want to keep my old data, correct? What about all the data in Kibana that is also missing like the dashboards and such? Is there a way to get that over too?

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.