# Elasticsearch crashed on development machine, trying to recover

**URL:** <https://discuss.elastic.co/t/elasticsearch-crashed-on-development-machine-trying-to-recover/160548>\
**Category:** Elasticsearch\
**Created:** [December 12, 2018, 1:20pm UTC](https://discuss.elastic.co/t/elasticsearch-crashed-on-development-machine-trying-to-recover/160548 "2018-12-12T13:20:21Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![fribse](https://avatars.discourse-cdn.com/v4/letter/f/c67d28/32.png) [@fribse](https://discuss.elastic.co/u/fribse)\
**Post date:** [December 12, 2018, 1:20pm UTC](https://discuss.elastic.co/t/elasticsearch-crashed-on-development-machine-trying-to-recover/160548/1 "2018-12-12T13:20:21Z")

</div>

We have a 3 node elasticsearch/kibana/logstash set up with docker-compose, each node has 8 GB for the Java.  
I tried working with some firewall rules to secure it a bit, but found out that firewalld (centos) messes up the internal docker firewall rules.  
I didn't find the problem until a bit later, and at that point it was crashing.  
So now I'm trying to start the 3 nodes again, I've stopped logstash as to give the elasticsearch a bit of peace to make things align again.

I'm seeing these messages in the logfile:  
elasticsearch2 | [2018-12-12T13:14:07,288][WARN][o.e.i.c.IndicesClusterStateService] [es02] [[filebeat-6.5.1-2016.05.14][4]] marking and sending shard failed due to [failed recovery]  
elasticsearch2 | org.elasticsearch.indices.recovery.RecoveryFailedException: [filebeat-6.5.1-2016.05.14][4]: Recovery failed from {es03}{utNZZop8SRuCuX\_KZffjuw}{6cOtisSfRYiB42LyNWUudQ}{172.18.0.5}{172.18.0.5:9300}{ml.machine\_memory=37779542016, ml.max\_open\_jobs=20, xpack.installed=true, ml.enabled=true} into {es02}{uEcPa\_WSTF2hpMgqXIC1Ww}{LWBSx4qVQv2-6hJ7KrrItA}{172.18.0.4}{172.18.0.4:9300}{ml.machine\_memory=37779542016, xpack.installed=true, ml.max\_open\_jobs=20, ml.enabled=true}

I'm not sure if it's a message that is trying to recover, or if it's a real error.  
I would hate to loose the database, but if that is the way to go, I'll have to, and then reload all the stored logfiles.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [December 13, 2018, 1:32pm UTC](https://discuss.elastic.co/t/elasticsearch-crashed-on-development-machine-trying-to-recover/160548/2 "2018-12-13T13:32:16Z")

</div>

Is this message still being reported or was it a one off? Is your cluster healthy now that you have fixed the firewall? What does `GET _cluster/health` report? If it's all ok now then there's nothing to worry about, this sort of message is what you'd expect if nodes cannot communicate with each other for a while.

If your cluster is not currently healthy, and you need help recovering it, can you share the output of `GET /_cluster/allocation/explain`?

---

<div class="post-metadata">

**Author:** ![fribse](https://avatars.discourse-cdn.com/v4/letter/f/c67d28/32.png) [@fribse](https://discuss.elastic.co/u/fribse)\
**Post date:** [December 17, 2018, 2:00pm UTC](https://discuss.elastic.co/t/elasticsearch-crashed-on-development-machine-trying-to-recover/160548/3 "2018-12-17T14:00:43Z")

</div>

Hi David

It has quiet down. The health for the cluster is yellow, and it has stopped syncing at 99.98494089300505%

It seems that there are four indices that have problems.  
All the green ones show:  
{"status":"green","number\_of\_shards":5,"number\_of\_replicas":1,"active\_primary\_shards":5,"active\_shards":10,"relocating\_shards":0,"initializing\_shards":0,"unassigned\_shards":0},"filebeat-6.5.1-2016.05.14":

While the four yellow ones show:  
{"status":"yellow","number\_of\_shards":5,"number\_of\_replicas":1,"active\_primary\_shards":5,"active\_shards":9,"relocating\_shards":0,"initializing\_shards":0,"unassigned\_shards":1},"filebeat-6.5.1-2016.05.13":

So it has 1 active\_shard missing on each of these four indices.  
This is also shown on \_cluster/health, it shows 4 unassigned shards.

This is the output from allocation:  
{  
"index": "filebeat-6.5.1-2017.09.20",  
"shard": 3,  
"primary": false,  
"current\_state": "unassigned",  
"unassigned\_info": {  
"reason": "ALLOCATION\_FAILED",  
"at": "2018-12-13T02:46:15.515Z",  
"failed\_allocation\_attempts": 5,  
"details": "failed shard on node [uEcPa\_WSTF2hpMgqXIC1Ww]: failed recovery, failure RecoveryFailedException[[filebeat-6.5.1-2017.09.20][3]: Recovery failed from {es03}{utNZZop8SRuCuX\_KZffjuw}{lDLyWs5ZR1mJHQ1pgIOQVw}{172.20.0.5}{172.20.0.5:9300}{ml.machine\_memory=37779542016, ml.max\_open\_jobs=20, xpack.installed=true, ml.enabled=true} into {es02}{uEcPa\_WSTF2hpMgqXIC1Ww}{BF-RbwnKRzecVVNE\_s8cvw}{172.20.0.3}{172.20.0.3:9300}{ml.machine\_memory=37779542016, xpack.installed=true, ml.max\_open\_jobs=20, ml.enabled=true}]; nested: RemoteTransportException[[es03][172.20.0.5:9300][internal:index/shard/recovery/start\_recovery]]; nested: RecoveryEngineException[Phase[1] prepare target for translog failed]; nested: RemoteTransportException[[es02][172.20.0.3:9300][internal:index/shard/recovery/prepare\_translog]]; nested: TranslogCorruptedException[translog from source [/usr/share/elasticsearch/data/nodes/0/indices/9AHmqT9SRECymSKzcGZH1w/3/translog/translog-21.tlog] is corrupted, expected shard UUID [35 53 30 78 55 35 35 70 51 69 71 66 47 6a 67 41 72 4a 36 59 5a 41] but got: [71 42 70 57 69 52 7a 6e 52 4a 75 69 49 70 75 67 4b 34 43 51 36 67] this translog file belongs to a different translog]; ",  
"last\_allocation\_status": "no\_attempt"  
},  
"can\_allocate": "no",  
"allocate\_explanation": "cannot allocate because allocation is not permitted to any of the nodes",  
"node\_allocation\_decisions": [  
{  
"node\_id": "RuY8hzATQLimTu4IQ0c--Q",  
"node\_name": "es01",  
"transport\_address": "172.20.0.4:9300",  
"node\_attributes": {  
"ml.machine\_memory": "37779542016",  
"xpack.installed": "true",  
"ml.max\_open\_jobs": "20",  
"ml.enabled": "true"  
},  
"node\_decision": "no",  
"deciders": [  
{  
"decider": "max\_retry",  
"decision": "NO",  
"explanation": "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/\_cluster/reroute?retry\_failed=true] to retry, [unassigned\_info[[reason=ALLOCATION\_FAILED], at[2018-12-13T02:46:15.515Z], failed\_attempts[5], delayed=false, details[failed shard on node [uEcPa\_WSTF2hpMgqXIC1Ww]: failed recovery, failure RecoveryFailedException[[filebeat-6.5.1-2017.09.20][3]: Recovery failed from {es03}{utNZZop8SRuCuX\_KZffjuw}{lDLyWs5ZR1mJHQ1pgIOQVw}{172.20.0.5}{172.20.0.5:9300}{ml.machine\_memory=37779542016, ml.max\_open\_jobs=20, xpack.installed=true, ml.enabled=true} into {es02}{uEcPa\_WSTF2hpMgqXIC1Ww}{BF-RbwnKRzecVVNE\_s8cvw}{172.20.0.3}{172.20.0.3:9300}{ml.machine\_memory=37779542016, xpack.installed=true, ml.max\_open\_jobs=20, ml.enabled=true}]; nested: RemoteTransportException[[es03][172.20.0.5:9300][internal:index/shard/recovery/start\_recovery]]; nested: RecoveryEngineException[Phase[1] prepare target for translog failed]; nested: RemoteTransportException[[es02][172.20.0.3:9300][internal:index/shard/recovery/prepare\_translog]]; nested: TranslogCorruptedException[translog from source [/usr/share/elasticsearch/data/nodes/0/indices/9AHmqT9SRECymSKzcGZH1w/3/translog/translog-21.tlog] is corrupted, expected shard UUID [35 53 30 78 55 35 35 70 51 69 71 66 47 6a 67 41 72 4a 36 59 5a 41] but got: [71 42 70 57 69 52 7a 6e 52 4a 75 69 49 70 75 67 4b 34 43 51 36 67] this translog file belongs to a different translog]; ], allocation\_status[no\_attempt]]]"  
}  
]  
},  
{  
"node\_id": "uEcPa\_WSTF2hpMgqXIC1Ww",  
"node\_name": "es02",  
"transport\_address": "172.20.0.3:9300",  
"node\_attributes": {  
"ml.machine\_memory": "37779542016",  
"ml.max\_open\_jobs": "20",  
"xpack.installed": "true",  
"ml.enabled": "true"  
},  
"node\_decision": "no",  
"deciders": [  
{  
"decider": "max\_retry",  
"decision": "NO",  
"explanation": "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/\_cluster/reroute?retry\_failed=true] to retry, [unassigned\_info[[reason=ALLOCATION\_FAILED], at[2018-12-13T02:46:15.515Z], failed\_attempts[5], delayed=false, details[failed shard on node [uEcPa\_WSTF2hpMgqXIC1Ww]: failed recovery, failure RecoveryFailedException[[filebeat-6.5.1-2017.09.20][3]: Recovery failed from {es03}{utNZZop8SRuCuX\_KZffjuw}{lDLyWs5ZR1mJHQ1pgIOQVw}{172.20.0.5}{172.20.0.5:9300}{ml.machine\_memory=37779542016, ml.max\_open\_jobs=20, xpack.installed=true, ml.enabled=true} into {es02}{uEcPa\_WSTF2hpMgqXIC1Ww}{BF-RbwnKRzecVVNE\_s8cvw}{172.20.0.3}{172.20.0.3:9300}{ml.machine\_memory=37779542016, xpack.installed=true, ml.max\_open\_jobs=20, ml.enabled=true}]; nested: RemoteTransportException[[es03][172.20.0.5:9300][internal:index/shard/recovery/start\_recovery]]; nested: RecoveryEngineException[Phase[1] prepare target for translog failed]; nested: RemoteTransportException[[es02][172.20.0.3:9300][internal:index/shard/recovery/prepare\_translog]]; nested: TranslogCorruptedException[translog from source [/usr/share/elasticsearch/data/nodes/0/indices/9AHmqT9SRECymSKzcGZH1w/3/translog/translog-21.tlog] is corrupted, expected shard UUID [35 53 30 78 55 35 35 70 51 69 71 66 47 6a 67 41 72 4a 36 59 5a 41] but got: [71 42 70 57 69 52 7a 6e 52 4a 75 69 49 70 75 67 4b 34 43 51 36 67] this translog file belongs to a different translog]; ], allocation\_status[no\_attempt]]]"  
}  
]  
},  
{  
"node\_id": "utNZZop8SRuCuX\_KZffjuw",  
"node\_name": "es03",  
"transport\_address": "172.20.0.5:9300",  
"node\_attributes": {  
"ml.machine\_memory": "37779542016",  
"ml.max\_open\_jobs": "20",  
"xpack.installed": "true",  
"ml.enabled": "true"  
},  
"node\_decision": "no",  
"deciders": [

---

<div class="post-metadata">

**Author:** ![fribse](https://avatars.discourse-cdn.com/v4/letter/f/c67d28/32.png) [@fribse](https://discuss.elastic.co/u/fribse)\
**Post date:** [December 17, 2018, 2:00pm UTC](https://discuss.elastic.co/t/elasticsearch-crashed-on-development-machine-trying-to-recover/160548/4 "2018-12-17T14:00:52Z")

</div>

```
    {
      "decider": "max_retry",
      "decision": "NO",
      "explanation": "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2018-12-13T02:46:15.515Z], failed_attempts[5], delayed=false, details[failed shard on node [uEcPa_WSTF2hpMgqXIC1Ww]: failed recovery, failure RecoveryFailedException[[filebeat-6.5.1-2017.09.20][3]: Recovery failed from {es03}{utNZZop8SRuCuX_KZffjuw}{lDLyWs5ZR1mJHQ1pgIOQVw}{172.20.0.5}{172.20.0.5:9300}{ml.machine_memory=37779542016, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} into {es02}{uEcPa_WSTF2hpMgqXIC1Ww}{BF-RbwnKRzecVVNE_s8cvw}{172.20.0.3}{172.20.0.3:9300}{ml.machine_memory=37779542016, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}]; nested: RemoteTransportException[[es03][172.20.0.5:9300][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[Phase[1] prepare target for translog failed]; nested: RemoteTransportException[[es02][172.20.0.3:9300][internal:index/shard/recovery/prepare_translog]]; nested: TranslogCorruptedException[translog from source [/usr/share/elasticsearch/data/nodes/0/indices/9AHmqT9SRECymSKzcGZH1w/3/translog/translog-21.tlog] is corrupted, expected shard UUID [35 53 30 78 55 35 35 70 51 69 71 66 47 6a 67 41 72 4a 36 59 5a 41] but got: [71 42 70 57 69 52 7a 6e 52 4a 75 69 49 70 75 67 4b 34 43 51 36 67] this translog file belongs to a different translog]; ], allocation_status[no_attempt]]]"
    },
    {
      "decider": "same_shard",
      "decision": "NO",
      "explanation": "the shard cannot be allocated to the same node on which a copy of the shard already exists [[filebeat-6.5.1-2017.09.20][3], node[utNZZop8SRuCuX_KZffjuw], [P], s[STARTED], a[id=TdtGlLWDQoCxYo5yM96dyw]]"
    }
  ]
}

```

]  
}

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [December 17, 2018, 2:11pm UTC](https://discuss.elastic.co/t/elasticsearch-crashed-on-development-machine-trying-to-recover/160548/5 "2018-12-17T14:11:23Z")

</div>

> [@fribse](#):
>
> TranslogCorruptedException[translog from source [/usr/share/elasticsearch/data/nodes/0/indices/9AHmqT9SRECymSKzcGZH1w/3/translog/translog-21.tlog] is corrupted, expected shard UUID [35 53 30 78 55 35 35 70 51 69 71 66 47 6a 67 41 72 4a 36 59 5a 41] but got: [71 42 70 57 69 52 7a 6e 52 4a 75 69 49 70 75 67 4b 34 43 51 36 67] this translog file belongs to a different translog

I do not know quite how we ended up in this state, but the simplest fix would be to set the `number_of_replicas` on each of the affected indices to `0`, wait for the cluster to report its health as `GREEN` and then set it back to `1` to rebuild those replicas again.

---

<div class="post-metadata">

**Author:** ![fribse](https://avatars.discourse-cdn.com/v4/letter/f/c67d28/32.png) [@fribse](https://discuss.elastic.co/u/fribse)\
**Post date:** [December 17, 2018, 3:07pm UTC](https://discuss.elastic.co/t/elasticsearch-crashed-on-development-machine-trying-to-recover/160548/6 "2018-12-17T15:07:28Z")

</div>

I see, that makes perfectly sense.  
Being very new in the elasticsearch world, could you explain me how to that? 😉

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [December 17, 2018, 3:22pm UTC](https://discuss.elastic.co/t/elasticsearch-crashed-on-development-machine-trying-to-recover/160548/7 "2018-12-17T15:22:07Z")

</div>

> [@fribse](#):
>
> could you explain me how to that

Of course:

```auto
PUT /filebeat-6.5.1-2017.09.20/_settings
{"number_of_replicas":0}

```

(do the same for the other yellow indices, replacing `filebeat-6.5.1-2017.09.20` with the index name). Your cluster should now report it's health as green, according to:

```auto
GET /_cluster/health

```

Then restore the replicas with this:

```auto
PUT /filebeat-6.5.1-2017.09.20/_settings
{"number_of_replicas":1}

```

(again, do the same thing for the other indices)

---

<div class="post-metadata">

**Author:** ![fribse](https://avatars.discourse-cdn.com/v4/letter/f/c67d28/32.png) [@fribse](https://discuss.elastic.co/u/fribse)\
**Post date:** [December 18, 2018, 7:50am UTC](https://discuss.elastic.co/t/elasticsearch-crashed-on-development-machine-trying-to-recover/160548/8 "2018-12-18T07:50:26Z")

</div>

Hi David

He, while I was learning and you were explaining, it has solved some of it.  
I now see this message:  
{"index":"filebeat-6.5.1-2015.04.11","shard":1,"primary":false,"current\_state":"unassigned","unassigned\_info":{"reason":"ALLOCATION\_FAILED","at":"2018-12-17T16:34:13.467Z","failed\_allocation\_attempts":5,"details":"failed shard on node [uEcPa\_WSTF2hpMgqXIC1Ww]: failed recovery, failure RecoveryFailedException[[filebeat-6.5.1-2015.04.11][1]: Recovery failed from {es03}{utNZZop8SRuCuX\_KZffjuw}{WkcmaW8vSiWMegcpeiTQmA}{172.22.0.2}{172.22.0.2:9300}{ml.machine\_memory=37779542016, ml.max\_open\_jobs=20, xpack.installed=true, ml.enabled=true} into {es02}{uEcPa\_WSTF2hpMgqXIC1Ww}{Le9IxadgSraVPcqzh0Np\_A}{172.22.0.3}{172.22.0.3:9300}{ml.machine\_memory=37779542016, xpack.installed=true, ml.max\_open\_jobs=20, ml.enabled=true}]; nested: RemoteTransportException[[es03][172.22.0.2:9300][internal:index/shard/recovery/start\_recovery]]; nested: RecoveryEngineException[Phase[1] prepare target for translog failed]; nested: RemoteTransportException[[es02][172.22.0.3:9300][internal:index/shard/recovery/prepare\_translog]]; nested: NoSuchFileException[/usr/share/elasticsearch/data/nodes/0/indices/ThT-y0wiTraU-tyAq4ML4Q/1/translog/translog.ckp]; ","last\_allocation\_status":"no\_attempt"},"can\_allocate":"no","allocate\_explanation":"cannot allocate because allocation is not permitted to any of the nodes","node\_allocation\_decisions":[{"node\_id":"RuY8hzATQLimTu4IQ0c--Q","node\_name":"es01","transport\_address":"172.22.0.4:9300","node\_attributes":{"ml.machine\_memory":"37779542016","xpack.installed":"true","ml.max\_open\_jobs":"20","ml.enabled":"true"},"node\_decision":"no","deciders":[{"decider":"max\_retry","decision":"NO","explanation":"shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/\_cluster/reroute?retry\_failed=true] to retry, [unassigned\_info[[reason=ALLOCATION\_FAILED],  
...

So I did a  
curl -XPOST [http://elasticsearch.onead.dk:9200/\_cluster/reroute?retry\_failed=true](http://elasticsearch.onead.dk:9200/_cluster/reroute?retry_failed=true)  
And now it's gone to green, and it shows  
{  
"cluster\_name" : "docker-cluster",  
"status" : "green",  
"timed\_out" : false,  
"number\_of\_nodes" : 3,  
"number\_of\_data\_nodes" : 3,  
"active\_primary\_shards" : 13341,  
"active\_shards" : 26682,  
"relocating\_shards" : 2,  
"initializing\_shards" : 0,  
"unassigned\_shards" : 0,  
"delayed\_unassigned\_shards" : 0,  
"number\_of\_pending\_tasks" : 3,  
"number\_of\_in\_flight\_fetch" : 0,  
"task\_max\_waiting\_in\_queue\_millis" : 4837,  
"active\_shards\_percent\_as\_number" : 100.0,

So I think it is on it's way to recovery

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [December 18, 2018, 7:54am UTC](https://discuss.elastic.co/t/elasticsearch-crashed-on-development-machine-trying-to-recover/160548/9 "2018-12-18T07:54:22Z")

</div>

That is a very, very high number of shards given the size of your cluster. Please read [this blog post which provides some practical guidelines arounds shard sizes and sharding](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster). Then look to change your sharding strategy and bring down the number of shards significantly.

---

<div class="post-metadata">

**Author:** ![fribse](https://avatars.discourse-cdn.com/v4/letter/f/c67d28/32.png) [@fribse](https://discuss.elastic.co/u/fribse)\
**Post date:** [December 18, 2018, 8:11am UTC](https://discuss.elastic.co/t/elasticsearch-crashed-on-development-machine-trying-to-recover/160548/10 "2018-12-18T08:11:16Z")

</div>

Hi Christian

Thankyou for the link. So far we've just had filebeat pump in logfiles to logstash, which then sends it to ES. And that will be out first primary focus, as the logfiles are unusable themselves, but with ES it gives us solid insights to trends.  
The problem with the logfiles is that they are not good, so we have a filebeat doing a 'multiline' interpretation, and then logstash putting the values into the right fields.

But as you say, it looks like that logstash sends to a new shard for each file?  
So should I just manually merge them or can I make the logstash or ES know that it should do it automatically?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [December 18, 2018, 8:14am UTC](https://discuss.elastic.co/t/elasticsearch-crashed-on-development-machine-trying-to-recover/160548/11 "2018-12-18T08:14:29Z")

</div>

If you can show your Logstash configuration we may be able to help spot any issues.

---

<div class="post-metadata">

**Author:** ![fribse](https://avatars.discourse-cdn.com/v4/letter/f/c67d28/32.png) [@fribse](https://discuss.elastic.co/u/fribse)\
**Post date:** [December 18, 2018, 8:19am UTC](https://discuss.elastic.co/t/elasticsearch-crashed-on-development-machine-trying-to-recover/160548/12 "2018-12-18T08:19:29Z")

</div>

```
input {
    beats {
            port => 5000
    }
}

#############################################
# Winlogbeat does not require processing #
#############################################

#############################################
# Interpret Portrait filebeats log #
#############################################

filter {
    if "portrait" in [tags] {
    kv {
                    source => "message"
                    #target => "kvpairs"
                    value_split => ":"
                    field_split => "\n"
                    include_keys => ["Time", "Username", "Instance name", "Exception", "Exception type", "ErrorCode", "Module", "Procedure", "Version", "Memory usage"]
    }
    date {
            match => ["Time", "MM/dd/yyyy hh:mm:ss aa", "M/d/yyyy hh:mm:ss aa", "dd-MM-yyyy HH:mm:ss", "d-M-yyyy HH:mm:ss", "MM/dd/yyyy", "M/d/yyyy", "dd-MM-yyyy", "d-M-yyyy"]
            timezone => "Europe/Copenhagen"
    }
}
}

#############################################

output {
# stdout {}

      elasticsearch {
          hosts => "elasticsearch:9200"
              manage_template => false
                  index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
                      document_type => "%{[@metadata][type]}"
        }

}

```

So as this shows it creates an index for each logfile, and each of them has 5 shards when I look into the cluster health thing.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [December 18, 2018, 8:25am UTC](https://discuss.elastic.co/t/elasticsearch-crashed-on-development-machine-trying-to-recover/160548/13 "2018-12-18T08:25:05Z")

</div>

That looks fine to me. I do not see how that can generate that many shards unless you have uploaded data going back several years. If you are indeed storing data for a very long time period, it would probably make sense to switch to monthly indices instead and also possibly reduce the number of primary shards per index.

---

<div class="post-metadata">

**Author:** ![fribse](https://avatars.discourse-cdn.com/v4/letter/f/c67d28/32.png) [@fribse](https://discuss.elastic.co/u/fribse)\
**Post date:** [December 18, 2018, 8:29am UTC](https://discuss.elastic.co/t/elasticsearch-crashed-on-development-machine-trying-to-recover/160548/14 "2018-12-18T08:29:05Z")

</div>

We do have logfiles going back 5-6 years for 25-30 "logfile producers", which has revealed very interesting problems that we were never aware of before we started this.  
So could I just modify the 'index' line in logstash to

```auto
                  index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM}"
                      document_type => "%{[@metadata][type]}"

```

?  
If that works, then I see two options, going through all these indexes and merging them, or kill all the data and reimport it, I guess there is no automatic way of merging the existing ones is there?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [December 18, 2018, 8:33am UTC](https://discuss.elastic.co/t/elasticsearch-crashed-on-development-machine-trying-to-recover/160548/15 "2018-12-18T08:33:03Z")

</div>

That looks correct. You should also look at setting the number of primary shards through an [index template](https://www.elastic.co/guide/en/elasticsearch/reference/6.5/indices-templates.html). Converting daily indices into monthly will however require reindexing. Given the number of indices you have you will probably need to script this.

---

<div class="post-metadata">

**Author:** ![fribse](https://avatars.discourse-cdn.com/v4/letter/f/c67d28/32.png) [@fribse](https://discuss.elastic.co/u/fribse)\
**Post date:** [December 18, 2018, 8:51am UTC](https://discuss.elastic.co/t/elasticsearch-crashed-on-development-machine-trying-to-recover/160548/16 "2018-12-18T08:51:09Z")

</div>

Ok, so if I change the index to this:  
index =\> "%{[@metadata][tags]}-%{[@metadata][beat]}-%{+YYYY.MM}"

It will identify the logfiles, and then I could create a template like this:

```
PUT _template/portrait
{
    "index_patterns": ["portrait*"],
    "settings": {
    "number_of_shards": 1
 }
}

```

right?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [December 18, 2018, 9:15am UTC](https://discuss.elastic.co/t/elasticsearch-crashed-on-development-machine-trying-to-recover/160548/17 "2018-12-18T09:15:33Z")

</div>

If your indices match the pattern `portrait*` that looks correct, even though that does not seem to be the case based on the config you shared earlier.

---

<div class="post-metadata">

**Author:** ![fribse](https://avatars.discourse-cdn.com/v4/letter/f/c67d28/32.png) [@fribse](https://discuss.elastic.co/u/fribse)\
**Post date:** [December 18, 2018, 9:33am UTC](https://discuss.elastic.co/t/elasticsearch-crashed-on-development-machine-trying-to-recover/160548/18 "2018-12-18T09:33:21Z")

</div>

Yes, true, but it should mach them going forward with the index set in this post, right?  
``index =\> "%{[@metadata][tags]}-%{[@metadata][beat]}-%{+YYYY.MM}"  
And then I need to create a script that will run through all the indices and merge them to new indices that match the naming set here.

---

<div class="post-metadata">

**Author:** ![fribse](https://avatars.discourse-cdn.com/v4/letter/f/c67d28/32.png) [@fribse](https://discuss.elastic.co/u/fribse)\
**Post date:** [December 18, 2018, 12:05pm UTC](https://discuss.elastic.co/t/elasticsearch-crashed-on-development-machine-trying-to-recover/160548/19 "2018-12-18T12:05:33Z")

</div>

Seems like I can't use tags directly in the index name in output.  
So for now I'm going to just limit me to changing the daily to monthly 🙂  
Thankyou VERY much for all the insights Christian!

And of course thankyou David for helping me solving the original issue.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 15, 2019, 12:05pm UTC](https://discuss.elastic.co/t/elasticsearch-crashed-on-development-machine-trying-to-recover/160548/20 "2019-01-15T12:05:35Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.