# All field error - Could not index event to Elasticsearch

**URL:** <https://discuss.elastic.co/t/all-field-error-could-not-index-event-to-elasticsearch/120906>\
**Category:** Elasticsearch\
**Created:** [February 21, 2018, 5:30pm UTC](https://discuss.elastic.co/t/all-field-error-could-not-index-event-to-elasticsearch/120906 "2018-02-21T17:30:58Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![atom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/atom/32/34758_2.png) [@atom](https://discuss.elastic.co/u/atom)\
**Post date:** [February 21, 2018, 5:30pm UTC](https://discuss.elastic.co/t/all-field-error-could-not-index-event-to-elasticsearch/120906/1 "2018-02-21T17:30:59Z")

</div>

HI,  
i have done an upgraded of my elkstack (elasticsearch, kibana, logstash, filebeat). Everything was fine yesterday. Now i am getting these error messages in logstash:

> [2018-02-21T18:29:46,901][WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=\>400, :action=\>["index", {:\_id=\>nil, :\_index=\>"logstash-2018.02.21", :\_type=\>"doc", :\_routing=\>nil}, #LogStash::Event:0x66c76646], :response=\>{"index"=\>{"\_index"=\>"logstash-2018.02.21", "\_type"=\>"doc", "\_id"=\>nil, "status"=\>400, "error"=\>{"type"=\>"mapper\_parsing\_exception", "reason"=\>"Failed to parse mapping [_default_]: [include\_in\_all] is not allowed for indices created on or after version 6.0.0 as [\_all] is deprecated. As a replacement, you can use an [copy\_to] on mapping fields to create your own catch all field.", "caused\_by"=\>{"type"=\>"mapper\_parsing\_exception", "reason"=\>"[include\_in\_all] is not allowed for indices created on or after version 6.0.0 as [\_all] is deprecated. As a replacement, you can use an [copy\_to] on mapping fields to create your own catch all field."}}}}}

In my logstash template i can see the \_all field is set to true.  
How can i fix this problem?

Thank you very much!

Regards,  
Ahmet

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 21, 2018, 5:42pm UTC](https://discuss.elastic.co/t/all-field-error-could-not-index-event-to-elasticsearch/120906/2 "2018-02-21T17:42:00Z")

</div>

You need to adjust your template.  
And remove the `_all` field all together from it.

---

<div class="post-metadata">

**Author:** ![atom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/atom/32/34758_2.png) [@atom](https://discuss.elastic.co/u/atom)\
**Post date:** [February 21, 2018, 8:42pm UTC](https://discuss.elastic.co/t/all-field-error-could-not-index-event-to-elasticsearch/120906/3 "2018-02-21T20:42:35Z")

</div>

Hi David,

how can i do this? Do i just need to update my template mapping? Or do i need to create a new tempalte?  
Is there an easy way of doing it?

Thank you so much.

Regards,  
Ahmet

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 21, 2018, 9:33pm UTC](https://discuss.elastic.co/t/all-field-error-could-not-index-event-to-elasticsearch/120906/4 "2018-02-21T21:33:17Z")

</div>

Run an update template request and update your template.

---

<div class="post-metadata">

**Author:** ![atom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/atom/32/34758_2.png) [@atom](https://discuss.elastic.co/u/atom)\
**Post date:** [February 22, 2018, 7:42am UTC](https://discuss.elastic.co/t/all-field-error-could-not-index-event-to-elasticsearch/120906/5 "2018-02-22T07:42:24Z")

</div>

I posted this in Kibana **GET /\_template/logstash** :

> {  
> "logstash": {  
> "order": 0,  
> "version": 50001,  
> "index\_patterns": [  
> "logstash-_"  
> ],  
> "settings": {  
> "index": {  
> "refresh\_interval": "5s"  
> }  
> },  
> "mappings": {  
> "default": {  
> "\_all": {  
> "enabled": true,  
> "norms": false  
> },  
> "dynamic\_templates": [  
> {  
> "message\_field": {  
> "path\_match": "message",  
> "match\_mapping\_type": "string",  
> "mapping": {  
> "type": "text",  
> "norms": false  
> }  
> }  
> },  
> {  
> "string\_fields": {  
> "match": "_",  
> "match\_mapping\_type": "string",  
> "mapping": {  
> "type": "text",  
> "norms": false,  
> "fields": {  
> "keyword": {  
> "type": "keyword"  
> }  
> }  
> }  
> }  
> }  
> ],  
> "properties": {  
> "@timestamp": {  
> "type": "date",  
> "include\_in\_all": false  
> },  
> "@version": {  
> "type": "keyword",  
> "include\_in\_all": false  
> },  
> "geoip": {  
> "dynamic": true,  
> "properties": {  
> "ip": {  
> "type": "ip"  
> },  
> "location": {  
> "type": "geo\_point"  
> },  
> "latitude": {  
> "type": "half\_float"  
> },  
> "longitude": {  
> "type": "half\_float"  
> }  
> }  
> }  
> }  
> }  
> },  
> "aliases": {}  
> }  
> }

How can i update this. This is a template and not a regular index. Right?  
I am a bit lost here, since i am the one who set this system up. I am also very new in elasticserach 🙂

Thanks for the help 👍

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 22, 2018, 8:15am UTC](https://discuss.elastic.co/t/all-field-error-could-not-index-event-to-elasticsearch/120906/6 "2018-02-22T08:15:52Z")

</div>

Yes you can update it with the PUT template API.  
I think that if you delete it and restart logstash it will be created by logstash again

---

<div class="post-metadata">

**Author:** ![atom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/atom/32/34758_2.png) [@atom](https://discuss.elastic.co/u/atom)\
**Post date:** [February 22, 2018, 9:04am UTC](https://discuss.elastic.co/t/all-field-error-could-not-index-event-to-elasticsearch/120906/7 "2018-02-22T09:04:43Z")

</div>

Hi David,

i've tried this:

> PUT /\_template/logstash  
> {  
> "order": 0,  
> "version": 50001,  
> "index\_patterns": [  
> "logstash-_"  
> ],  
> "settings": {  
> "index": {  
> "refresh\_interval": "5s"  
> }  
> },  
> "mappings": {  
> "default": {  
> "dynamic\_templates": [  
> {  
> "message\_field": {  
> "path\_match": "message",  
> "match\_mapping\_type": "string",  
> "mapping": {  
> "type": "text",  
> "norms": false  
> }  
> }  
> },  
> {  
> "string\_fields": {  
> "match": "_",  
> "match\_mapping\_type": "string",  
> "mapping": {  
> "type": "text",  
> "norms": false,  
> "fields": {  
> "keyword": {  
> "type": "keyword"  
> }  
> }  
> }  
> }  
> }  
> ],  
> "properties": {  
> "@timestamp": {  
> "type": "date"  
> },  
> "@version": {  
> "type": "keyword"  
> },  
> "geoip": {  
> "dynamic": true,  
> "properties": {  
> "ip": {  
> "type": "ip"  
> },  
> "location": {  
> "type": "geo\_point"  
> },  
> "latitude": {  
> "type": "half\_float"  
> },  
> "longitude": {  
> "type": "half\_float"  
> }  
> }  
> }  
> }  
> }  
> },  
> "aliases": {}
> 
> }

Now i am getting this:

> [2018-02-22T10:03:42,661][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 503 ({"type"=\>"unavailable\_shards\_exception", "reason"=\>"[logstash-2018.02.22][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[logstash-2018.02.22][0]] containing [17] requests]"})

---

<div class="post-metadata">

**Author:** ![atom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/atom/32/34758_2.png) [@atom](https://discuss.elastic.co/u/atom)\
**Post date:** [February 22, 2018, 9:07am UTC](https://discuss.elastic.co/t/all-field-error-could-not-index-event-to-elasticsearch/120906/8 "2018-02-22T09:07:26Z")

</div>

**GET \_cat/indices/logstash-2018.02.22**  
returns:  
**red open logstash-2018.02.22 4Z7sktgvRJKUEaZMDGeC5Q 5 1**

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 22, 2018, 9:24am UTC](https://discuss.elastic.co/t/all-field-error-could-not-index-event-to-elasticsearch/120906/9 "2018-02-22T09:24:47Z")

</div>

Can you start from scratch or is it a production server?

Also please format your code with `</>` and not the citation icon.

---

<div class="post-metadata">

**Author:** ![atom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/atom/32/34758_2.png) [@atom](https://discuss.elastic.co/u/atom)\
**Post date:** [February 22, 2018, 9:28am UTC](https://discuss.elastic.co/t/all-field-error-could-not-index-event-to-elasticsearch/120906/10 "2018-02-22T09:28:01Z")

</div>

unfortunately it is a production server :-S

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 22, 2018, 9:41am UTC](https://discuss.elastic.co/t/all-field-error-could-not-index-event-to-elasticsearch/120906/11 "2018-02-22T09:41:58Z")

</div>

Can you run:

```auto
GET /_cat/nodes?v
GET /_cat/indices?v
GET /_cat/health?v

```

---

<div class="post-metadata">

**Author:** ![atom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/atom/32/34758_2.png) [@atom](https://discuss.elastic.co/u/atom)\
**Post date:** [February 22, 2018, 9:44am UTC](https://discuss.elastic.co/t/all-field-error-could-not-index-event-to-elasticsearch/120906/12 "2018-02-22T09:44:57Z")

</div>

Of course:

**GET /\_cat/nodes?v**

```
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
127.0.0.1 55 98 12 0.32 0.20 0.29 mdi * shogun

```

**GET /\_cat/indices?v**  
see: [https://paste.ee/p/JkRM4](https://paste.ee/p/JkRM4)

**GET /\_cat/health?v**

```
epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1519292668 10:44:28 logstash red 1 1 1752 1752 0 0 1751 0 - 50.0%
```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 22, 2018, 10:07am UTC](https://discuss.elastic.co/t/all-field-error-could-not-index-event-to-elasticsearch/120906/13 "2018-02-22T10:07:58Z")

</div>

You have too many shards here for a single node.

Can you also run:

```auto
GET /_cluster/allocation/explain

```

---

<div class="post-metadata">

**Author:** ![atom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/atom/32/34758_2.png) [@atom](https://discuss.elastic.co/u/atom)\
**Post date:** [February 22, 2018, 10:29am UTC](https://discuss.elastic.co/t/all-field-error-could-not-index-event-to-elasticsearch/120906/14 "2018-02-22T10:29:08Z")

</div>

Here it is:

```
{
  "index": "logstash-2017.09.23",
  "shard": 2,
  "primary": false,
  "current_state": "unassigned",
  "unassigned_info": {
    "reason": "CLUSTER_RECOVERED",
    "at": "2018-02-22T09:13:32.695Z",
    "last_allocation_status": "no_attempt"
  },
  "can_allocate": "no",
  "allocate_explanation": "cannot allocate because allocation is not permitted to any of the nodes",
  "node_allocation_decisions": [
    {
      "node_id": "--eGugLCRjCZbQO-Fq0ICQ",
      "node_name": "shogun",
      "transport_address": "127.0.0.1:9300",
      "node_decision": "no",
      "deciders": [
        {
          "decider": "enable",
          "decision": "NO",
          "explanation": "no allocations are allowed due to cluster setting [cluster.routing.allocation.enable=none]"
        },
        {
          "decider": "same_shard",
          "decision": "NO",
          "explanation": "the shard cannot be allocated to the same node on which a copy of the shard already exists [[logstash-2017.09.23][2], node[--eGugLCRjCZbQO-Fq0ICQ], [P], s[STARTED], a[id=bxlffJrSTHKX9fXwg6ODXg]]"
        }
      ]
    }
  ]
}
```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 22, 2018, 10:42am UTC](https://discuss.elastic.co/t/all-field-error-could-not-index-event-to-elasticsearch/120906/15 "2018-02-22T10:42:40Z")

</div>

Why `cluster.routing.allocation.enable` is set to `none`?

Can you also run:

```auto
GET /_nodes/stats?human

```

Also as I said you have too many indices and shards here.  
Do you need to keep all those historical data around?

---

<div class="post-metadata">

**Author:** ![atom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/atom/32/34758_2.png) [@atom](https://discuss.elastic.co/u/atom)\
**Post date:** [February 22, 2018, 10:51am UTC](https://discuss.elastic.co/t/all-field-error-could-not-index-event-to-elasticsearch/120906/16 "2018-02-22T10:51:09Z")

</div>

I did not set this setting. I don't know, why this has been set to none.

This is the output:  
see: [https://paste.ee/p/7FDMO](https://paste.ee/p/7FDMO)

This elasticsearch instance stores our production server logs and needs to be kept.  
How can i minimize shards? I had another issue, where you had ansewerd my question there too 🙂 .  
I wanted to minimize shard numbers after i solve this problem here.  
Currently our logs are not stored to elasticsearch anymore 😢

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 22, 2018, 11:12am UTC](https://discuss.elastic.co/t/all-field-error-could-not-index-event-to-elasticsearch/120906/17 "2018-02-22T11:12:51Z")

</div>

Free space is `36.3gb` on `200gb`, so less than 20%. Not a problem but I prefer tell you.  
At some point, you will hit [Disk-based shard allocation | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html).

May be change the value of `cluster.routing.allocation.enable` to `all`. See: [Cluster-level shard allocation | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/shards-allocation.html#_shard_allocation_settings)

May be you can "force" again allocating the missing shard with a cluster reroute call: [Cluster reroute API | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-reroute.html)

`allocate_empty_primary` might help.

Worse case, you can remove all together the `logstash-2017.09.23` index.

> This elasticsearch instance stores our production server logs and needs to be kept.

That does not sound reasonable on a single node with only 4gb of HEAP IMHO.

> How can i minimize shards?

Shrink API might help. But in general I'd suggest looking at the following resources about sizing:

> **[Quantitative Cluster Sizing](https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing)**

> **[How many shards should I have in my Elasticsearch cluster?](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster)**

> **[NetSecureDay: Managing your Black Friday Logs](https://speakerdeck.com/elastic/netsecureday-managing-your-black-friday-logs)**
>
> Surveiller une application complexe n’est pas une tâche aisée, mais avec les bons outils, ce n’est pas si sorcier. Néanmoins, des périodes fortes telles que les opérations de type « Black Friday » (Vendredi noir) ou période de Noël peuvent pousser...

[![](https://us1.discourse-cdn.com/elastic/original/3X/7/c/7c2edebd5bac194c58a5df87ede6872cb41d61a8.jpeg "Managing your Black Friday Logs, Pablo Musa, Elastic, TechSummit Amsterdam") ](https://www.youtube.com/watch?v=ilP7tG6tabI)

> I wanted to minimize shard numbers after i solve this problem here.

I'd may be reduce the number of replicas to 0 (which will not change anything) and then add a new server to share the load on multiple machines.  
You can also think about closing old indices. They will consume less resources which might help.

---

<div class="post-metadata">

**Author:** ![atom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/atom/32/34758_2.png) [@atom](https://discuss.elastic.co/u/atom)\
**Post date:** [February 22, 2018, 11:41am UTC](https://discuss.elastic.co/t/all-field-error-could-not-index-event-to-elasticsearch/120906/18 "2018-02-22T11:41:53Z")

</div>

Hi David,  
_cluster.routing.allocation.enable: all_ in elasticsearch.yml was the solution.

Our ES instance is on an kvm based hypervisor with enough storage. So assigning more disk space is easy to do 🙂  
Thanks for the articles. They will be helpful optimizing our infrastructure.

Thank you so much for your help!

Many Greetings from Germany 🤝

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 22, 2018, 11:42am UTC](https://discuss.elastic.co/t/all-field-error-could-not-index-event-to-elasticsearch/120906/19 "2018-03-22T11:42:13Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.