# Elasticsearch missing index fields after upgrade
**URL:** https://discuss.elastic.co/t/elasticsearch-missing-index-fields-after-upgrade/171457
**Category:** Elasticsearch
**Created:** [March 8, 2019, 8:49am UTC](https://discuss.elastic.co/t/elasticsearch-missing-index-fields-after-upgrade/171457 "2019-03-08T08:49:05Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@Rigrawl](https://discuss.elastic.co/u/Rigrawl)
#### Post date: [March 8, 2019, 8:49am UTC](https://discuss.elastic.co/t/elasticsearch-missing-index-fields-after-upgrade/171457/1 "2019-03-08T08:49:05Z")
Hey,
We are facing an issue after upgrading from ELK 5.6 to 6.6.1. We had around 1,5k index fields prior to update, now we are left with ~30.
We have 2TC nodes with 1 controller on which we have ELK, APM, Metricbeats
This is out current index filed list that we are using.
This is our input\_tomcat.conf file
[Config](https://snippets.cacher.io/snippet/e7c3c5428d1597be61a2)
This is part of our old config, logstash wouldn't start with it
[config\_old](https://snippets.cacher.io/snippet/d16e41d44029e467d070)
We thought that there might be an issue with our custom grok filter but we still cannot get the fields that we want to filter our logs with.
In logstash logs we have this:
Mar 08 10:14:33 portal-nfs-elk logstash[11997]: [2019-03-08T10:14:33,074][WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=\>400, :action=\>["index", {:\_id=\>nil, :\_index=\>"filebeat-2019.03.08", :\_type=\>"doc", :routing=\>nil}, #LogStash::Event:0x19e1d95], :response=\>{"index"=\>{"\_index"=\>"filebeat-2019.03.08", "\_type"=\>"doc", "\_id"=\>"zNFeXGkB9jtq3mp7Oh9j", "status"=\>400, "error"=\>{"type"=\>"illegal\_argument\_exception", "reason"=\>"Rejecting mapping update to [filebeat-2019.03.08] as the final mapping would have more than 1 type: [log, doc]"}}}}
Here is our shard list
Of course we can find the "correlation" in raw log but we don't have the option to filter our logs after correlation in Kibana.
Thanks,
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [March 8, 2019, 6:57pm UTC](https://discuss.elastic.co/t/elasticsearch-missing-index-fields-after-upgrade/171457/2 "2019-03-08T18:57:46Z")
You are trying to create a second mapping type in these indices, but [this is not allowed in indices created since 6.0](https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html). You should either be indexing into the `doc` type or the `log` type, but not both.
Also 1500 fields is too many. Elasticsearch normally [limits you to 1000](https://www.elastic.co/guide/en/elasticsearch/reference/6.6/mapping.html#mapping-limit-settings).
Also you have far too many shards. Each shard should be aiming for the 20-40GB range, whereas you are creating 5 shards daily and storing just a few MB in each. Given the amount of data you're indexing you probably want to consider one-shard monthly indices rather than 5-shard daily ones. Here is an article about oversharding:
> **[How many shards should I have in my Elasticsearch cluster?
| Elastic](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster)**
>
> Elasticsearch is a very versatile platform, that supports a variety of use cases, and provides great flexibility around data organisation and replication strategies. This flexibility can however somet...
---
### Author:  [@Rigrawl](https://discuss.elastic.co/u/Rigrawl)
#### Post date: [March 11, 2019, 12:32pm UTC](https://discuss.elastic.co/t/elasticsearch-missing-index-fields-after-upgrade/171457/3 "2019-03-11T12:32:39Z")
Hey, Thanks for your reply
Where do i change my configuration to have only 1 mapping type in my indices?
Thanks for the tip with the shards, we will try to implement one shard per month.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [March 11, 2019, 1:18pm UTC](https://discuss.elastic.co/t/elasticsearch-missing-index-fields-after-upgrade/171457/4 "2019-03-11T13:18:05Z")
> [@Rigrawl](#):
>
> Where do i change my configuration to have only 1 mapping type in my indices?
Logstash is trying to use the `doc` type:
```plaintext
... :_type=>"doc", ...
```
I've no idea where the `log` type is coming from. Perhaps your index templates?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 8, 2019, 1:18pm UTC](https://discuss.elastic.co/t/elasticsearch-missing-index-fields-after-upgrade/171457/5 "2019-04-08T13:18:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.