Elasticsearch missing index fields after upgrade

[Rigrawl (Rafał Grochala)]

Hey,

We are facing an issue after upgrading from ELK 5.6 to 6.6.1. We had around 1,5k index fields prior to update, now we are left with ~30.
We have 2TC nodes with 1 controller on which we have ELK, APM, Metricbeats

This is out current index filed list that we are using.

This is our input_tomcat.conf file

Config

This is part of our old config, logstash wouldn't start with it
config_old

We thought that there might be an issue with our custom grok filter but we still cannot get the fields that we want to filter our logs with.

In logstash logs we have this:
Mar 08 10:14:33 portal-nfs-elk logstash[11997]: [2019-03-08T10:14:33,074][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-2019.03.08", :_type=>"doc", :routing=>nil}, #LogStash::Event:0x19e1d95], :response=>{"index"=>{"_index"=>"filebeat-2019.03.08", "_type"=>"doc", "_id"=>"zNFeXGkB9jtq3mp7Oh9j", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [filebeat-2019.03.08] as the final mapping would have more than 1 type: [log, doc]"}}}}

Here is our shard list

Of course we can find the "correlation" in raw log but we don't have the option to filter our logs after correlation in Kibana.

Thanks,

[DavidTurner]

You are trying to create a second mapping type in these indices, but this is not allowed in indices created since 6.0. You should either be indexing into the doc type or the log type, but not both.

Also 1500 fields is too many. Elasticsearch normally limits you to 1000.

Also you have far too many shards. Each shard should be aiming for the 20-40GB range, whereas you are creating 5 shards daily and storing just a few MB in each. Given the amount of data you're indexing you probably want to consider one-shard monthly indices rather than 5-shard daily ones. Here is an article about oversharding:

How many shards should I have in my Elasticsearch cluster? | Elastic

[Rigrawl (Rafał Grochala)]

Hey, Thanks for your reply

Where do i change my configuration to have only 1 mapping type in my indices?

Thanks for the tip with the shards, we will try to implement one shard per month.

[DavidTurner]

@Rigrawl

Where do i change my configuration to have only 1 mapping type in my indices?

Logstash is trying to use the doc type:

... :_type=>"doc", ...

I've no idea where the log type is coming from. Perhaps your index templates?