ElasticSearch 7.0 upgrade from 6.7.1 issue with Metricbeat templates

[RogerDavies]

Hi

I'm looking at a test ElasticSearch instance @ 6.7.1 upgrade to 7.0. The upgrade assistant says that the metricbeat templates for the cluster need looking at. Basically it says:-

"Index templates [metricbeat-7.0.0, metricbeat-6.6.1, metricbeat-6.6.2] have a number of fields which exceeds the automatic field expansion limit of [1024] and does not have [index.query.default_field] set, which may cause queries which use automatic field expansion, such as query_string, simple_query_string, and multi_match to fail if fields are not explicitly specified in the query."

Any ideas how I might resolve this issue before upgrading?

Thanks

Roger

[json (Jason Bryan)]

Hi Roger,

Interesting. metricbeat-7.0.0 should have the index.query.default_field setting in the template. Can you check GET /_template/metricbeat-7.0.0?

The setting was added to the Metricbeat template in libbeat 6.7.x per https://github.com/elastic/beats/pull/11205.

You can set indices.query.bool.max_clause_count: 2000 in elasticsearch.yml to avoid any issues during the upgrade.

Jason

[RogerDavies]

Sorry Jason, got sidetracked. I've changed the elasticsearch.yml as you suggested and upgraded. The error didn't disappear, but I went ahead anyway.

All works nicely, the metricbeat stuff is much better than the previous version. My logstash config though isn't being respected entirely. my output in the logstash config is elasticsearch { hosts => "http://127.0.0.1:9200" index => "logstash-%{+YYYY.MM.dd}" } but all the logs come through to an index called 'logstash' instead of logstash-YYYY.MM.dd

Probably just have to get used to that. Although I'd like to tidy my config up if it's just going to ignore that.

Thanks

Roger

[json (Jason Bryan)]

Hi Roger,

Sorry for the delay. You could be seeing the one index named logstash due to ILM- https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-ilm which would be enabled by default in Basic and above licenses. Can you try setting ilm_enabled => false?

Jason