Logstash Templates Incorrect

[Borrelworst (Jasper)]

Hi All,

This is not a real issue, well maybe for some of you, but I want to clear some things out:

I am already using Elasticseach for a couple of years and really like the software and what you can do with it. However, I am not really happy with the update/upgrade strategy. It seems that every time when there is a minor or major update, my cluster get's screwed over because of changed mappings,functionality etc.

All of this should not really matter if you can provide is with good examples, however they do not come along with the software.

As an example: I upgraded from 5.6.3 to 6.1.0 and immediately all of my templates are useless, for example the _all field is deprecated (while this is in all of my templates). It kind of screws up all of my historic data.

Now the annoying thing is, I thought using the Logstash ELK6 template as a base, to see what exactly has changed, and this is what really is weird:

In the default template in logstash 6.1.0 there is this part:

{
"template" : "logstash-*",
"version" : 60001,
"settings" : {
"index.refresh_interval" : "5s"
},
"mappings" : {
"default" : {
"dynamic_templates" : [ {

Then when using this template, you will get this in your deprecation logs:

[WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[WARN ][o.e.d.i.m.MapperService ] [default] mapping is deprecated since it is not useful anymore now that indexes cannot have more than one type

This kind of baffles me: While it is already known that these things will get deprecated, it is still in the logstash templates of the same version as Elasticsearch. To me this seems kind of messy and people who develop these things should know better.

I am not sure if this is accidental or on purpose so we all need to take ES courses, but as you promoted that from version 5 and on ES, Logstash and Kibana are in line, just get it right.

I had a discussion yesterday with someone about the same topic, and as I was seeing this in the logs I just wanted to share this. Please feel free to comment or share your own experience with upgrading. If I am the only one having this issue, then I should probably take all the ELK courses !

[thiago (Thiago Souza)]

You don't need to take full ES courses to learn about the breaking changes from one version to another.

Yet some reading is still needed so you stay updated and, for that, I strongly recommend that you read our blog. For instance, for the changes you are referring to, which is the document type deprecation, we blogged about it before and after the 6.0 release.

[Dieter]

I noticed the same.
The logstash defaults seem to create indices with these deprecated mappings.

[val (Val Crettaz)]

@thiago I think that what @Borrelworst is referring to is that in the Logstash elasticsearch output plugin the template for ES 6.x still contains deprecated settings, i.e.

{
 "template" : "logstash-*",                 <---- this is deprecated
 "version" : 60001,
 "settings" : {
   "index.refresh_interval" : "5s"
 },
 "mappings" : {
   "_default_" : {                          <---- this is deprecated
     "dynamic_templates" : [ {

The template for ES 7.x doesn't contain the _default_ mapping anymore but still uses template instead of index_patterns.

Note, though, that there's an open (still unmerged) PR for the first problem and another open issue for the second problem.

[thiago (Thiago Souza)]

The change I mentioned was only an example. My point was to follow our blog to keep updated with breaking changes.

[val (Val Crettaz)]

Sure @thiago but my point was that what is currently shipping with Logstash 6 is somewhat deprecated