Logstash (6.8.4) template handling and include_type_name

[mw_jko]

Hi there,

I'm currently running an ELK stack with version 6.8.4 in a single node setup and preparing it for migration to 7.4.x.

My data is stored through filebeat -> logstash -> elasticsearch where logstash also does the template handling. My output section looks like this:

      elasticsearch {
          hosts => "elasticsearch:9200"
          user => "${LOGSTASH_USER}"
          password => "${LOGSTASH_PASSWORD}"
          index => "mydata-%{[@metadata][index_day]}"
          manage_template => "true"
          template_name => "mydata-template"
          template => "/usr/share/logstash/templates/elasticsearch-7x-template-mydata.json"
          template_overwrite => true
          document_id => "%{[@metadata][fingerprint]}"          
      }

I'm currently checking all the WARNings in the logs of both elasticseach and logstash and found include_type_name warnings like this:

[2019-11-12T10:52:19,789][WARN ][o.e.d.r.a.a.i.RestGetIndexTemplateAction] [Jnsuswi] [types removal] The parameter include_type_name should be explicitly specified in get template requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions.
[2019-11-12T10:52:20,100][WARN ][o.e.d.r.a.a.i.RestPutIndexTemplateAction] [Jnsuswi] [types removal] The parameter include_type_name should be explicitly specified in put template requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', and requests are expected to omit the type name in mapping definitions.

This WARNings are raised when I restart logstash and it tries to fetch/update the template.

Is there a way I can fix this warnings when going the logstash way? From checking the docs it looks like the parameter is a query parameter and in the docs for logstash elasticsearch output don't say a single word about include_type_name)

[mw_jko]

Is there really no way to configure logstash for ?include_type_names=true or ?include_type_names=false request parameter while it's stated everywhere how important this parameter is? (e.g. here: https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html)

Who is creating indexes manually and not through logstash/filebeat?

[mw_jko]

Before someone points me to this. I added the following to the Elasticsearch output plugin already:

parameters => {
   "include_type_name" => false
}

This makes the WARNing in the elasticsearch logs go away, but also poluted the logstash logs with

[WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash:xxxxxx@elasticsearch:9200/?include_type_name=false", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '400' contacting Elasticsearch at URL 'http://elasticsearch:9200/?include_type_name=false'"}

Because the parameter are added to each request, even when it's not supported (like http:///elasticsearch:9200/?include_type_name=true). So it's an even worse solution.

So I'm kinda giving up here, ignore the WARNing in Elasticsearch log and hope I manage to successfully upgrade to 7.x. no matter what.

Looks like I'm either blind and missing the relevant part in the documentation, doing something really wrong or Logstash is becoming a 2nd class citizen in the Elastic world.

[davidhowell.tx (David)]

I'd be curious to know how things go for you. I am on 6.8.2 and planning to move to 7.4. I don't currently use Logstash to manage my templates, but I was thinking about doing it that way. Sounds like I need to wait until after the upgrade..

Currently I manually publish them with the appropriate index patterns using Kibana's Dev Tools. If you end up doing this, I've found one issue.
Creating a test template like so:
PUT _template/test?include_type_name=false
{ "index_patterns": [ "test-*" ],
"mappings": {
"properties": { ... }
}
}
Then if you check the template using GET _template/test you see that it's defaulting to the type of "_doc". However if you check the Logstash documentation, Logstash defaults to a type of "doc". If you try to ingest data into this template you'll get an error stating "illegal_argument_exception", "reason"=>"Rejecting mapping update to [test-2019.11.21] as the final mapping would have more than 1 type: [_doc, doc]"

To get past this, in the Elasticsearch output you have to specify document_type => "_doc" so it matches the default behavior of the new template. I don't know if I'm doing something wrong here, but it doesn't seem that way. I was trying to find out if this is expected behavior when I came across your post.

[davidhowell.tx (David)]

From the Elasticsearch documentation:
In 6.7, the index creation, index template, and mapping APIs support a query string parameter ( include_type_name ) which indicates whether requests and responses should include a type name. It defaults to true , and should be set to an explicit value to prepare to upgrade to 7.0. Not setting include_type_name will result in a deprecation warning. Indices which don’t have an explicit type will use the dummy type name _doc

From the Logstash documentation:
document_type

• Value type is string
• There is no default value for this setting
• This option is deprecated
  ...
  for elasticsearch clusters 6.x and above: the value of doc will be used

These defaults don't align. I updated one of my templates and saw the deprecation warning, so I modified my template and submitted it so that I wouldn't get the warning. This led to errors in Logstash from my previous post. I had to set the document_type to _doc to address the errors. I tried using Logstash 7.4 ingesting into my Elasticsearch 6.8.2 to see if that was any different, but still get the same errors.

[mw_jko]

In my opinion you have to do 2 steps:

1. Make the deprecation WARNing go away (this is something you can do in elasticsearch 6.x)
2. Adjust the template when using elasticsearch 7.x (remove the pseudo type - I think this will not work in 6.x)

About step 1:

What I did is basically:

a) Stop Logstash (I'm using a filebeat which realizes LS is gone and will cache and retry later - no data lost)
b) Update the template using the REST-API (not Logstash)
c) Reindex the current index (to change the document type)
d) Start Logstash

It looks like this procedure does solve most of the deprecations. (I don't care about the include_type_name WARNings anymore, maybe they are gone as well)

My template looked like this before:

{
    "template" : "test-*",
    "settings" : {
        ...
    },
    "mappings" : {
      "_default_" : {
         ...

The "real" document type was set by Logstash output configuration which is deprecated.

The template now looks like this:

{
    "index_patterns" : "test-*",
    "settings" : {
        ...
    },
    "mappings" : {
      "doc" : {
         ...

It was a bit confusing about the _doc pseudotype because it's not actually _doc but doc in the template. My new Logstash output configuration is included in my initial post and works now.

Why the reindex? It is necessary because the current index (e.g. for the current day) already has a document type. Changing the document type in the template will result in a new (second) document type which isn't allowed in Elasticsearch 6.x anymore:

[2019-11-19T14:48:36,512][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"test-2019-11-01", :_type=>"doc", :routing=>nil}, #<LogStash::Event:0x4628236a>], :response=>{"index"=>{"_index"=>"test-2019-11-01", "_type"=>"doc", "_id"=>"DQUihG4BEV4GLD-0__I6", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [test-2019-11-01] as the final mapping would have more than 1 type: [test, doc]"}}}}

I'm pretty much done with step 1 by now. Next one is step 2

Hope it helps