Final mapping would have more than 1 type After Ugrade 5.x -> 6.x

Elastic Stack Logstash
1

Hi,
i have problem with mappings after upgrade 5.6 to 6.8. Actually priority is how i can set new template for 6.8 version with multiple types

Error
"reason"=>"Rejecting mapping update to [logstash-2020.02.04] as the final mapping would have more than 1 type: [_doc, logfiles]"}}}}

Template

"logstash" : {
            "order" : 0,
            "index_patterns" : [
              "logstash-*"
            ],
            "settings" : {
              "index" : {
                "number_of_shards" : "5",
                "number_of_replicas" : "1",
                "refresh_interval" : "5s"
              }
            },
            "mappings" : {
              "_doc" : {
                "dynamic_templates" : [
                  {
                    "message_field" : {
                      "mapping" : {
                        "norms" : false,
                        "index" : "true",
                        "type" : "keyword"
                      },
                      "match_mapping_type" : "string",
                      "match" : "message"
                    }
                  },
                  {
                    "string_fields" : {
                      "mapping" : {
                        "norms" : false,
                        "index" : "true",
                        "type" : "text",
                        "fields" : {
                          "raw" : {
                            "ignore_above" : 256,
                            "index" : "true",
                            "type" : "keyword"
                          }
                        }
                      },
                      "match_mapping_type" : "string",
                      "match" : "*"
                    }
                  }
                ],
                "properties" : {
                  "server" : {
                    "index" : "true",
                    "type" : "keyword"
                  },
                  "environment" : {
                    "index" : "true",
                    "type" : "keyword"
                  },
                  "component" : {
                    "index" : "true",
                    "type" : "keyword"
                  },
                  "system" : {
                    "index" : "true",
                    "type" : "keyword"
                  },
                  "geoip" : {
                    "dynamic" : true,
                    "type" : "object",
                    "properties" : {
                      "location" : {
                        "type" : "geo_point"
                      }
                    }
                  },
                  "processed_by" : {
                    "index" : "true",
                    "type" : "keyword"
                  },
                  "@version" : {
                    "index" : "true",
                    "type" : "keyword"
                  },
                  "type" : {
                    "properties" : {
                      "index" : {
                        "norms" : false,
                        "type" : "text",
                        "fields" : {
                          "raw" : {
                            "ignore_above" : 256,
                            "type" : "keyword"
                          }
                        }
                      }
                    }
                  }
                }
              }
            },
            "aliases" : { }
          }
 }

is this possible? I try add properties to "types"

"type": {
"properties": {
          "logfiles": {
           "index": true,
           "type": "keyword"
}

But probably this is wrong way :(, Also when i set type "_doc" to "logfiles" is works for this index like logstash-{date} but what when i have something like that logstash-{number}-{date}.

Message form logstash (JSON)

{
  "_index": "logstash-2020.02.04",
  "_type": "appname",
  "_id": "Wv9OEHABJQBVNSJTx_z8",
  "_version": 1,
  "_score": null,
  "_source": {
    "app": "appname",
    "server": "serwer.com",
    "clientid": "KlientID",
    "identify": "logstash-2020.02.04",
    "contact_info": "person",
    "message": "text message",
    "type": "logfiles",
    "redis_host": "server.com",
    "path": "/var/log/test.log",
    "environment": "prod",
    "@timestamp": "2020-02-04T13:06:03.064Z",
    "system": "TEST",
    "processed_by": "server.com",
    "loglevel": "ERROR",
    "@version": "1",
    "host": "server",
    "timestamp": "2020-02-04T14:06:03.064+0100"
  },
  "fields": {
    "@timestamp": [
      "2020-02-04T13:06:03.064Z"
    ]
  },
  "sort": [
    1580821563064
  ]
}
2

No, you can't do that. These were already deprecated in 5.6 and, starting in 6.0, new indices cannot have multiple mapping types:

IMPORTANT: Indices created in Elasticsearch 6.0.0 or later may only contain a single mapping type. Indices created in 5.x with multiple mapping types will continue to function as before in Elasticsearch 6.x. Types will be deprecated in APIs in Elasticsearch 7.0.0, and completely removed in 8.0.0.

1 Like
3

So, what should i do now? change output logstash configuration? All template or what?
How to set logstash/template to receive data in new 6.8 env.

4

If you read the link @DavidTurner shared, you will see some solutions. https://www.elastic.co/guide/en/elasticsearch/reference/5.6/removal-of-types.html#_alternatives_to_mapping_types

5

Can i change types in logstash output to another field? line types => systeminfo or some think like that?
I try remove "_doc" from template, create new without type, but after i create new _template types (_doc) appears again. Also i try add ?include_type_name=false

6

I moved your question to #logstash where I believe you can get some advices about this.

1 Like
7

I'll have a guess here: You are trying to upgrade both Logstash and Elasticsearch at the same time to 6.8.

Logstash 5.6 used to take the document type from the pipeline input. In your case this is set to logfiles. Logstash 6.8 actually uses the doc document type here, but you can set it to anything in the Elasticsearch output with the document_type parameter. Logstash 7.x uses _doc. To prepare for a future upgrade to 7.x, I would set this parameter to _doc already.

As Elasticsearch 6.8 already has the ability to use mappings without the document type, I would simply remove the _doc level of the mapping. This would then become

"mappings": {
  "dynamic_templates" : [
[...]
  ],
  "properties": {
[...]
  }
}

Under 6.8, you have to use the include_type_name=false parameter on the REST API to PUT this template. The next time the 6.8 cluster creates an index from this template, it will assume that the document type is _doc.

This then leaves one issue with the current day. As the existing index has already been created with the logfiles document type, you'll have to either delete and re-create it with the _doc document type, or write to a different index. You could for example configure logstash to write to a logstash-6.8-* index for the next day or so.

8

change template like this

"mappings" : {
    "dynamic_templates" : [
      {
        "message_field" : {
          "mapping" : {
            "norms" : false,
            "index" : "true",
            "type" : "keyword"
          },
          "match_mapping_type" : "string",
          "match" : "message"
        }
      },
      {
        "string_fields" : {
          "mapping" : {
            "norms" : false,
            "index" : "true",
            "type" : "text",
            "fields" : {
              "raw" : {
                "ignore_above" : 256,
                "index" : "true",
                "type" : "keyword"
              }
            }
          },
          "match_mapping_type" : "string",
          "match" : "*"
        }
      }
    ],
    "properties" : {

and i have error

"type": "illegal_argument_exception",
"reason": "Malformed [mappings] section for type [dynamic_templates], should include an inner object describing the mapping"

with PUT _template/logstash?include_type_name=false it works but after GET _template/logstash it still here ( i mean "_doc")

I upgrade only elastic and kibana, logstash is steel in older version (5.4.0)

9

That's to be expected. Internally, if no document type is given, Elasticsearch 6.8 treats this as _doc. The API flag allows you to switch between the two representations.

With this template in place, you could re-configure your Logstash 5.4 Elasticsearch output to use the _doc document type. Again, this will require a different index name if the current day's index already exists.

10

Summarizing

  1. Set new PUT _template/logstash?include_type_name=false without "_doc"
"mappings": {
  "dynamic_templates" : [
[...]
  ],
  "properties": {
[...]
  }
}

Then in logstash set type => _doc ? (and it can't be different?) (or i make set mutate to change any type to _doc?).
so where i can change this types? in Kibana/elasticsearch to see? i mean before i have eg. logs
now i se this type eg. index/_doc/logs? or something like that?
Regarding to this

The API flag allows you to switch between the two representations.

Thanks Magnus for patience :slight_smile: You help me understand new functions a lot.

11

When forcing a document type in a Logstash pipeline, I would set it to _doc, and no other value. In Elasticsearch 7.x the _doc part becomes a part of the endpoint and must not be anything different.

As far as Kibana is concerned, nothing changes. You'd still simply declare your index pattern to include the indices you want to search on. So, if you have an index pattern called logstash-*, any index starting with logstash- will be part of this pattern. That allows you to introduce a new index with e.g. the logstash-6.8- prefix during the conversion. Otherwise, you end up having the same issue again that the updated Logstash pipeline (using _doc) sends documents to an existing index that uses the previous document type.

1 Like
12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.