How to enforce accepting one specific index mapping type?

Elastic Stack Elasticsearch
1

We have a daily generic logstash-YYYY.mm.dd index that many clients insert various documents into. However, one client was using meta-field “{_type: posts}” while others' documents were being rejected because they used “{_type: fluentd}”. The first client (using type posts) was pointing directly at one Elasticsearch data node.

Then something happened to that host where the elasticsearch process stopped. When we restarted elasticsearch on that host, suddenly all documents with “{_type: posts}” were rejected and only accepted “{_type: fluentd}” documents. This is the message the first client was seeing …
"Rejecting mapping update to [logstash-2020.04.17] as the final mapping would have more than 1 type: [posts, fluentd]"

We want to accept the “{_type: posts}” documents again moving forward from today on, even at the expense of rejecting other types. Though not ideal, unfortunately the clients can not make any changes on their end so we have to fix it on our end.

How do I enforce the logstash indices to accept only "{_type: posts}" documents going forward as they are generated daily?

Thanks.

2

Welcome :slight_smile:

Please start by seeing https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html, you should not be using the _type field, and it provides some alternatives that you should look at.

3

Thank you, Mark :slightly_smiling_face:

I'm not sure if makes a difference or not but our Elasticsearch version is 6.8.

Question 1:
From reading the link you provided, does it seem like creating a custom field called "type" under the "Custom type field" section should be my preferred alternative?

Question 2:
If so, would this be the command to issue to act on today's (already-existing) or tomorrow's (not created yet) index?

    PUT logstash-2020.05.13
    {
      "mappings": {
        "_doc": {
          "properties": {
            "type": { "type": "posts" }
          }
        }
      }
    }

Question 3:
Will that custom field "type" persist for all future logstash indices that will be created?

Thank you.

4

One day you will upgrade, so you're better off following that now and save less hassle later on.

  1. Yes. The best _type value to use is _doc.
  2. You cannot change a mapping once it has been applied. You will need to adjust the _template for tomorrow and future indices.
  3. You will need to define that field in your ingest tool, ie fluentd, you can't pre-define it in the mapping/template.

How similar are your posts and fluentd documents/logs?

5

Thanks for your reply Mark.

Unfortunately I'm new to Elasticsearch and not sure how to go about implementing the changes you suggest. For example, I dont know how to find out which template is for our logstash indices.

How similar are your posts and fluentd documents/logs?
It looks like posts and fluentd documents are very different. They have very different fields.

6

No worries!

What does your Logstash config look like now? Specifically the output section to Elasticsearch.

7

Sorry abt the delay Mark.
How do I get that information to you?

8

Do you have access to your Logstash config?

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.