X-pack seems to be creating a logstash index template automatically

Elastic Stack Elasticsearch
1

I have an issue with conflicting logstash index templates, and when deleting the one I don't need, it gets immediately recreated. considering it's called logstash-index-template this follows the similar naming conventions to x-pack. I haven't touched x-pack in my ES configuration, and am trying to figure out how to prevent this template from being created automatically.

2

internal templates are recreated on deletion. However, you can just ignore this template, as it does not apply to indices created by logstash, but only on the internal .logstash index - which you should not use for anything else.

Just having the template created does not really pose a problem from my point of view

3

hmm you are correct, it only cares for the .logstash type. However, checking all my ES templates, I only have one template that applies to logstash-* indices. This template maps the index to a syslog type, yet I'm getting "Rejecting mapping update to [logstash-2019.08.28] as the final mapping would have more than 1 type: [doc, syslog]"

Do you have any pointers as to what would be causing this problem? Typically the "more than 1 type" error is when you have two templates competing with each other, but in this instance I only have the one.

4

do you happen to have two different templates that get applied on the same index patterns for the logstash index, where one template uses doc as its type and one uses syslog?

5

a curl -X GET "[log.server]:9200/_template/*"?pretty shows that there is only one template that get applied to the logstash- pattern, and it is my syslog one

"logstash_template" : {
    "order" : 1,
    "index_patterns" : [
      "logstash-*"
    ],
    "settings" : {
      "index" : {
        "number_of_shards" : "2",
        "number_of_replicas" : "1"
      }
    },
    "mappings" : {
      "syslog" : {
        "properties" : {
          "before" : {
            "type" : "date",
            "format" : "strict_date_time"
          },
          "after" : {
            "type" : "date",
            "format" : "strict_date_time"
          },
          "logsource" : {
            "type" : "ip"
          },
          "time" : {
            "type" : "date",
            "format" : "basic_time_no_millis"
          }
        }
      }
    },
    "aliases" : { }
  },

in the container logs for logstash i see Attempting to install template and the syslog template i listed above.

a curl -X GET "9.3.254.132:9200/logstash-2019.08.29/_mappings/*"?pretty shows the mappings are assigned properly

{
  "logstash-2019.08.29" : {
    "mappings" : {
      "syslog" : {
        "properties" : {
          "after" : {
            "type" : "date",
            "format" : "strict_date_time"
          },
          "before" : {
            "type" : "date",
            "format" : "strict_date_time"
          },
          "logsource" : {
            "type" : "ip"
          },
          "time" : {
            "type" : "date",
            "format" : "basic_time_no_millis"
          }
        }
      }
    }
  }
}
6

is it possible, that the instance trying to write data to Elasticsearch is using the doc type, so that when indexing results in a new index creation you end up with two types?

7

Yes, the issue seems to be that since the template is only caring about the three fields I specified, the other fields in my config will default to doc type, thus creating the type inconsistency. I talked it over, and even though the app we take logs in from are in the form of syslog we do not need them outputted to syslog once they get logstashed, if that's a word.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.