Why manage_template not works for logstash?

Elastic Stack Elasticsearch
1

My logstash config:

input {
  redis {
    host => "127.0.0.1"
    port => 6379
    type => "redis-input"
    data_type => "list"
    key => "key_count"
  }   
}
    
output {
  stdout {}
  elasticsearch {
    manage_template => false
    host => "1.1.1.156"
    cluster => "elasticsearch_dc_001"
    codec => "json"
    protocol => "http"
  }
}

logstash will create index logstash-YYYY.MM.DD automatically, and when I GET the mapping of a brand new type in a index, I find:

{
  "logstash-2015.10.13" : {
    "mappings" : {
      "redis-input" : {
        "dynamic_templates" : [ {
          "string_fields" : {
            "mapping" : {
              "index" : "analyzed",
              "omit_norms" : true,
              "type" : "string",
              "fields" : {
                "raw" : {
                  "ignore_above" : 256,
                  "index" : "not_analyzed",
                  "type" : "string"
                }
              }
            },
            "match" : "*",
            "match_mapping_type" : "string"
          }
        }, {
          "message_field" : {
            "mapping" : {
              "index" : "analyzed",
              "omit_norms" : true,
              "type" : "string"
            },
            "match" : "message",
            "match_mapping_type" : "string"
          }
        } ],
        "_all" : {
          "enabled" : true,
          "omit_norms" : true
        },
        "properties" : {
          "@timestamp" : {
            "type" : "date",
            "format" : "dateOptionalTime"
          },
          "@version" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "geoip" : {
            "dynamic" : "true",
            "properties" : {
              "location" : {
                "type" : "geo_point"
              }
            }
          },
          "geoip.location" : {
            "type" : "string",
            "norms" : {
              "enabled" : false
            },
            "fields" : {
              "raw" : {
                "type" : "string",
                "index" : "not_analyzed",
                "ignore_above" : 256
              }
            }
          }
        }
      },
      "_default_" : {
        "dynamic_templates" : [ {
          "string_fields" : {
            "mapping" : {
              "index" : "analyzed",
              "omit_norms" : true,
              "type" : "string",
              "fields" : {
                "raw" : {
                  "ignore_above" : 256,
                  "index" : "not_analyzed",
                  "type" : "string"
                }
              }
            },
            "match" : "*",
            "match_mapping_type" : "string"
          }
        }, {
          "message_field" : {
            "mapping" : {
              "index" : "analyzed",
              "omit_norms" : true,
              "type" : "string"
            },
            "match" : "message",
            "match_mapping_type" : "string"
          }
        } ],
        "_all" : {
          "enabled" : true,
          "omit_norms" : true
        },
        "properties" : {
          "@version" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "geoip" : {
            "dynamic" : "true",
            "properties" : {
              "location" : {
                "type" : "geo_point"
              }
            }
          }
        }
      }
    }
  }
}

Please notice the dynamic template redis-input mapping my geoip.location to string rather than geo_point. But I think I never make a dynamic template so called redis-input, and I think it must be created by logstash. I make manage_template => false and delete all indices in my ES, but I see that template again...

My question, how to avoid the dynamic template redis-input mapping my geoip.location to string?

2

Did you also remove existing templates?

3

I have checked my template by curl -XGET http://localhost:9200/_template/?pretty, but there is nothing about redis-input.

(ignore template for watch)

{
  "watch_history" : {
    ...
  },
  "logstash" : {
    "order" : 0,
    "template" : "logstash-*",
    "settings" : {
      "index.refresh_interval" : "5s"
    },
    "mappings" : {
      "_default_" : {
        "dynamic_templates" : [ {
          "string_fields" : {
            "mapping" : {
              "index" : "analyzed",
              "omit_norms" : true,
              "type" : "string",
              "fields" : {
                "raw" : {
                  "ignore_above" : 256,
                  "index" : "not_analyzed",
                  "type" : "string"
                }
              }
            },
            "match_mapping_type" : "string",
            "match" : "*"
          }
        } ],
        "_all" : {
          "enabled" : true
        },
        "properties" : {
          "geoip" : {
            "path" : "full",
            "dynamic" : true,
            "type" : "object",
            "properties" : {
              "location" : {
                "type" : "geo_point"
              }
            }
          },
          "@version" : {
            "index" : "not_analyzed",
            "type" : "string"
          }
        }
      }
    },
    "aliases" : { }
  },
  "watches" : {
    ...
  },
  "triggered_watches" : {
    ...
  },
  "logstash_template" : {
    "order" : 3,
    "template" : "logstash-*",
    "settings" : {
      "index.refresh_interval" : "5s"
    },
    "mappings" : {
      "_default_" : {
        "dynamic_templates" : [ {
          "message_field" : {
            "mapping" : {
              "index" : "analyzed",
              "omit_norms" : true,
              "type" : "string"
            },
            "match_mapping_type" : "string",
            "match" : "message"
          }
        }, {
          "string_fields" : {
            "mapping" : {
              "index" : "analyzed",
              "omit_norms" : true,
              "type" : "string",
              "fields" : {
                "raw" : {
                  "ignore_above" : 256,
                  "index" : "not_analyzed",
                  "type" : "string"
                }
              }
            },
            "match_mapping_type" : "string",
            "match" : "*"
          }
        } ],
        "_all" : {
          "omit_norms" : true,
          "enabled" : true
        },
        "properties" : {
          "geoip" : {
            "dynamic" : true,
            "type" : "object",
            "properties" : {
              "location" : {
                "type" : "geo_point"
              }
            }
          },
          "@version" : {
            "index" : "not_analyzed",
            "type" : "string"
          }
        }
      }
    },
    "aliases" : { }
  }
}
4

You need to use overwrite_template => true in your Logstash configuration to overwrite the existing template in Elasticsearch.

5

Sorry for my poor English but I think you misunderstood my problem:

I want to use the template in the ES instead of the one which logstash provides.

If I'm not mistaken the official docs, overwrite_template => true will use the template which logstash provides rather then the one which I configured in ES(logstash-template).

6

I have checked my template by curl -XGET http://localhost:9200/_template/?pretty, but there is nothing about redis-input.

Yes, and that's exactly why Elasticsearch's dynamic mapper kicks in. Unfortunately in your case it guesses wrong about how to map geoip.location and it becomes a string. Just disabling the Logstash template won't help here, you'll have to define your own template that configures that field as being geoip.