Set a template_name in Logstash output filter to send data elasticsearch

[aveek]

Hi Guys,

We am using Elastic Stack 7.2.0 . We have setup filebeat to ship data directly to elasticsearch but we have some application log files which requires special parsing. We are sending these logs from filebeat to logstash where we are using custom grok patterns to parse the custom fields.

We want to store this data from logstash in the same index that filebeat creates in elasticsearch . For this purpose we are specifying the template_name to be using in the logstash output filter .

output {
  elasticsearch {
    hosts => "http://hostname:9200"
    index => "filebeat-7.2.0-%{+YYYY.MM.dd}"
    template_name => "filebeat-7.2.0"
  }
}

The template_name specified exists with some special shard settings . We want these setting to be applied to the indices created by logstash and not the default shard settings.

$ curl -X GET http://hostname:9200/_cat/templates | grep filebeat
filebeat-7.2.0              [filebeat-7.2.0-*]  

What is the correct way to use the template settings in logstash output elasticsearch plugin so that the existing filebeat-7.2.0 template is not overwritten and the template is applied to the new indexes created by logstash.

We tried to set the logstash output in the following way but logstash is over writing the template settings in filebeat-7.2.0. However , the indexes are created in the "index" format specified.

output {
  elasticsearch {
    hosts => "http://hostname:9200"
    index => "filebeat-7.2.0-%{+YYYY.MM.dd}"
    manage_template => false
  }
}

[rugenl (Len Rugen)]

The template* parameters for the plugin are for creating/updating managed templates, not for influencing what template the index will use.

At index creation, all templates that have an index pattern matching the index name will be applied in template priority order. You can create a template with any name with an index pattern matching the desired index with only the shard settings, something like my "filebeat-default" template:

{
    "order" : 90,
    "index_patterns" : [
      "filebeat*"
    ],
    "settings" : {
      "number_of_shards" : "1",
      "number_of_replicas" : "1",
      "refresh_interval" : "30s",
      "index.routing.allocation.require.box_type": "hot"
    }
}

[aveek]

Thanks Len for you response.

We already have a template which will match the index pattern created by logstash.

  "filebeat-7.2.0" : {
    "order" : 1,
    "index_patterns" : [
      "filebeat-7.2.0-*"
    ],
    "settings" : {
      "index" : {
        "mapping" : {
          "total_fields" : {
            "limit" : "10000"
          }
        },
        "refresh_interval" : "5s",
        "number_of_routing_shards" : "32",
        "number_of_shards" : "4",
        "number_of_replicas" : "1"

Is it possible to force logstash to set the template for the index it creates on elasticsearch or is the selection of template for an index is completely handled by elasticsearch.

We already set the order of the template to 1 so it takes precedence over all over templates . But if the selection of template is based on index patterns of the templates thenwe can set it order to -1 to guarantee that "filebeat-7.2.0" is picked up everything.

I have set the output with the template_name and it seems logstash is picking up the desired template .

output {
  elasticsearch {
    hosts => "http://hostname:9200"
    index => "filebeat-7.2.0-%{+YYYY.MM.dd}"
    template_name => "filebeat-7.2.0"
  }
}