Using the %{foo} syntax in ES output - template parameter

Elastic Stack Logstash
1

Hello all,

Using logstash 5.2 .. can I use the %{foo} syntax in the ES output template_path variable?

output {
    elasticsearch {
        hosts => ["xxxxxx:9200","xxxxxxxx:9200"]
        index => "%{application}-%{+YYYY.MM.dd}"
        template_name => "%{application}"
        template => "/etc/logstash/elastic.d/%{application}_template.json"
        template_overwrite => true
    }
}

It causes error when restart logstash. File does not exist or cannot be opened.

Kind Regards,

Stefaan

1 Like
2

If it complains about not being able to open /etc/logstash/elastic.d/%{application}_template.json then it's indeed not supported.

3

Seeing the same thing on my end, it seems like a weird omission to not allow interpolation for those settings

Only some fields have this note:
This can be dynamic using the `%{foo}` syntax.

4

Seeing the same thing on my end, it seems like a weird omission to not allow interpolation for those settings

Not really. Not all output parameters are suitable to be event-dependent, or at least require great care when implementing.

5

Sorry, that was unfair of me to categorize it like that. I think that it would be a very useful thing to have based on comments here and on stack overflow. http://stackoverflow.com/questions/26724871/logstash-dynamically-assign-template

In our case it would be very valuable to be able to dynamically set the template path especially in order to consolidate down to one elasticsearch output

6

You cannot dynamically assign a template, because templates are uploaded only once, at initialization. Without the flow of traffic, deterministic variable completion does not happen. Since there is no traffic flow during initialization, there is nothing there which can "fill in the blank" for %{application}.

Index templates are only used when a new index is created, and so it is that templates are not uploaded every time a document reached the Elasticsearch output block in Logstash (can you imagine how much slower it would be if Logstash had to do that?). If you intend to have multiple templates, they need to be preexistent, and uploaded to Elasticsearch before any data gets sent there. You can do this with a script of your own making using curl and Elasticsearch API calls. This also permits you to update templates without having to restart Logstash. You could run the script any time before index rollover, and when the new indices get created, they'll have the new template settings.

Logstash can send data to a dynamically configured index name, just as you have above. The template must be pre-existent in Elasticsearch before that is attempted, or Elasticsearch will create a best-guess mapping, rather than what you wanted to send in a template. Honestly, templates can and ought to be completely independent of Logstash. We only added this functionality for an improved out-of-the-box experience for brand new users. The default template is less than ideal for advanced use cases, and Logstash is not a good tool for template management if you have more than one index template.

If you manage the templates outside of Logstash, then this:

can be replaced by this:

output {
   elasticsearch {
        hosts => ["xxxxxx:9200","xxxxxxxx:9200"]
        index => "%{application}-%{+YYYY.MM.dd}"
    }
}

And you get to keep a single elasticsearch output block in Logstash.

2 Likes
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.