Logstash output: dynamically multiple indexes

Elastic Stack Logstash
1

Hi,

we have logs of our production, qa and dev systems in the same elasticsearch database.
Currently we have one index for all, and I want to separate it, with the target:

  • higher performance
  • lower storage usage
  • different retention time for production and non production.

To not to have to many indizes and shards, I would like to have indizes as followed:
metricbeat-prod-YYYY.MM.DD
metricbeat-staging-YYY.MM.DD
logcluster1-prod-YYYY.MM.DD
logcluster1-staging-YYYY.MM.DD
...

I noticed, that I can use field names in output using %{fieldname} within the index value field.
So I could create the index prefix (prod / staging) a filter or send it by filebeat.

but the point is, can I set a local variable in the output filter, which is not stored as field?
because I don't need the information as field in elasticsearch, only for routing.

Thanks, Andreas

2

but the point is, can I set a local variable in the output filter, which is not stored as field?

Yes, store the field as a subfield to @metadata.

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.