Split index in runtime

Elastic Stack Logstash
1

HI,

we have logstash and elastic on a k8s environment.
in our topology logstash receives data from multiple applications while part of the data is application name \ id. is there a way to add a dynamic variable to the index name or alias and still maintain ILM?
important to note we do not know all application names at logstash startup, and need to react based on arriving traffic

manipulating the "ilm_rollover_alias" or using "index" allows us to change index name but it brakes ILM

    output {
            ..
            elasticsearch {
                ilm_enabled => true
                ilm_rollover_alias => "events"
                ilm_pattern => "{now/d}-000001"
                ilm_policy => "policy_1"
           ..

we would like the index to be: "events--date-000001"

Appreciate your help!

2

When you define an ILM policy you define which template gets used.
When you define a template you can define an index pattern that matches to any index based on that pattern.
So instead of defining your ILM policy in logstash you can create a template in Elasticsearch that is already inherited by an ILM policy.
Then when you write out to Elasticsearch write to an index that uses that template (index pattern matches) and then it should automatically be included in the ILM policy.

3

@AquaX , thanks for your reply!

it actually makes a lot of sense.
while we seem to have made progress, we're still struggling in the implementation.
now the Logstash config looks like so:

elasticsearch {
  index => "event-%{param1}-000001"
  template_name => "event-tmpl"

so the indexes to look:

event-<app_name>-000001

in elastic the index template, "event-tmpl" has the following config:

{ 
 "index": {
  "lifecycle": {
  "name": "events_ilm",
  "rollover_alias": "event"
 },

now the problem is setting the rollover alias, seeing as the name is dynamic (it's located in the mapping) we can't seem to find the correct way to assign it.
please note we have several active log sources and need them routed to different indexes, each should have ILM based on the "events_ilm" policy.

thanks for the assist!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.