Split up indices based on? tags?

Elastic Stack Logstash
1

Hello fellow logstash peeps,
At the moment i am using winlogbeat agents to collect logs/documents to Logstash which sends it to ElasticSearch.

My logstash config looks like this:
input {
beats {
port => 5044
type => "log"
}
}

filter{ .... }

output {
elasticsearch {
hosts => "elasticsearchIP:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

Now i would like to create separated indices based on data so i can remove indices based on type and timestamp.

Now i want to gather logs from a new windows server with winlogbeat, but want the events to be created inside a new index type. And i guess in the future split up the documents into different index types.

I do tag data based on eventlog and specific servers, but the data is still inside the same winlogbeat-yyyy-MM-dd indices. And its harder to remove certain type of data then.

I also use Kibana and define index pattern, today i use winlogbeat-*.
So maybe the new index should be called winlogbeat-logtypeX-yyyy-MM-dd so i still see all events in the same index pattern, but different indices are created in elasticsearch.

If i am on the right track, is it ok to crate an if statement based on tag (from filter) inside the output in logstash and send index names based on that?

Help needed how to Think

Thanks

2

This seems to work, which also makes sure my Kibana index parttern winlogbeat-* still works

output {
  if "radius" in [tags] {
    elasticsearch {
      hosts => "elasticsearchIP:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-radius-%{+YYYY.MM.dd}"
      document_type => "%{[@metadata][type]}"
    }
  } else {
    elasticsearch {
      hosts => "elasticsearchIP:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
      document_type => "%{[@metadata][type]}"
    }
  }
}

How do you other guys separate indices?

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.