How to split index by field value?

I have a VM. There are 2 similiar apps (docker apps) inside it, staging and development. They use 1 filebeat to push log to logstash. On these log, I have field container.labels.com_docker_compose_project. I want to create 2 index. with condition:
First index, will collect log if there is value start with stag on container.labels.com_docker_compose_project field.
Second index, will collect log if there is value start with dev on container.labels.com_docker_compose_project field.

Is it possible? If possible how?

I want to try something like this:
if "^aeon-piccolo" in [container.labels.com_docker_compose_project]
But I think it didnt work.

Hi you could change the metadata fields for target index like this

in a filter

filter {
 if "^dev" in [tags] {      mutate{ add_field =>{"[@metadata][target_index]"=> "devindex-%{+YYYY-MM-dd}"}}   }
 output {
index => "%{[@metadata][target_index]}"



Is [tags] mean [container.labels.com_docker_compose_project] in my case? Or it is literraly [tags]?

1 Like

Could you explain what this syntax did? :sweat_smile:

Yes yes just adapt it to the fields you want to check on, sure i did use tags in my example but it's not mandatory

Use your field not my example :wink:

So my field is something look like this:

- []: dev-example-nginx
- []: stag-example-nginx
- []: dev-example-mysql
  if "^dev-example" in [] {
    mutate {
      add_field =>{"[@metadata][target_index]"=> "example-dev"}
  else {
    mutate {
      add_field =>{"[@metadata][target_index]"=> "example-stag"}


With above config, I only retrieve index example-stag. Did my regex wrong?

And my index become like this %{[@metadata][target_index]}-example-dev-mysql-2021.03`

My config:

index => "%{[@metadata][target_index]}-%{[container][image][name]}-%{+YYYY.MM}"

You might want to try =~ with the regexp but yeah that's the idea.

Always try to match full strings when you can ( less cpu and more reliable )

"in" here is a substring match, not a regular expression. You cannot use an anchor. So

if "v-ex" in [] {

would match. I think you want

if [] =~ /^dev-example/

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.