How to split index by field value?

Elastic Stack Logstash
#1

I have a VM. There are 2 similiar apps (docker apps) inside it, staging and development. They use 1 filebeat to push log to logstash. On these log, I have field container.labels.com_docker_compose_project. I want to create 2 index. with condition:
First index, will collect log if there is value start with stag on container.labels.com_docker_compose_project field.
Second index, will collect log if there is value start with dev on container.labels.com_docker_compose_project field.

Is it possible? If possible how?

I want to try something like this:
if "^aeon-piccolo" in [container.labels.com_docker_compose_project]
But I think it didnt work.

#2

Hi you could change the metadata fields for target index like this

in a filter

filter {
 if "^dev" in [tags] {      mutate{ add_field =>{"[@metadata][target_index]"=> "devindex-%{+YYYY-MM-dd}"}}   }
}
 output {
index => "%{[@metadata][target_index]}"

}

:wink:

#3

Is [tags] mean [container.labels.com_docker_compose_project] in my case? Or it is literraly [tags]?

1 Like
#4

Could you explain what this syntax did? :sweat_smile:

#5

Yes yes just adapt it to the fields you want to check on, sure i did use tags in my example but it's not mandatory

#6

Use your field not my example :wink:

#7

So my field is something look like this:

- [container.image.name]: dev-example-nginx
- [container.image.name]: stag-example-nginx
- [container.image.name]: dev-example-mysql

  if "^dev-example" in [container.image.name] {
    mutate {
      add_field =>{"[@metadata][target_index]"=> "example-dev"}
    }
  }
  else {
    mutate {
      add_field =>{"[@metadata][target_index]"=> "example-stag"}
      }
  }
  }

output{...}

With above config, I only retrieve index example-stag. Did my regex wrong?

#8

And my index become like this %{[@metadata][target_index]}-example-dev-mysql-2021.03`

My config:

index => "%{[@metadata][target_index]}-%{[container][image][name]}-%{+YYYY.MM}"
#9

You might want to try =~ with the regexp but yeah that's the idea.

Always try to match full strings when you can ( less cpu and more reliable )

#10

"in" here is a substring match, not a regular expression. You cannot use an anchor. So

if "v-ex" in [container.image.name] {

would match. I think you want

if [container.image.name] =~ /^dev-example/
#11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.