Logstash loops

#1

Hi,
how to use loops for output in logstash.

output {

                    if [my_index] == "order"{
            elasticsearch {
                    hosts => [  "172.160.0.234" ]
                    index => "order-%{+YYYY.MM.dd}"
            }
    }
                    if [my_index] == "merchant"{
            elasticsearch {
                    hosts => [  "172.160.0.234}" ]
                    index => "merchant-%{+YYYY.MM.dd}"
            }
    }
	
	                        if [my_index] == "service"{
            elasticsearch {
                    hosts => [  "172.160.0.234" ]
                    index => "service-%{+YYYY.MM.dd}"
            }
    }
                    if [my_index] == "reorder"{
            elasticsearch {
                    hosts => [  "172.160.0.234}" ]
                    index => "reorder-%{+YYYY.MM.dd}"
            }
    }
            else {
                    elasticsearch {
                    hosts => [  "172.160.0.234}" ]
                    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
                            }
    }



 stdout { codec => rubydebug }

}

(Christian Dahlqvist) #2

Why would you need loops? What is the problem you are trying to solve?

#3

I had 12 docker containers, I need to create seperate indices for each container.

(Christian Dahlqvist) #4

Why?

#5

To reduce indices size, usually it is morethan 100gb perday

(Christian Dahlqvist) #6

You do not necessarily need multiple outputs (which can be inefficient) and can do something like this:

filter {
  if [my_index] == "order"{
    mutate { add_field => { "[@metadata][prefix]" => "order" } }
  } else if [my_index] == "merchant" {
    mutate { add_field => { "[@metadata][prefix]" => "merchant" } }
    ...
  } else  {
    mutate { add_field => { "[@metadata][prefix]" => "%{[@metadata][beat]}" } }
  }
}

output {
  elasticsearch {
    index => "%{[@metadata][prefix]}-%{+YYYY.MM.dd}"
  }
}
#7

Thanks for your help
I had one more query in above filter, using if condition is ok for 2 0r 3 items(like order , merchant).
But if i have 8 items then how to do...?

(Christian Dahlqvist) #8

Either expand it (I cut it short to save space) or perhaps try using the translate plugin to set the field.

#9

actually my requirement is to get logs based on condition if condition fails it should be in else part, but in my case last indices that is filebeat-yyyy-xx-xx contains all logs...

filter {
if [my_index] == "order"{
mutate { add_field => { "[@metadata][prefix]" => "order" } }
} else if [my_index] == "merchant" {
mutate { add_field => { "[@metadata][prefix]" => "merchant" } }
...
} else {
mutate { add_field => { "[@metadata][prefix]" => "%{[@metadata][beat]}" } }
}
}

output {
elasticsearch {
index => "%{[@metadata][prefix]}-%{+YYYY.MM.dd}"
}
}

(Christian Dahlqvist) #10

Check what your data looks like and why the conditions in that case appear to be failing.

#11

How can i get @metadata field in kibana

(Christian Dahlqvist) #12

You can not, but you can look at the fields used in the conditionals populating it , e.g. the my_index field in the example above. If this does not exist or is at the wrong level, everything will go to the else clause. If you can show an event that has ended up in the wrong index, we might be able to help.

#13

condition is working fine, may be there will be two @metadata I guess in each log.

(Christian Dahlqvist) #14

I do not understand. Can you please show your config and an example of a document that goes to the incorrect index?