Can we send logs to different outputs depending on filename they come from?

[RITZ_VERMA]

in logstash i am trying to send logs from different files to different elasticsearch depending on their filename. is this possible?

[rugenl (Len Rugen)]

Sure

output {
    if <your logic here> {
      elasticsearch {..first target..}
    } else {
      elasticsearch {..other target..}
    }
}

[guyboertje]

@RITZ_VERMA

Do you mean a different cluster or a different index in the same cluster?

[RITZ_VERMA]

In the same elastic cluster but on different indexes.i figured it out thanks

[guyboertje]

Did you figure out that the index setting can be interpolated from values held in the event itself?

This eliminates the logic in the output section.

From this:

    if [es_sub_index_] == "metrics" {
      elasticsearch { index => "app_metrics" }
    } else {
      elasticsearch { index => "app_logs" }
    }

to:

     elasticsearch { index => "app_%{es_sub_index}" ... }

You will still need some logic in the filter section to add a value to a field called es_sub_index but using interpolation means that the communication with ES will not be broken into smaller chunks that are less efficient. The full batch will be sent to ES in one REST call and the body contains the divisions. In English, something like "In this index put these docs then in this other index put these docs...".

Using logic in the output section will mean a separate REST call per conditional block and they are not parallelised.

[RITZ_VERMA]

this make things a lot easier
thanks @guyboertje and apologies for late reply