Issue with output plugin type elasticsearch in logstash

[sbk-elk (Bob)]

Hello All,

Below is my logstash configuration. I have a few questions with respect to how i added the fields and how they carry over in the stages of my pipeline.

1. Do the fields set in hash in my input plugin are added as metadata to the other plugins in my pipeline?
2. Also with this configuration instead of getting the index name as set in the filter i am getting the name as below in Elasticsearch kibana with the output plugin PFA, instead of logging-my-apps-dev-us-west1-hello

"_index. %{[@metadata][target_index]}"

input {
  unix {
    path => "/var/run/logstash.sock"
    codec => json
    tags => ["json", "cloud"]
    add_field => {
      "type" => "${LOGSTASH_TYPE:undefined}"
      "env" => "${LOGSTASH_ENV:undefined}"
      "platform" => "${LOGSTASH_PLATFORM:undefined}"
      "project_id" => "${LOGSTASH_PROJECT_ID:undefined}"
      "cloud_region" => "${LOGSTASH_CLOUD_REGION:undefined}"
    }
  }
}


filter {

    if ([input][type] == "unix") {
        json {
          source => "message"
    }
    if ([type] == "hello") {
            mutate { add_field => { "[@metadata][target_index]" => "logging-%{[platform]}-%{[env]}-%{[cloud_region]}-hello" } }
    }
    else {
            mutate { add_field => { "[@metadata][target_index]" => "logging-%{[platform]}-%{[env]}-%{[cloud_region]}-services" } }
    }

    }

}

output {
    elasticsearch {
        cloud_id => "${LOGSTAH_ELASTIC_ID}"
        user => "${LOGSTASH_ELASTIC_USER}"
        password => "${LOGSTASH_ELASTIC_PASSWORD}"
        index => "%{[@metadata][target_index]}"
        retry_initial_interval => 6
        sniffing => false
        manage_template => false
    }
}

[leandrojmp (Leandro Pereira)]

An index name as %{[@metadata][target_index]} means that this field does not exist in your documents.

The fields added in your input will be added at the root of the document, so you will have type, not [input][type] that you are using in your filter.

Since your conditional is using a non-existent field, the field [@metadata][target_index] is not being populate, which explains your index name.

Try to change your conditional to use [type] instead of [input][type].

[sbk-elk (Bob)]

Appreciate the quick response but I updated the filter as below and retried it. No luck!

filter {

    if ([type] == "unix") {
        json {
          source => "message"
    }
    if ([type] == "hello") {
            mutate { add_field => { "[@metadata][target_index]" => "logging-%{[platform]}-%{[env]}-%{[cloud_region]}-hello" } }
    }
    else {
            mutate { add_field => { "[@metadata][target_index]" => "logging-%{[platform]}-%{[env]}-%{[cloud_region]}-services" } }
    }

    }

}

[leandrojmp (Leandro Pereira)]

What is your output from Logstash?

I would suggest that you change your output to stdout to see what is the output.

Add this in your output block:

stdout { codec => rubydebug { metadata => true } }

[sbk-elk (Bob)]

Hey @leandrojmp , thanks for your input. The below configuration worked, the condition before json type in filter plugin isn't right.

input {
  unix {
    path => "/var/run/logstash.sock"
    codec => json
    tags => ["json", "cloud"]
    add_field => {
      "type" => "${LOGSTASH_TYPE:undefined}"
      "env" => "${LOGSTASH_ENV:undefined}"
      "platform" => "${LOGSTASH_PLATFORM:undefined}"
      "project_id" => "${LOGSTASH_PROJECT_ID:undefined}"
      "cloud_region" => "${LOGSTASH_CLOUD_REGION:undefined}"
    }
  }
}


filter {
    json {
          source => "message"
    }
    if ([type] == "hello") {
            mutate { add_field => { "[@metadata][target_index]" => "logging-%{[platform]}-%{[env]}-%{[cloud_region]}-hello" } }
    }
    else {
            mutate { add_field => { "[@metadata][target_index]" => "logging-%{[platform]}-%{[env]}-%{[cloud_region]}-services" } }
    }
}

output {
    elasticsearch {
        cloud_id => "${LOGSTASH_ELASTIC_ID}"
        user => "${LOGSTASH_ELASTIC_USER}"
        password => "${LOGSTASH_ELASTIC_PASSWORD}"
        index => "%{[@metadata][target_index]}"
        retry_initial_interval => 6
        sniffing => false
        manage_template => false
    }
}