Conditional statement in output

[andrew-waters]

I have a logstash configuration file (5.2:alpine) and want to use the same file for development and production environments.

I have loaded in environment variables and confirmed they exist.

However, when trying to change the output based on the environment variable (in this example [environment] is configured in the filter), the elasticsearch configuration combines both declarations and does not see them as unique settings, causing development to fail:

output {

    if [environment] == "development" {

        elasticsearch {
            hosts => ["${ELS_HOSTNAME}:${ELS_PORT}"]
            index => "%{els_index}"
            action => "%{els_action}"
            document_type => "%{type}"
            document_id => "%{id}"
        }
        stdout {
            codec => rubydebug
        }


    } else {

        elasticsearch {
            hosts => ["${ELS_HOSTNAME}:${ELS_PORT}"]
            ssl => "${ELS_SSL:false}"
            ssl_certificate_verification => "${ELS_SSL_VERIFY:false}"
            healthcheck_path => "https://${ELS_USERNAME:guest}:${ELS_PASSWORD:guest}@${ELS_HOSTNAME}:${ELS_PORT}/"
            absolute_healthcheck_path => "true"
            user => "${ELS_USERNAME:guest}"
            password => "${ELS_PASSWORD:guest}"
            index => "%{els_index}"
            action => "%{els_action}"
            document_type => "%{type}"
            document_id => "%{id}"
        }

    }
}

Is my approach to this incorrect or is this a potential bug?

[magnusbaeck (Magnus Bäck)]

You mean both elasticsearch outputs are used as if the if and else lines had just been commented out?

[andrew-waters]

Hi @magnusbaeck, yes - the output is analogous to the following declaration:

output {

    elasticsearch {
        hosts => ["${ELS_HOSTNAME}:${ELS_PORT}"]
        index => "%{els_index}"
        ssl => "${ELS_SSL:false}"
        ssl_certificate_verification => "${ELS_SSL_VERIFY:false}"
        healthcheck_path => "https://${ELS_USERNAME:guest}:${ELS_PASSWORD:guest}@${ELS_HOSTNAME}:${ELS_PORT}/"
        absolute_healthcheck_path => "true"
        user => "${ELS_USERNAME:guest}"
        password => "${ELS_PASSWORD:guest}"
        action => "%{els_action}"
        document_type => "%{type}"
        document_id => "%{id}"
    }
    stdout {
        codec => rubydebug
    }

}

It's as if the conditional statement is all interpreted and the last variable through wins...

[joniba]

Just wondering if you found a way to resolve this? I'm running into the same issue. It seems like the conditionals are completely ignored in this case.

[joniba]

Apparently there is an issue open on this, and also a workaround (though I personally cannot get it to work).

E.g.,

mutate {
add_field => { "[@metadata][LS_ENDP_JDBC]" => "${LS_ENDP_JDBC:a}" }
}

if [@metadata][LS_ENDP_JDBC] == "a" {
...
}

[andrew-waters]

Thanks @joniba , my workaround is to use two logstash configuration files - and use the ENV var in a Dockerfile to copy the correct one across depending on the build pipeline.

I don't mind maintaining that for a little while but it's obviously a lot of superfluous code and prone to mistakes when a larger team maintains it. I'll keep an eye on that issue, thanks for the link - good to know I'm not going crazy over here...