Non-sprintf reference to fields

Elastic Stack Logstash
1

I'm trying to set up environment variables to control my logstash elasticsearch output , like below

    # filter
    environment {
           add_field_from_env => [ "elasticsearch_host", "ES_HOST", "elasticsearch_port", "ES_PORT", "elasticsearch_cluster", "ES_CLUSTER" ]
    }

    # output
    elasticsearch {
            host => "%{[elasticsearch_host]}"
            port => [elasticsearch_port]
            cluster => elasticsearch_cluster
            protocol => "transport"
            flush_size => 10
            index => "application_logs"
    }

However, when I try to run the agent, the output fails - looking at the verbose output it's trying to connect to:

New Elasticsearch output {:cluster=>"elasticsearch_cluster", :host=>"%{[elasticsearch_host]}", :port=>"elasticsearch_port", :embedded=>false, :protocol=>"transport", :level=>:info}

So not using the fields from the event. Looking at the code, it seems like sprintf() needs to be called in the output to use the %{} format, and I'll raise a bug against the output if someone can confirm that is the correct assessment, however is there an alternative way to inject fields from the event without relying on the plugin correctly calling sprintf?

2

Ah, I see why it's done this way. The ElasticSearch client is created at startup, rather than initialised with each event. So there's no way to use the environment variables as I've described - I'll have to modify my own plugin.

3

Yeah, you should be able to use conditionals for this, though it's not perfect if you have unpredictable destinations in each event:

For example, using the environment field name to determine where to send logs for production/staging/etc:

if [environment] == "production" {
  elasticsearch {
    host => "production.example.com"
    ...
  }
} else if [environment] == "staging" {
  elasticsearch {
    host => "staging.example.com"
    ...
  }
} else {
  # neither production nor staging
  ...
}