Using multiple hosts in env var / Elasticsearch filter

[whatgeorgemade (George Bridgeman | georgebridgeman.com)]

Hi all,

I have a three node Elasticsearch cluster. I'm using Logstash to do some enrichment by pulling un-enriched documents from an index, using the elasticsearch filter to look up values, then output again to Elasticsearch.

The URLs for the Elasticsearch nodes are in an environment variable, which - after much reading on GitHub - I have managed to craft into a format that works for both the input and output.

Here's the environment variable defined in /etc/default/logstash:

ES_CLUSTER="http://ip-172-31-2-10.eu-west-2.compute.internal:9200 http://ip-172-31-2-11.eu-west-2.compute.internal:9200 http://ip-172-31-2-12.eu-west-2.compute.internal:9200"

This is the input block:

input {
    elasticsearch {
        hosts => "${ES_CLUSTER}"
        index => "${MGMT_INDEX}"
        schedule => "*/5 * * * *"
        tags => ["_from_elasticsearch"]
    }
}

This is the output block:

output {
    if [class] == "firstClass" {
        elasticsearch {
            hosts => "${ES_CLUSTER}"
            action => "create"
            index => "%{[@metadata][destIndex]}"
        }
    } else {
        elasticsearch {
            hosts => "${ES_CLUSTER}"
            document_id => "%{[idField]}"
            action => "%{[@metadata][op_type]}"
            index => "%{[@metadata][destIndex]}"
        }
    }
}

The filter is in the same format:

elasticsearch {
    hosts => "${ES_CLUSTER}"
    query_template => "tag_lookup.json"
    index => "${MGMT_INDEX}"
    fields => {
        "name" => "tags"
    }
    add_tag => ["ENRICH_SUCCESS_tag_lookup"]
    tag_on_failure => ["ENRICH_FAILURE", "ENRICH_FAILURE_tag_lookup"]
}

I can see in the logs that Logstash correctly interprets the space-delimited list of URIs for both the input and output, and establishes connections to the cluster just fine. The filter, however, fails:

New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://ip-172-31-2-10.eu-west-2.compute.internal:9200", "http://ip-172-31-2-11.eu-west-2.compute.internal:9200", "http://ip-172-31-2-12.eu-west-2.compute.internal:9200"]}
...
New ElasticSearch filter client {:hosts=>["http://ip-172-31-2-10.eu-west-2.compute.internal:9200 http://ip-172-31-2-11.eu-west-2.compute.internal:9200 http://ip-172-31-2-12.eu-west-2.compute.internal:9200"]}
Pipeline error {:pipeline_id=>"main", :exception=>#<URI::InvalidURIError: bad URI(is not URI?): http://ip-172-31-2-10.eu-west-2.compute.internal:9200 http://ip-172-31-2-11.eu-west-2.compute.internal:9200 http://ip-172-31-2-12.eu-west-2.compute.internal:9200>,  ...

Does the Elasticsearch filter not support this syntax? Is there something I'm doing wrong?

Thanks in advance.

[Badger]

@whatgeorgemade

Does the Elasticsearch filter not support this syntax?

No. The conversion of a string that looks like an array into an array of strings is specific to :validate => :uri in the option declaration. The filter expects host:port combos, not uris.