Hi all,
I have a three node Elasticsearch cluster. I'm using Logstash to do some enrichment by pulling un-enriched documents from an index, using the elasticsearch
filter to look up values, then output again to Elasticsearch.
The URLs for the Elasticsearch nodes are in an environment variable, which - after much reading on GitHub - I have managed to craft into a format that works for both the input
and output
.
Here's the environment variable defined in /etc/default/logstash
:
ES_CLUSTER="http://ip-172-31-2-10.eu-west-2.compute.internal:9200 http://ip-172-31-2-11.eu-west-2.compute.internal:9200 http://ip-172-31-2-12.eu-west-2.compute.internal:9200"
This is the input block:
input {
elasticsearch {
hosts => "${ES_CLUSTER}"
index => "${MGMT_INDEX}"
schedule => "*/5 * * * *"
tags => ["_from_elasticsearch"]
}
}
This is the output block:
output {
if [class] == "firstClass" {
elasticsearch {
hosts => "${ES_CLUSTER}"
action => "create"
index => "%{[@metadata][destIndex]}"
}
} else {
elasticsearch {
hosts => "${ES_CLUSTER}"
document_id => "%{[idField]}"
action => "%{[@metadata][op_type]}"
index => "%{[@metadata][destIndex]}"
}
}
}
The filter is in the same format:
elasticsearch {
hosts => "${ES_CLUSTER}"
query_template => "tag_lookup.json"
index => "${MGMT_INDEX}"
fields => {
"name" => "tags"
}
add_tag => ["ENRICH_SUCCESS_tag_lookup"]
tag_on_failure => ["ENRICH_FAILURE", "ENRICH_FAILURE_tag_lookup"]
}
I can see in the logs that Logstash correctly interprets the space-delimited list of URIs for both the input
and output
, and establishes connections to the cluster just fine. The filter
, however, fails:
New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://ip-172-31-2-10.eu-west-2.compute.internal:9200", "http://ip-172-31-2-11.eu-west-2.compute.internal:9200", "http://ip-172-31-2-12.eu-west-2.compute.internal:9200"]}
...
New ElasticSearch filter client {:hosts=>["http://ip-172-31-2-10.eu-west-2.compute.internal:9200 http://ip-172-31-2-11.eu-west-2.compute.internal:9200 http://ip-172-31-2-12.eu-west-2.compute.internal:9200"]}
Pipeline error {:pipeline_id=>"main", :exception=>#<URI::InvalidURIError: bad URI(is not URI?): http://ip-172-31-2-10.eu-west-2.compute.internal:9200 http://ip-172-31-2-11.eu-west-2.compute.internal:9200 http://ip-172-31-2-12.eu-west-2.compute.internal:9200>, ...
Does the Elasticsearch filter
not support this syntax? Is there something I'm doing wrong?
Thanks in advance.