Logstash-filter-elasticsearch plugin fails to look up in amazon es

[srama]

Hi,

New to ELK.
I am trying to use logstash-filter-elasticsearch plugin for some look up and computation.
It works fine with on prem 5.6.3 version.
The same config fails while connection to Amazon ES 6.0.

Here is my config sample.

filter {
if "ABC.process" in [service_name] {
elasticsearch {
hosts => ["someprivatecloud.amazonaws.com:443"]
ssl => true
index => "testindex"
query => 'service_name:"XYZ.process" AND transnumber:%{[transnumber]}'
fields => {"logtimestamp" => "startdate" }
}
ruby {
init => "require 'time'"
code => "duration = ((Time.parse(event.get('logtimestamp')[0]).to_f * 1000) - (Time.parse(event.get('startdate')[0]).to_f * 1000)) rescue nil; event.set('endtime', duration);"
}
}
}

[warkolm (Mark Walkom)]

Can you elaborate on the failure?

[srama]

All I see is a tag with "_elasticsearch_lookup_failure"
I do not know how to enable debug/trace within "filter".
I can provide any trace if you can direct the instructions to capture. ( Thats would be amazing help )

Thank you in advance.

[warkolm (Mark Walkom)]

That implies nothing matched your query, so I would check that there is indeed something that matches.

[srama]

I am certain that the data exists as the same is working fine with 5.6 onprem ES instance.
Is there anyway else I could troubleshoot?

[Christian_Dahlqvist]

In order to authenticate with AWS Elasticsearch service, a special Elasticsearch output plugin provided by Amazon (logstash-output-amazon_es) is required. I suspect this would also be a problem for the standard Elasticsearch filter plugin, so you may want to ask AWS support if they have a version of the Elasticsearch filter plugin.

These standard Logstash plugins should however work with Elastic Cloud as it has a different authentication mechanism, so that might also be an option.

[srama]

Finally, I am able to lookup in amazon es instance.

All I had to do was change these two lines within filter's elasticsearch
hosts => ["someprivatecloud.amazonaws.com:443"]
ssl => true

to

hosts => ["https://someprivatecloud.amazonaws.com:443"]
removed this line -- ssl => true

Thank you all !!