Using amazon_es output plugin via proxy

Elastic Stack Logstash
1

Hi,

I'm trying to use amazon_es output plugin for Logstash to send data to my Elasticsearch cluster in AWS (Elasticsearch service). However, the Logstash instance is installed on a server that needs to go through a proxy to access the internet, and as far as I can tell from the amazon_es documentation (https://github.com/awslabs/logstash-output-amazon_es) there is no way of specifying a proxy for the outbound traffic.

When trying to send data with the amazon_es plugin, I now get the following error in the logstash.log file:

{:timestamp=>"2016-10-04T18:40:13.697000+0200", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"https://subdomain.mydomain.com:443\"]', but an error occurred and it failed! Are you sure you can reach elasticsearch from this machine using the configuration provided?", :client_config=>{:hosts=>["https://subdomain.mydomain.com:443"], :region=>"eu-west-1", :aws_access_key_id=>nil, :aws_secret_access_key=>nil, :transport_options=>{:request=>{:open_timeout=>0, :timeout=>60}, :proxy=>nil}, :transport_class=>Elasticsearch::Transport::Transport:: ...

Judging by the error message, it seems that it would be possible to specify a proxy in the transport_options but not exactly sure how.

My output plugin configuration looks like this:

output {
  amazon_es {
    hosts => ["subdomain.mydomain.com"]
    region => "eu-west-1"
    index => "acc-foobar-%{+YYYY.MM.dd}"
  }
}

(I'm not using my actual domain in the above examples...)

Also, I am reluctant to set HTTP_PROXY or HTTPS_PROXY due to security reasons.

Hope someone has an idea of how to approach this.

Many thanks,

Kristoffer Vinell

2

I think you'll need to ask Amazon on their forums, this isn't a plugin we support.

3

Hi kvinell,

Did you open an AWS ticket for this? Could you post a link?

However I've realised that the plugin does accept a proxy parameter. I've used this successfully with a http proxy but I did have issues trying to post through a secure proxy, I think the SSL certs weren't installed in the key store or something like that
shrug

Gog