How to connect beats to AWS elasticsearch with authentication

[wb303]

I am trying to send data from remote filebeats and metricbeats to AWS elasticsearch service. I have authentication enabled but I cannot figure out how I would put AWS credentials in the config. In the metricbeat.yml file there is a stanza:

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["https://search-xxx.us-east-1.es.amazonaws.com:443"]

  # Optional protocol and basic auth credentials.
  protocol: "https"
  username: "xxxxxx"
  password: "xxxxx"

But this doesn't follow the format for AWS credentials nor would I assume auth is basic. This doesn't seem to be in the documentation either. In the error logs I get strings of these:

2017-04-24T22:24:00Z ERR Connecting error publishing events (retrying): 403 Forbidden

What is the correct way to set up AWS authentication?

[andrewkroh]

Does their ES service require a request signed with an "AWS Signature"? Beats doesn't support any sort of provider specific authentication types. You can add custom static headers to the HTTP requests from Beats, use HTTP basic auth, or TLS mutual auth (aka client certs).

I see Amazon has a custom Logstash output for their service. You could route the data through Logstash (like Beats -> Logstash -> AWS ES). Or you could try Elastic Cloud which is Elastic's own hosted ES service that uses X-Pack to secure the cluster using standard mechanisms (TLS + basic auth, or TLS mutual auth).

[wb303]

The service requires Signature v4. I believe that means I can customize the header, except that I don't see how to do that anywhere in the Metricbeat documentation? Are there any guides? For that matter it would be really helpful if the documentation called out that beats don't "support any sort of provider specific authentication types". Logstash adds considerable complexity, and Elastic Cloud is not an option for us.

[andrewkroh]

If you checkout the documentation for the elasticsearch output you will see a field called headers that allows you to add your own header to request.

output.elasticsearch.headers:
  X-My-Header: Header contents

But I don't think a static header will work since the signature for each request needs to be calculated based on the contents of the request. But I don't have any experience with using the AWS ES service and its custom authentication mechanisms.