Couldn't connect to AWS ES

Hi Gurus,
I'm setting Filebeat to send nginx logs to AWS ElasticSearch service. I do follow exactly this guide


But in step 4, I got this error

Exiting: Couldn't connect to any of the configured Elasticsearch hosts. Errors: [Error connection to Elasticsearch https://search-abcxyz.ap-northeast-1.es.amazonaws.com:443: 403 Forbidden: {"message":"'ZWxhc3RpYzpIYW5vaUAyMDE5' not a valid key=value pair (missing equal-sign) in Authorization header: 'Basic ZWxhc3RpYzpIYW5vaUAyMDE5'."}]

filebeat.yml:

filebeat.inputs:
- type: log

  # Change to true to enable this input configuration.
  enabled: false

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/nginx/*.log

setup.kibana:
  host: "https://search-abcxyz.ap-northeast-1.es.amazonaws.com:443/_plugin/kibana/"
output.elasticsearch:
  hosts: ["https://search-abcxyz.ap-northeast-1.es.amazonaws.com:443"]
  # Enabled ilm (beta) to use index lifecycle management instead daily indices.
  #ilm.enabled: false
  # Optional protocol and basic auth credentials.
  #protocol: "https"
  username: "elastic"
  password: "xxxxxx"

Can anyone help me on this problem?
Thanks.

Does AWS ES support HTTPS basic auth? Can you curl from that host to your Elasticsearch cluster?

Hi Christian,

I dont know if AWS ES supports basic auth or not.
I try with this command
curl --user elastic:xxxxxx "https://search-abcxyz.ap-northeast-1.es.amazonaws.com/"
then I get the response:
{"message":"'ZWxhc3RpYzpIYW5vaUAyMDE5' not a valid key=value pair (missing equal-sign) in Authorization header: 'Basic ZWxhc3RpYzpIYW5vaUAyMDE5'."}

Seems like it does not so you may have to go through Logstash where you can install a specific AWS ES output plugin.

Thanks for your time, Christian.
If anyone centralizes nginx logs from EC2 to AWS ES before, please advise me.
Ah, I forgot to tell you that I used Amazon Cognito for AWS ES authentication.

Many thanks.

I have not used AWS ES but did not have similar problems with Elastic Cloud, which does support HTTPS and basic auth.

Christian,
I can also disable Cognito authentication for AWS ES, but I'm afraid that its security will be quite weak.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.