Couldn't connect to AWS ES

Elastic Stack Beats
1

Hi Gurus,
I'm setting Filebeat to send nginx logs to AWS Elasticsearch service. I do follow exactly this guide

filebeat
filebeat1164×863 36.5 KB

But in step 4, I got this error

Exiting: Couldn't connect to any of the configured Elasticsearch hosts. Errors: [Error connection to Elasticsearch https://search-abcxyz.ap-northeast-1.es.amazonaws.com:443: 403 Forbidden: {"message":"'ZWxhc3RpYzpIYW5vaUAyMDE5' not a valid key=value pair (missing equal-sign) in Authorization header: 'Basic ZWxhc3RpYzpIYW5vaUAyMDE5'."}]

filebeat.yml:

filebeat.inputs:
- type: log

  # Change to true to enable this input configuration.
  enabled: false

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/nginx/*.log

setup.kibana:
  host: "https://search-abcxyz.ap-northeast-1.es.amazonaws.com:443/_plugin/kibana/"
output.elasticsearch:
  hosts: ["https://search-abcxyz.ap-northeast-1.es.amazonaws.com:443"]
  # Enabled ilm (beta) to use index lifecycle management instead daily indices.
  #ilm.enabled: false
  # Optional protocol and basic auth credentials.
  #protocol: "https"
  username: "elastic"
  password: "xxxxxx"

Can anyone help me on this problem?
Thanks.

2

Does AWS ES support HTTPS basic auth? Can you curl from that host to your Elasticsearch cluster?

3

Hi Christian,

I dont know if AWS ES supports basic auth or not.
I try with this command
curl --user elastic:xxxxxx "https://search-abcxyz.ap-northeast-1.es.amazonaws.com/"
then I get the response:
{"message":"'ZWxhc3RpYzpIYW5vaUAyMDE5' not a valid key=value pair (missing equal-sign) in Authorization header: 'Basic ZWxhc3RpYzpIYW5vaUAyMDE5'."}

4

Seems like it does not so you may have to go through Logstash where you can install a specific AWS ES output plugin.

5

Thanks for your time, Christian.
If anyone centralizes nginx logs from EC2 to AWS ES before, please advise me.
Ah, I forgot to tell you that I used Amazon Cognito for AWS ES authentication.

Many thanks.

6

I have not used AWS ES but did not have similar problems with Elastic Cloud, which does support HTTPS and basic auth.

7

Christian,
I can also disable Cognito authentication for AWS ES, but I'm afraid that its security will be quite weak.