Failed to connect AWS ES cluster

[richzw]

Hi

We try to connect to AWS ES through file beat, However, the following errors comes up

pipeline/output.go:100	Failed to connect to backoff(elasticsearch(https://xxx.us-east-1.es.amazonaws.com:443)): 
Connection marked as failed because the onConnect callback failed: 
cannot retrieve the elasticsearch license: unauthorized access, 
could not connect to the xpack endpoint, verify your credentials

and the config file as

  filebeat.yml: |-
    filebeat.config:
      inputs:
        enabled: true
        path: ${path.config}/inputs.d/*.yml
        reload.enabled: true
        reload.period: 60s
      modules:
        enabled: true
        path: ${path.config}/modules.d/*.yml
        reload.enabled: true
        reload.period: 60s

    output.elasticsearch:
      protocol: "https"
      hosts: ["xxxx.us-east-1.es.amazonaws.com:443"]
      username: "user_id"
      password: "password"
      index: "test-log--%{+yyyy.MM.dd}"
    setup.template.name: "test-log"
    setup.template.pattern: "test-log-*"
    setup.ilm.enabled: false

The filebeat docker image is docker.elastic.co/beats/filebeat:7.13.1

The ES version of Amazon Elasticsearch Service is 7.10

Could someone help us to find the invalid configuration here?

Thanks

[warkolm (Mark Walkom)]

Please see Breaking changes in 7.13 | Beats Platform Reference [7.13] | Elastic

[richzw]

Hi @warkolm

Thank you for your response, we connect to AWS ES with filebeat 7.10.2. The incorrect filebeat version is root cause.

[warkolm (Mark Walkom)]

FYI the aws service is a fork and we cannot provide support for it.

If you want to upgrade to our Elasticsearch Service you will be able to run the latest version of everything.