Filebeat doesn't work with AWS ES domain

Elastic Stack Beats
1

Hi Team,
I've been trying to make filebeat working with AWS ES domain , but I keep getting this error:

2020-11-17T11:42:25.740Z ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://search-pogo-logging-3oegsu4hacueitzstmqkmj2jkm.us-east-1.es.amazonaws.com:443)): Connection marked as failed because the onConnect callback failed: cannot retrieve the elasticsearch license from the /_license endpoint, Filebeat requires the default distribution of Elasticsearch. Please make the endpoint accessible to Filebeat so it can verify the license.: unauthorized access, could not connect to the xpack endpoint, verify your credentials

Here is the configurations for filebeat:

 ---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    #filebeat.autodiscover:
    #  providers:
    #    - type: kubernetes
    #      node: ${NODE_NAME}
    #      hints.enabled: true
    #      hints.default_config:
    #        type: container
    #        paths:
    #          - /var/log/containers/*${data.kubernetes.container.id}.log

    processors:
      - add_cloud_metadata:
      - add_host_metadata:

    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
    setup.ilm.enabled: false

I'm using docker.elastic.co/beats/filebeat:7.10 and AWS ES 7.8 , any comments/helps will be really appreciated!

2

Hi,

Welcome to this forum! :partying_face:

I think you have the wrong FileBeat version: ElasticSearch on AWS is not the same as the Elastic ElasticSearch. Both build upon the opensource code of ElasticSearch but they are not the same. Because of this the closedsource version of Elastic FileBeat is incompatible with the AWS Elasticsearch - in your case it fails to read the license endpoint which does not exist.

Therefore, you cannot use the standard version of FileBeat - you have to use the opensource version(I did not find the 7.10 version yet): docker.elastic.co/beats/filebeat-oss:7.9.3

Best regards
Wolfram

3

YOU.ARE.THE.MAN :+1:
Thanks so much for this, I've been trapped for 3 hrs and finally it's working!

4

FYI there is no closed source version of Filebeat.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.