Connection marked as failed because the onConnect callback failed: cannot retrieve the elasticsearch license: error from server, response code: 500

[ishu52 (Shubham Sureka)]

I am trying to connect to IBM Database for ES service from filebeat. Since ES is SSL connection for which I can get hosts,username,password and base64 encoded certificate. I have converted base64 in .pem format and passed to filebeat and gets below error:

ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(https://ca3cb361-b113-485c-ac08-6c3a237412fb.4b2136ddd30a46e9b7bdb2b2db7f8cd0.databases.appdomain.cloud:31601)): Connection marked as failed because the onConnect callback failed: cannot retrieve the elasticsearch license: error from server, response code: 500

This was working fine earlier with same filebeat yaml and ES but now I get above error. I have tried all possible things. If I point to non-SSL ES then filebeat is connecting.

Let me know what has changed which is not allowing to setup connection and how I can resolve this?

Below is the filebeat.yml configuration:

apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: kube-system
labels:
k8s-app: filebeat
kubernetes.io/cluster-service: "true"
data:
filebeat.yml: |-
filebeat.config:
modules:
path: ${path.config}/modules.d/*.yml
# Reload module configs as they change:
reload.enabled: false

filebeat.autodiscover:
  providers:
    - type: kubernetes
      include_pod_uid: true
      in_cluster: true
      hints.enabled: true
      include_annotations: '*'
      templates:
        - condition.regexp:
            kubernetes.container.name: '.+'
          config:
            - type: docker
              combine_partial: true
              cri.parse_flags: true
              cri.force: true
              containers:
                path: "/var/log/pods/${data.kubernetes.pod.uid}/"
                ids:
                  - "${data.kubernetes.container.name}"
                
                path: "/var/data/kubeletlogs/${data.kubernetes.pod.uid}/"
processors:
  - add_kubernetes_metadata:
      in_cluster: true
  - decode_json_fields:
      fields: ["message"]
      process_array: true
      max_depth: 8
      
output.elasticsearch:
  hosts: ["ca3cb361-b113-485c-ac08-6c3a237412fb.4b2136ddd30a46e9b7bdb2b2db7f8cd0.databases.appdomain.cloud:31601"]
  username: ibm_cloud_46ce6d85_57e4_4f35_8cca_77c4dc85187d
  password: a3eeacc4bf1c5ce017cea8885eca9d2d174be2e11d2f261d655c3716e33e17ca        
  protocol: https
  ssl.certificate_authorities: 
    - /etc/certificate.pem
  indices:
    - index: "max-logs-12"

[TimV (Tim Vernum)]

@ishu52

response code: 500

This implies that Elasticsearch failed when processing the request for the license.
Do your elasticsearch logs show any errors?

[ishu52 (Shubham Sureka)]

Elastic search seems to be running fine. I guess filebeat is failing while connecting.

Is it because of SSL base64 string ? Do i need some other parameter as well ?

[TimV (Tim Vernum)]

Yes, Elasticsearch is running fine - but it is sending an error when filebeat tries to access the license.
Did you check the elasticsearch logs as I suggested?

[ishu52 (Shubham Sureka)]

I checked and seems to be the issue with Filebeat 6.7.2 version. I downgraded filebeat to 6.5.4 and it is wokring fine.

[ishu52 (Shubham Sureka)]

But now the issue is seen with kibana when connecting to ES:

Any idea on this ?