# Connection marked as failed because the onConnect callback failed: cannot retrieve the elasticsearch license: error from server, response code: 500

**URL:** <https://discuss.elastic.co/t/connection-marked-as-failed-because-the-onconnect-callback-failed-cannot-retrieve-the-elasticsearch-license-error-from-server-response-code-500/185704>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [June 13, 2019, 5:08pm UTC](https://discuss.elastic.co/t/connection-marked-as-failed-because-the-onconnect-callback-failed-cannot-retrieve-the-elasticsearch-license-error-from-server-response-code-500/185704 "2019-06-13T17:08:32Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![ishu52](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ishu52/32/44325_2.png) [@ishu52](https://discuss.elastic.co/u/ishu52)\
**Post date:** [June 13, 2019, 5:08pm UTC](https://discuss.elastic.co/t/connection-marked-as-failed-because-the-onconnect-callback-failed-cannot-retrieve-the-elasticsearch-license-error-from-server-response-code-500/185704/1 "2019-06-13T17:08:32Z")

</div>

I am trying to connect to IBM Database for ES service from filebeat. Since ES is SSL connection for which I can get hosts,username,password and base64 encoded certificate. I have converted base64 in .pem format and passed to filebeat and gets below error:

**ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch([https://ca3cb361-b113-485c-ac08-6c3a237412fb.4b2136ddd30a46e9b7bdb2b2db7f8cd0.databases.appdomain.cloud:31601](https://ca3cb361-b113-485c-ac08-6c3a237412fb.4b2136ddd30a46e9b7bdb2b2db7f8cd0.databases.appdomain.cloud:31601))): Connection marked as failed because the onConnect callback failed: cannot retrieve the elasticsearch license: error from server, response code: 500**

This was working fine earlier with same filebeat yaml and ES but now I get above error. I have tried all possible things. If I point to non-SSL ES then filebeat is connecting.

Let me know what has changed which is not allowing to setup connection and how I can resolve this?

Below is the filebeat.yml configuration:

apiVersion: v1  
kind: ConfigMap  
metadata:  
name: filebeat-config  
namespace: kube-system  
labels:  
k8s-app: filebeat  
[kubernetes.io/cluster-service:](http://kubernetes.io/cluster-service:) "true"  
data:  
filebeat.yml: |-  
filebeat.config:  
modules:  
path: ${path.config}/modules.d/\*.yml  
# Reload module configs as they change:  
reload.enabled: false

```
filebeat.autodiscover:
  providers:
    - type: kubernetes
      include_pod_uid: true
      in_cluster: true
      hints.enabled: true
      include_annotations: '*'
      templates:
        - condition.regexp:
            kubernetes.container.name: '.+'
          config:
            - type: docker
              combine_partial: true
              cri.parse_flags: true
              cri.force: true
              containers:
                path: "/var/log/pods/${data.kubernetes.pod.uid}/"
                ids:
                  - "${data.kubernetes.container.name}"
                
                path: "/var/data/kubeletlogs/${data.kubernetes.pod.uid}/"
processors:
  - add_kubernetes_metadata:
      in_cluster: true
  - decode_json_fields:
      fields: ["message"]
      process_array: true
      max_depth: 8
      
output.elasticsearch:
  hosts: ["ca3cb361-b113-485c-ac08-6c3a237412fb.4b2136ddd30a46e9b7bdb2b2db7f8cd0.databases.appdomain.cloud:31601"]
  username: ibm_cloud_46ce6d85_57e4_4f35_8cca_77c4dc85187d
  password: a3eeacc4bf1c5ce017cea8885eca9d2d174be2e11d2f261d655c3716e33e17ca        
  protocol: https
  ssl.certificate_authorities: 
    - /etc/certificate.pem
  indices:
    - index: "max-logs-12"
```

---

<div class="post-metadata">

**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\
**Post date:** [June 14, 2019, 3:56am UTC](https://discuss.elastic.co/t/connection-marked-as-failed-because-the-onconnect-callback-failed-cannot-retrieve-the-elasticsearch-license-error-from-server-response-code-500/185704/2 "2019-06-14T03:56:46Z")

</div>

> [@ishu52](#):
>
> response code: 500

This implies that Elasticsearch failed when processing the request for the license.  
Do your elasticsearch logs show any errors?

---

<div class="post-metadata">

**Author:** ![ishu52](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ishu52/32/44325_2.png) [@ishu52](https://discuss.elastic.co/u/ishu52)\
**Post date:** [June 14, 2019, 5:13am UTC](https://discuss.elastic.co/t/connection-marked-as-failed-because-the-onconnect-callback-failed-cannot-retrieve-the-elasticsearch-license-error-from-server-response-code-500/185704/3 "2019-06-14T05:13:59Z")

</div>

![image](https://us1.discourse-cdn.com/elastic/original/3X/8/5/858b282db6cc1f57ae1f1ce80e5ab2cb182f79f5.png)

Elastic search seems to be running fine. I guess filebeat is failing while connecting.

Is it because of SSL base64 string ? Do i need some other parameter as well ?

---

<div class="post-metadata">

**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\
**Post date:** [June 14, 2019, 5:54am UTC](https://discuss.elastic.co/t/connection-marked-as-failed-because-the-onconnect-callback-failed-cannot-retrieve-the-elasticsearch-license-error-from-server-response-code-500/185704/4 "2019-06-14T05:54:25Z")

</div>

Yes, Elasticsearch is running fine - but it is sending an error when filebeat tries to access the license.  
Did you check the elasticsearch logs as I suggested?

---

<div class="post-metadata">

**Author:** ![ishu52](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ishu52/32/44325_2.png) [@ishu52](https://discuss.elastic.co/u/ishu52)\
**Post date:** [June 14, 2019, 7:31am UTC](https://discuss.elastic.co/t/connection-marked-as-failed-because-the-onconnect-callback-failed-cannot-retrieve-the-elasticsearch-license-error-from-server-response-code-500/185704/5 "2019-06-14T07:31:07Z")

</div>

I checked and seems to be the issue with Filebeat 6.7.2 version. I downgraded filebeat to 6.5.4 and it is wokring fine.

---

<div class="post-metadata">

**Author:** ![ishu52](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ishu52/32/44325_2.png) [@ishu52](https://discuss.elastic.co/u/ishu52)\
**Post date:** [June 14, 2019, 7:32am UTC](https://discuss.elastic.co/t/connection-marked-as-failed-because-the-onconnect-callback-failed-cannot-retrieve-the-elasticsearch-license-error-from-server-response-code-500/185704/6 "2019-06-14T07:32:01Z")

</div>

But now the issue is seen with kibana when connecting to ES:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/6/56b097327966b9830ebe7d52275b70d279725da5.png)

Any idea on this ?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 12, 2019, 7:32am UTC](https://discuss.elastic.co/t/connection-marked-as-failed-because-the-onconnect-callback-failed-cannot-retrieve-the-elasticsearch-license-error-from-server-response-code-500/185704/7 "2019-07-12T07:32:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
