I am trying to connect to IBM Database for ES service from filebeat. Since ES is SSL connection for which I can get hosts,username,password and base64 encoded certificate. I have converted base64 in .pem format and passed to filebeat and gets below error:
ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(https://ca3cb361-b113-485c-ac08-6c3a237412fb.4b2136ddd30a46e9b7bdb2b2db7f8cd0.databases.appdomain.cloud:31601)): Connection marked as failed because the onConnect callback failed: cannot retrieve the elasticsearch license: error from server, response code: 500
This was working fine earlier with same filebeat yaml and ES but now I get above error. I have tried all possible things. If I point to non-SSL ES then filebeat is connecting.
Let me know what has changed which is not allowing to setup connection and how I can resolve this?
Below is the filebeat.yml configuration:
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: kube-system
labels:
k8s-app: filebeat
kubernetes.io/cluster-service: "true"
data:
filebeat.yml: |-
filebeat.config:
modules:
path: ${path.config}/modules.d/*.yml
# Reload module configs as they change:
reload.enabled: false
filebeat.autodiscover:
providers:
- type: kubernetes
include_pod_uid: true
in_cluster: true
hints.enabled: true
include_annotations: '*'
templates:
- condition.regexp:
kubernetes.container.name: '.+'
config:
- type: docker
combine_partial: true
cri.parse_flags: true
cri.force: true
containers:
path: "/var/log/pods/${data.kubernetes.pod.uid}/"
ids:
- "${data.kubernetes.container.name}"
path: "/var/data/kubeletlogs/${data.kubernetes.pod.uid}/"
processors:
- add_kubernetes_metadata:
in_cluster: true
- decode_json_fields:
fields: ["message"]
process_array: true
max_depth: 8
output.elasticsearch:
hosts: ["ca3cb361-b113-485c-ac08-6c3a237412fb.4b2136ddd30a46e9b7bdb2b2db7f8cd0.databases.appdomain.cloud:31601"]
username: ibm_cloud_46ce6d85_57e4_4f35_8cca_77c4dc85187d
password: a3eeacc4bf1c5ce017cea8885eca9d2d174be2e11d2f261d655c3716e33e17ca
protocol: https
ssl.certificate_authorities:
- /etc/certificate.pem
indices:
- index: "max-logs-12"