I have an ECK setup Overview | Elastic Cloud on Kubernetes [1.0-beta] | Elastic. I am trying to add fluentd so k8 logs can be sent to elasticsearch to be viewed in kibana.
However when i look at the fluentd pod i can see the following errors. It looks like its having trouble connecting to Elasticsearch or finding it?
2020-07-02 15:47:54 +0000 [warn]: #0 [out_es] Could not communicate to
Elasticsearch, resetting connection and trying again. end of file
reached (EOFError) 2020-07-02 15:47:54 +0000 [warn]: #0 [out_es]
Remaining retry: 14. Retry to communicate after 2 second(s).
2020-07-02 15:47:58 +0000 [warn]: #0 [out_es] Could not communicate to
Elasticsearch, resetting connection and trying again. end of file
reached (EOFError) 2020-07-02 15:47:58 +0000 [warn]: #0 [out_es]
Remaining retry: 13. Retry to communicate after 4 second(s).
elastic.yml
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: es-gp2
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
reclaimPolicy: Delete
---
apiVersion: elasticsearch.k8s.elastic.co/v1beta1
kind: Elasticsearch
metadata:
name: data-es
spec:
version: 7.4.2
spec:
http:
tls:
certificate:
secretName: es-cert
nodeSets:
- name: default
count: 2
volumeClaimTemplates:
- metadata:
name: es-data
annotations:
volume.beta.kubernetes.io/storage-class: es-gp2
spec:
accessModes:
- ReadWriteOnce
storageClassName: es-gp2
resources:
requests:
storage: 10Gi
config:
node.master: true
node.data: true
node.ingest: true
node.store.allow_mmap: false
xpack.security.authc.realms:
native:
native1:
order: 1
---
apiVersion: kibana.k8s.elastic.co/v1beta1
kind: Kibana
metadata:
name: data-kibana
spec:
version: 7.4.2
count: 1
elasticsearchRef:
name: data-es
fluentd.yml
apiVersion: v1
kind: ServiceAccount
metadata:
name: fluentd
namespace: elastic
labels:
app: fluentd
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: fluentd
labels:
app: fluentd
rules:
- apiGroups:
- ""
resources:
- pods
- namespaces
verbs:
- get
- list
- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: fluentd
roleRef:
kind: ClusterRole
name: fluentd
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: fluentd
namespace: elastic
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluentd
namespace: elastic
labels:
app: fluentd
spec:
selector:
matchLabels:
app: fluentd
template:
metadata:
labels:
app: fluentd
spec:
serviceAccount: fluentd
serviceAccountName: fluentd
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
containers:
- name: fluentd
image: fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch
env:
- name: FLUENT_ELASTICSEARCH_HOST
value: "data-es-es-http.default"
- name: FLUENT_ELASTICSEARCH_PORT
value: "9200"
- name: FLUENT_ELASTICSEARCH_SCHEME
value: "http"
- name: FLUENTD_SYSTEMD_CONF
value: disable
resources:
limits:
memory: 512Mi
requests:
cpu: 100m
memory: 200Mi
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
terminationGracePeriodSeconds: 30
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
Im unsure about the FLUENT_ELASTICSEARCH_HOST variable. The value i have set is data-es-es-http.default. Because i have a service called data-es-es-http and it's within the default namespace that is exposed a http service over 9200. I also have a data-es-es-default which is not exposed as http.
Ive setup fluentd and only fluentd using the following guide How To Set Up an Elasticsearch, Fluentd and Kibana (EFK) Logging Stack on Kubernetes | DigitalOcean. An existing elasticsearch was already present in the kubernetes cluser which was setup using ECK elastic link above.
data-es-es-default:
Does not look like its exposed as a http service over 9200
data-es-es-http
It looks like i have second service exposed a http service over 9200, im not sure what the difference between these two are.
Curl from within the fluentd pod:
curl -u elastic:mypassword https://data-es-es-http.default:9200 -k
{
"name" : "data-es-es-default-1",
"cluster_name" : "data-es",
"cluster_uuid" : "vPWB0jbBT76Aq6Tbo7ta7w",
"version" : {
"number" : "7.4.2",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "2f90bbf7b93631e52bafb59b3b049cb44ec25e96",
"build_date" : "2019-10-28T20:40:44.881551Z",
"build_snapshot" : false,
"lucene_version" : "8.2.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}