Hi all,
Recently installed ECK on Kubernetes (hosted on DigitalOcean). Followed the Deploy ECK in your Kubernetes cluster | Elastic Cloud on Kubernetes [2.8] | Elastic tutorial and got Elasticsearch and Kibana up and running. I am now trying to connect Fluentd to Elasticsearch, but am having some issues.
Logs for fluentd are outputting the following continuously:
2023-06-01 09:40:53 +0000 [error]: Worker 0 exited unexpectedly with status 1
2023-06-01 09:40:54 +0000 [info]: #0 init worker0 logger path=nil rotate_age=nil rotate_size=nil
2023-06-01 09:40:54 +0000 [info]: adding match in @FLUENT_LOG pattern="fluent.**" type="null"
2023-06-01 09:40:54 +0000 [info]: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
2023-06-01 09:40:55 +0000 [warn]: #0 [filter_kube_metadata] !! The environment variable 'K8S_NODE_NAME' is not set to the node name which can affect the API server and watch efficiency !!
2023-06-01 09:40:55 +0000 [info]: adding match pattern="**" type="elasticsearch"
The client is unable to verify that the server is Elasticsearch. Some functionality may not be compatible if the server is running an unsupported product.
2023-06-01 09:40:55 +0000 [error]: #0 unexpected error error_class=Elastic::Transport::Transport::Error error="EOFError (EOFError)"
2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/elastic-transport-8.2.1/lib/elastic/transport/transport/base.rb:324:in `rescue in perform_request'
2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/elastic-transport-8.2.1/lib/elastic/transport/transport/base.rb:285:in `perform_request'
2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/elastic-transport-8.2.1/lib/elastic/transport/transport/http/faraday.rb:36:in `perform_request'
2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/elastic-transport-8.2.1/lib/elastic/transport/client.rb:176:in `perform_request'
2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/elasticsearch-8.7.0/lib/elasticsearch.rb:71:in `method_missing'
2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/elasticsearch-api-8.7.0/lib/elasticsearch/api/actions/info.rb:41:in `info'
2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:498:in `detect_es_major_version'
2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:489:in `block in handle_last_seen_es_major_version'
2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/elasticsearch_index_template.rb:56:in `retry_operate'
2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:486:in `handle_last_seen_es_major_version'
2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:338:in `configure'
2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.1/lib/fluent/plugin.rb:187:in `configure'
2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:132:in `add_match'
2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:74:in `block in configure'
2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:64:in `each'
2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:64:in `configure'
2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.1/lib/fluent/root_agent.rb:149:in `configure'
2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.1/lib/fluent/engine.rb:105:in `configure'
2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.1/lib/fluent/engine.rb:80:in `run_configure'
2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.1/lib/fluent/supervisor.rb:616:in `block in run_worker'
2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.1/lib/fluent/supervisor.rb:962:in `main_process'
2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.1/lib/fluent/supervisor.rb:608:in `run_worker'
2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.1/lib/fluent/command/fluentd.rb:372:in `<top (required)>'
2023-06-01 09:40:55 +0000 [error]: #0 <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
2023-06-01 09:40:55 +0000 [error]: #0 <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.1/bin/fluentd:15:in `<top (required)>'
2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/bin/fluentd:25:in `load'
2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/bin/fluentd:25:in `<main>'
My Elasticsearch yaml file:
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: elasticsearch
spec:
version: 8.8.0
nodeSets:
- name: default
count: 1
config:
node.store.allow_mmap: false
podTemplate:
spec:
containers:
- name: elasticsearch
resources:
requests:
memory: 1Gi
limits:
memory: 1Gi
volumeClaimTemplates:
- metadata:
name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: do-block-storage
and my fluentd yaml file:
apiVersion: v1
kind: ServiceAccount
metadata:
name: fluentd
labels:
app: fluentd
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: fluentd
labels:
app: fluentd
rules:
- apiGroups:
- ""
resources:
- pods
- namespaces
verbs:
- get
- list
- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: fluentd
roleRef:
kind: ClusterRole
name: fluentd
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: fluentd
namespace: elastic-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluentd
labels:
app: fluentd
spec:
selector:
matchLabels:
app: fluentd
template:
metadata:
labels:
app: fluentd
spec:
serviceAccount: fluentd
serviceAccountName: fluentd
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
initContainers:
- name: config-fluentd
image: busybox
imagePullPolicy: IfNotPresent
command: ["/bin/sh","-c"]
args:
- cp /fluentd/etc2/tail_container_parse.conf /fluentd/etc/tail_container_parse.conf
volumeMounts:
- name: config-path
mountPath: /fluentd/etc
- name: config-source
mountPath: /fluentd/etc2
containers:
- name: fluentd
image: fluent/fluentd-kubernetes-daemonset:v1.16-debian-elasticsearch8-1
env:
- name: FLUENT_ELASTICSEARCH_HOST
value: "elasticsearch-es-http.elastic-system.svc.cluster.local"
- name: FLUENT_ELASTICSEARCH_PORT
value: "9200"
- name: FLUENT_ELASTICSEARCH_SCHEME
value: "http"
- name: FLUENTD_SYSTEMD_CONF
value: disable
- name: FLUENT_CONTAINER_TAIL_PARSER_TYPE
value: /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/
- name: FLUENT_CONTAINER_TAIL_EXCLUDE_PATH
value: /var/log/containers/fluentd*,/var/log/containers/konnectivity-agent*
- name: FLUENT_ELASTICSEARCH_USER
value: admin
- name: FLUENT_ELASTICSEARCH_PASSWORD
value: n8zQy43i0f16Df7pI2fh5G2a
resources:
limits:
memory: 512Mi
requests:
cpu: 100m
memory: 200Mi
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: config-path
mountPath: /fluentd/etc/tail_container_parse.conf
subPath: tail_container_parse.conf
terminationGracePeriodSeconds: 30
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: config-source
configMap:
name: fluentd-config
- name: config-path
emptyDir: {}
Any help would be appreciated on how I could fix this, can give further logs/code if necessary.