Fluentd crashing on startup when trying to connect to Elasticsearch

Hi all,

Recently installed ECK on Kubernetes (hosted on DigitalOcean). Followed the Deploy ECK in your Kubernetes cluster | Elastic Cloud on Kubernetes [2.8] | Elastic tutorial and got Elasticsearch and Kibana up and running. I am now trying to connect Fluentd to Elasticsearch, but am having some issues.

Logs for fluentd are outputting the following continuously:

2023-06-01 09:40:53 +0000 [error]: Worker 0 exited unexpectedly with status 1
2023-06-01 09:40:54 +0000 [info]: #0 init worker0 logger path=nil rotate_age=nil rotate_size=nil
2023-06-01 09:40:54 +0000 [info]: adding match in @FLUENT_LOG pattern="fluent.**" type="null"
2023-06-01 09:40:54 +0000 [info]: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
2023-06-01 09:40:55 +0000 [warn]: #0 [filter_kube_metadata] !! The environment variable 'K8S_NODE_NAME' is not set to the node name which can affect the API server and watch efficiency !!
2023-06-01 09:40:55 +0000 [info]: adding match pattern="**" type="elasticsearch"
The client is unable to verify that the server is Elasticsearch. Some functionality may not be compatible if the server is running an unsupported product.
2023-06-01 09:40:55 +0000 [error]: #0 unexpected error error_class=Elastic::Transport::Transport::Error error="EOFError (EOFError)"
  2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/elastic-transport-8.2.1/lib/elastic/transport/transport/base.rb:324:in `rescue in perform_request'
  2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/elastic-transport-8.2.1/lib/elastic/transport/transport/base.rb:285:in `perform_request'
  2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/elastic-transport-8.2.1/lib/elastic/transport/transport/http/faraday.rb:36:in `perform_request'
  2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/elastic-transport-8.2.1/lib/elastic/transport/client.rb:176:in `perform_request'
  2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/elasticsearch-8.7.0/lib/elasticsearch.rb:71:in `method_missing'
  2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/elasticsearch-api-8.7.0/lib/elasticsearch/api/actions/info.rb:41:in `info'
  2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:498:in `detect_es_major_version'
  2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:489:in `block in handle_last_seen_es_major_version'
  2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/elasticsearch_index_template.rb:56:in `retry_operate'
  2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:486:in `handle_last_seen_es_major_version'
  2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:338:in `configure'
  2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.1/lib/fluent/plugin.rb:187:in `configure'
  2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:132:in `add_match'
  2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:74:in `block in configure'
  2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:64:in `each'
  2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:64:in `configure'
  2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.1/lib/fluent/root_agent.rb:149:in `configure'
  2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.1/lib/fluent/engine.rb:105:in `configure'
  2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.1/lib/fluent/engine.rb:80:in `run_configure'
  2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.1/lib/fluent/supervisor.rb:616:in `block in run_worker'
  2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.1/lib/fluent/supervisor.rb:962:in `main_process'
  2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.1/lib/fluent/supervisor.rb:608:in `run_worker'
  2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.1/lib/fluent/command/fluentd.rb:372:in `<top (required)>'
  2023-06-01 09:40:55 +0000 [error]: #0 <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
  2023-06-01 09:40:55 +0000 [error]: #0 <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
  2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.16.1/bin/fluentd:15:in `<top (required)>'
  2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/bin/fluentd:25:in `load'
  2023-06-01 09:40:55 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/bin/fluentd:25:in `<main>'

My Elasticsearch yaml file:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
spec:
  version: 8.8.0
  nodeSets:
  - name: default
    count: 1
    config:
      node.store.allow_mmap: false
    podTemplate:
      spec:
        containers:
        - name: elasticsearch
          resources:
            requests:
              memory: 1Gi
            limits:
              memory: 1Gi
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        storageClassName: do-block-storage

and my fluentd yaml file:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd
  labels:
    app: fluentd
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluentd
  labels:
    app: fluentd
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - namespaces
  verbs:
  - get
  - list
  - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: fluentd
roleRef:
  kind: ClusterRole
  name: fluentd
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: fluentd
  namespace: elastic-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  labels:
    app: fluentd
spec:
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      serviceAccount: fluentd
      serviceAccountName: fluentd
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      initContainers:
      - name: config-fluentd
        image: busybox
        imagePullPolicy: IfNotPresent
        command: ["/bin/sh","-c"]
        args:
        - cp /fluentd/etc2/tail_container_parse.conf /fluentd/etc/tail_container_parse.conf
        volumeMounts:
        - name: config-path
          mountPath: /fluentd/etc
        - name: config-source
          mountPath: /fluentd/etc2
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1.16-debian-elasticsearch8-1
        env:
          - name:  FLUENT_ELASTICSEARCH_HOST
            value: "elasticsearch-es-http.elastic-system.svc.cluster.local"
          - name:  FLUENT_ELASTICSEARCH_PORT
            value: "9200"
          - name: FLUENT_ELASTICSEARCH_SCHEME
            value: "http"
          - name: FLUENTD_SYSTEMD_CONF
            value: disable
          - name: FLUENT_CONTAINER_TAIL_PARSER_TYPE
            value: /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/
          - name: FLUENT_CONTAINER_TAIL_EXCLUDE_PATH
            value: /var/log/containers/fluentd*,/var/log/containers/konnectivity-agent*
          - name: FLUENT_ELASTICSEARCH_USER
            value: admin
          - name: FLUENT_ELASTICSEARCH_PASSWORD
            value: n8zQy43i0f16Df7pI2fh5G2a
        resources:
          limits:
            memory: 512Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: config-path
          mountPath: /fluentd/etc/tail_container_parse.conf
          subPath: tail_container_parse.conf
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: config-source
        configMap:
          name: fluentd-config
      - name: config-path
        emptyDir: {}

Any help would be appreciated on how I could fix this, can give further logs/code if necessary.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.