Fluentd configuration is not creating indexes in elasticsearch

Elastic Stack Elasticsearch
1

Hello everyone,
I need some help with the EFK stack. I've already installed Elasticsearch, Kibana and Fluentd with their respective Helm charts in a k8s environment.

Every pod is running fine but I'm my Fluentd configuration is supposed to create elasticsearch indexes to start storing the data. I'll paste my config map on the fluentd-forwarder-cm

kind: ConfigMap
apiVersion: v1
metadata:
  name: fulentd-fluentd-forwarder-cm
  namespace: efk-stack
  uid: e985af21-c736-487e-aaea-55a06f4d4748
  resourceVersion: '301766'
  creationTimestamp: '2023-03-29T18:06:22Z'
  labels:
    app.kubernetes.io/component: forwarder
    app.kubernetes.io/instance: fulentd
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: fluentd
    helm.sh/chart: fluentd-5.6.1
  annotations:
    meta.helm.sh/release-name: fulentd
    meta.helm.sh/release-namespace: efk-stack
  managedFields:
    - manager: helm
      operation: Update
      apiVersion: v1
      time: '2023-03-29T18:06:22Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:data':
          .: {}
          'f:metrics.conf': {}
        'f:metadata':
          'f:annotations':
            .: {}
            'f:meta.helm.sh/release-name': {}
            'f:meta.helm.sh/release-namespace': {}
          'f:labels':
            .: {}
            'f:app.kubernetes.io/component': {}
            'f:app.kubernetes.io/instance': {}
            'f:app.kubernetes.io/managed-by': {}
            'f:app.kubernetes.io/name': {}
            'f:helm.sh/chart': {}
    - manager: dashboard
      operation: Update
      apiVersion: v1
      time: '2023-03-30T01:31:41Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:data':
          'f:fluentd-inputs.conf': {}
          'f:fluentd-output.conf': {}
          'f:fluentd.conf': {}
data:
  fluentd-inputs.conf: |
    # HTTP input for the liveness and readiness probes
    <source>
      @type http
      port 9880
    </source>
    # Get the logs from the containers running in the node
    <source>
      @type tail
      path /var/log/containers/*-app*.log
      pos_file /opt/bitnami/fluentd/logs/buffers/fluentd-docker.pos
      tag kubernetes.*
      read_from_head true
      format json
    </source>
    # enrich with kubernetes metadata
    <filter kubernetes.**>
      @type kubernetes_metadata
    </filter>
  fluentd-output.conf: |
    # Throw the healthcheck to the standard output instead of forwarding it
    <match fluentd.healthcheck>
      @type null
    </match>

    <match kubernetes.var.log.containers.**java-app**.log>
      @type elasticsearch
      include_tag_key true
      host "elasticsearch-master.efk-stack.svc.cluster.local"
      port "9200"
      user elastic
      password gaGDRLWtsJ54INuo
      scheme https
      ssl_verify false
      index_name java-app-logs
      <buffer>
        @type file
        path /opt/bitnami/fluentd/logs/buffers/java-logs.buffer
        flush_thread_count 2
        flush_interval 5s
      </buffer>
    </match>
    <match kubernetes.var.log.containers.**node-app**.log>
      @type elasticsearch
      include_tag_key true
      host "elasticsearch-master.efk-stack.svc.cluster.local"
      port "9200"
      user elastic
      password gaGDRLWtsJ54INuo
      scheme https
      ssl_verify false
      index_name "node-app-logs"
      <buffer>
        @type file
        path /opt/bitnami/fluentd/logs/buffers/node-logs.buffer
        flush_thread_count 2
        flush_interval 5s
      </buffer>
    </match>
  fluentd.conf: |
    # Ignore fluentd own events
    <match fluent.**>
      @type null
    </match>

    @include fluentd-inputs.conf
    @include fluentd-output.conf
  metrics.conf: |
    # Prometheus Exporter Plugin
    # input plugin that exports metrics
    <source>
      @type prometheus
      port 24231
    </source>
    # input plugin that collects metrics from MonitorAgent
    <source>
      @type prometheus_monitor
      <labels>
        host ${hostname}
      </labels>
    </source>
    # input plugin that collects metrics for output plugin
    <source>
      @type prometheus_output_monitor
      <labels>
        host ${hostname}
      </labels>
    </source>
    # input plugin that collects metrics for in_tail plugin
    <source>
      @type prometheus_tail_monitor
      <labels>
        host ${hostname}
      </labels>
    </source>

Can you help please?

2

We aren't fluentd experts here, so we may not be able to help.

However, what do the fluentd logs show?

3

Let me share the logs from the three pods I have

e[38;5;6mfluentd e[38;5;5m01:32:29.23 e[0m
e[38;5;6mfluentd e[38;5;5m01:32:29.23 e[0me[1mWelcome to the Bitnami fluentd containere[0m
e[38;5;6mfluentd e[38;5;5m01:32:29.23 e[0mSubscribe to project updates by watching e[1mhttps://github.com/bitnami/containerse[0m
e[38;5;6mfluentd e[38;5;5m01:32:29.23 e[0mSubmit issues and feature requests at e[1mhttps://github.com/bitnami/containers/issuese[0m
e[38;5;6mfluentd e[38;5;5m01:32:29.24 e[0m
e[38;5;6mfluentd e[38;5;5m01:32:29.24 e[0me[38;5;2mINFO e[0m ==> ** Starting Fluentd setup **
find: '/docker-entrypoint-initdb.d/': No such file or directory
e[38;5;6mfluentd e[38;5;5m01:32:29.24 e[0me[38;5;2mINFO e[0m ==> No custom scripts in /docker-entrypoint-initdb.d
e[38;5;6mfluentd e[38;5;5m01:32:29.25 e[0me[38;5;2mINFO e[0m ==> ** Fluentd setup finished! **

e[38;5;6mfluentd e[38;5;5m01:32:29.25 e[0me[38;5;2mINFO e[0m ==> ** Starting Fluentd **
2023-03-30 01:32:30 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-03-30 01:32:30 +0000 [info]: parsing config file is succeeded path="/opt/bitnami/fluentd/conf/fluentd.conf"
2023-03-30 01:32:31 +0000 [info]: gem 'fluentd' version '1.15.3'
2023-03-30 01:32:31 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2023-03-30 01:32:31 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.14'
2023-03-30 01:32:31 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.2.4'
2023-03-30 01:32:31 +0000 [info]: gem 'fluent-plugin-grafana-loki' version '1.2.20'
2023-03-30 01:32:31 +0000 [info]: gem 'fluent-plugin-kafka' version '0.18.1'
2023-03-30 01:32:31 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '3.1.3'
2023-03-30 01:32:31 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2023-03-30 01:32:31 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.3'
2023-03-30 01:32:31 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2023-03-30 01:32:31 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2023-03-30 01:32:31 +0000 [info]: gem 'fluent-plugin-s3' version '1.7.2'
2023-03-30 01:32:31 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2023-03-30 01:32:32 +0000 [warn]: !! The environment variable 'K8S_NODE_NAME' is not set to the node name which can affect the API server and watch efficiency !!
2023-03-30 01:32:33 +0000 [warn]: define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2023-03-30 01:32:33 +0000 [info]: using configuration file: <ROOT>
  <match fluent.**>
    @type null
  </match>
  <source>
    @type http
    port 9880
  </source>
  <source>
    @type tail
    path "/var/log/containers/*-app*.log"
    pos_file "/opt/bitnami/fluentd/logs/buffers/fluentd-docker.pos"
    tag "kubernetes.*"
    read_from_head true
    format json
    <parse>
      @type json
      unmatched_lines 
    </parse>
  </source>
  <filter kubernetes.**>
    @type kubernetes_metadata
  </filter>
  <match fluentd.healthcheck>
    @type null
  </match>
  <match kubernetes.var.log.containers.**java-app**.log>
    @type elasticsearch
    include_tag_key true
    host "elasticsearch-master.efk-stack.svc.cluster.local"
    port 9200
    user "elastic"
    password xxxxxx
    scheme https
    ssl_verify false
    index_name "java-app-logs"
    <buffer>
      @type "file"
      path "/opt/bitnami/fluentd/logs/buffers/java-logs.buffer"
      flush_thread_count 2
      flush_interval 5s
    </buffer>
  </match>
  <match kubernetes.var.log.containers.**node-app**.log>
    @type elasticsearch
    include_tag_key true
    host "elasticsearch-master.efk-stack.svc.cluster.local"
    port 9200
    user "elastic"
    password xxxxxx
    scheme https
    ssl_verify false
    index_name "node-app-logs"
    <buffer>
      @type "file"
      path "/opt/bitnami/fluentd/logs/buffers/node-logs.buffer"
      flush_thread_count 2
      flush_interval 5s
    </buffer>
  </match>
</ROOT>
2023-03-30 01:32:33 +0000 [info]: starting fluentd-1.15.3 pid=1 ruby="3.1.3"
2023-03-30 01:32:33 +0000 [info]: spawn command to main:  cmdline=["/opt/bitnami/ruby/bin/ruby", "-Eascii-8bit:ascii-8bit", "/opt/bitnami/fluentd/bin/fluentd", "--config", "/opt/bitnami/fluentd/conf/fluentd.conf", "--plugin", "/opt/bitnami/fluentd/plugins", "--under-supervisor"]
2023-03-30 01:32:33 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-03-30 01:32:34 +0000 [info]: #0 init worker0 logger path=nil rotate_age=nil rotate_size=nil
2023-03-30 01:32:34 +0000 [info]: adding match pattern="fluent.**" type="null"
2023-03-30 01:32:35 +0000 [info]: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
2023-03-30 01:32:35 +0000 [warn]: #0 !! The environment variable 'K8S_NODE_NAME' is not set to the node name which can affect the API server and watch efficiency !!
2023-03-30 01:32:35 +0000 [info]: adding match pattern="fluentd.healthcheck" type="null"
2023-03-30 01:32:36 +0000 [info]: adding match pattern="kubernetes.var.log.containers.**java-app**.log" type="elasticsearch"
2023-03-30 01:32:36 +0000 [info]: adding match pattern="kubernetes.var.log.containers.**node-app**.log" type="elasticsearch"
2023-03-30 01:32:36 +0000 [info]: adding source type="http"
2023-03-30 01:32:37 +0000 [info]: adding source type="tail"
2023-03-30 01:32:37 +0000 [warn]: #0 define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2023-03-30 01:32:37 +0000 [info]: #0 starting fluentd worker pid=34 ppid=1 worker=0
2023-03-30 01:32:37 +0000 [info]: #0 following tail of /var/log/containers/java-app-549d7577fb-9fsbx_efk-stack_java-app-dd19de943b0669f0f5f6f313f315c9cc6402687a21a8cd5cb5178b12c4fe9597.log
2023-03-30 01:32:37 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:58:55.953726094Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-29T06:58:55.917+00:00\",\"logger\":\"com.example.App\",\"level\":\"INFO\",\"class\":\"org.springframework.boot.StartupInfoLogger\",\"method\":\"logStarting\",\"file\":\"StartupInfoLogger.java\",\"line\":55,\"thread\":\"main\",\"msg\":\"Starting App on java-app-549d7577fb-9fsbx with PID 1 (/usr/app/java-app-1.0-SNAPSHOT.jar started by root in /usr/app)\"}"
2023-03-30 01:32:37 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:58:55.960623084Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-29T06:58:55.960+00:00\",\"logger\":\"com.example.App\",\"level\":\"INFO\",\"class\":\"org.springframework.boot.SpringApplication\",\"method\":\"logStartupProfileInfo\",\"file\":\"SpringApplication.java\",\"line\":651,\"thread\":\"main\",\"msg\":\"No active profile set, falling back to default profiles: default\"}"
2023-03-30 01:32:37 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:58:57.567979296Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-29T06:58:57.567+00:00\",\"logger\":\"org.apache.coyote.http11.Http11NioProtocol\",\"level\":\"INFO\",\"class\":\"org.apache.juli.logging.DirectJDKLog\",\"method\":\"log\",\"file\":\"DirectJDKLog.java\",\"line\":173,\"thread\":\"main\",\"msg\":\"Initializing ProtocolHandler [\\\"http-nio-8080\\\"]\"}"
2023-03-30 01:32:37 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:58:57.569236491Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-29T06:58:57.568+00:00\",\"logger\":\"org.apache.catalina.core.StandardService\",\"level\":\"INFO\",\"class\":\"org.apache.juli.logging.DirectJDKLog\",\"method\":\"log\",\"file\":\"DirectJDKLog.java\",\"line\":173,\"thread\":\"main\",\"msg\":\"Starting service [Tomcat]\"}"
2023-03-30 01:32:37 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:58:57.56993596Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-29T06:58:57.569+00:00\",\"logger\":\"org.apache.catalina.core.StandardEngine\",\"level\":\"INFO\",\"class\":\"org.apache.juli.logging.DirectJDKLog\",\"method\":\"log\",\"file\":\"DirectJDKLog.java\",\"line\":173,\"thread\":\"main\",\"msg\":\"Starting Servlet engine: [Apache Tomcat/9.0.29]\"}"
2023-03-30 01:32:37 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:58:57.663176606Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-29T06:58:57.662+00:00\",\"logger\":\"org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/]\",\"level\":\"INFO\",\"class\":\"org.apache.juli.logging.DirectJDKLog\",\"method\":\"log\",\"file\":\"DirectJDKLog.java\",\"line\":173,\"thread\":\"main\",\"msg\":\"Initializing Spring embedded WebApplicationContext\"}"
2023-03-30 01:32:37 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:58:57.732736979Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-29T06:58:57.732+00:00\",\"logger\":\"com.example.App\",\"level\":\"INFO\",\"class\":\"com.example.App\",\"method\":\"init\",\"file\":\"App.java\",\"line\":23,\"thread\":\"main\",\"msg\":\"I am a Java app\"}"
2023-03-30 01:32:37 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:58:57.73312415Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-29T06:58:57.732+00:00\",\"logger\":\"com.example.App\",\"level\":\"INFO\",\"class\":\"com.example.App\",\"method\":\"init\",\"file\":\"App.java\",\"line\":24,\"thread\":\"main\",\"msg\":\"Just logging stuff\"}"
2023-03-30 01:32:37 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:58:57.737325255Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-29T06:58:57.735+00:00\",\"logger\":\"com.example.App\",\"level\":\"ERROR\",\"class\":\"com.example.App\",\"method\":\"init\",\"file\":\"App.java\",\"line\":29,\"thread\":\"main\",\"stack\":\"java.lang.NullPointerException: Ooh noes! Something bad happened\\n\\tat com.example.App.init(App.java:26)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\\n\\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\\n\\tat java.lang.reflect.Method.invoke(Method.java:498)\\n\\tat org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:389)\\n\\tat org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:333)\\n\\tat org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:157)\\n\\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:416)\\n\\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1788)\\n\\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:595)\\n\\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517)\\n\\tat org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323)\\n\\tat org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)\\n\\tat org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321)\\n\\tat org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)\\n\\tat org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:879)\\n\\tat org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:878)\\n\\tat org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)\\n\\tat org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141)\\n\\tat org.springframework.boot.SpringApplication.refresh(SpringApplication.java:747)\\n\\tat org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)\\n\\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:315)\\n\\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)\\n\\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:1215)\\n\\tat com.example.App.main(App.java:14)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\\n\\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\\n\\tat java.lang.reflect.Method.invoke(Method.java:498)\\n\\tat org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)\\n\\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:87)\\n\\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:51)\\n\\tat org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:52)\\n\",\"msg\":\"Error occurred!\"}"
2023-03-30 01:32:37 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:58:58.128968328Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-29T06:58:58.128+00:00\",\"logger\":\"org.apache.coyote.http11.Http11NioProtocol\",\"level\":\"INFO\",\"class\":\"org.apache.juli.logging.DirectJDKLog\",\"method\":\"log\",\"file\":\"DirectJDKLog.java\",\"line\":173,\"thread\":\"main\",\"msg\":\"Starting ProtocolHandler [\\\"http-nio-8080\\\"]\"}"
2023-03-30 01:32:37 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:58:58.162547261Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-29T06:58:58.162+00:00\",\"logger\":\"com.example.App\",\"level\":\"INFO\",\"class\":\"org.springframework.boot.StartupInfoLogger\",\"method\":\"logStarted\",\"file\":\"StartupInfoLogger.java\",\"line\":61,\"thread\":\"main\",\"msg\":\"Started App in 3.037 seconds (JVM running for 4.196)\"}"
2023-03-30 01:32:37 +0000 [info]: #0 following tail of /var/log/containers/node-app-5674575ddc-b8nsj_default_node-app-7fda84f4efe1999ea0646602cb3ddb344b5d51bcf1d3d78b4b750bc3ba62cf03.log
2023-03-30 01:32:37 +0000 [warn]: #0 pattern not matched: "2023-03-27T21:04:37.343184371Z stdout F {\"level\":30,\"time\":\"2023-03-27T21:04:37.342Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-b8nsj\",\"msg\":\"hello elastic world\"}"
2023-03-30 01:32:37 +0000 [warn]: #0 pattern not matched: "2023-03-27T21:04:37.343369806Z stdout F {\"level\":30,\"time\":\"2023-03-27T21:04:37.343Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-b8nsj\",\"msg\":\"This is some great stuff\"}"
2023-03-30 01:32:37 +0000 [warn]: #0 pattern not matched: "2023-03-27T21:04:37.343416778Z stdout F {\"level\":30,\"time\":\"2023-03-27T21:04:37.343Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-b8nsj\",\"msg\":\"Some more entries for our logging\"}"
2023-03-30 01:32:37 +0000 [warn]: #0 pattern not matched: "2023-03-27T21:04:37.343456669Z stdout F {\"level\":30,\"time\":\"2023-03-27T21:04:37.343Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-b8nsj\",\"msg\":\"another line\"}"
2023-03-30 01:32:37 +0000 [warn]: #0 pattern not matched: "2023-03-27T21:04:37.343543261Z stdout F {\"level\":30,\"time\":\"2023-03-27T21:04:37.343Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-b8nsj\",\"msg\":\"This never stops\"}"
2023-03-30 01:32:37 +0000 [warn]: #0 pattern not matched: "2023-03-27T21:04:37.343583712Z stdout F {\"level\":30,\"time\":\"2023-03-27T21:04:37.343Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-b8nsj\",\"msg\":\"Logging logging all the way\"}"
2023-03-30 01:32:37 +0000 [warn]: #0 pattern not matched: "2023-03-27T21:04:37.343622373Z stdout F {\"level\":30,\"time\":\"2023-03-27T21:04:37.343Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-b8nsj\",\"msg\":\"I think this is enough\"}"
2023-03-30 01:32:37 +0000 [warn]: #0 pattern not matched: "2023-03-27T21:04:37.343660364Z stdout F {\"level\":30,\"time\":\"2023-03-27T21:04:37.343Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-b8nsj\",\"msg\":\"nope, one more!\"}"
2023-03-30 01:32:37 +0000 [warn]: #0 pattern not matched: "2023-03-27T21:04:37.346210555Z stdout F {\"level\":30,\"time\":\"2023-03-27T21:04:37.346Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-b8nsj\",\"msg\":\"app listening on port 3000!\"}"
2023-03-30 01:32:37 +0000 [info]: #0 fluentd worker is now running worker=0
4

2nd

e[38;5;6mfluentd e[38;5;5m01:31:52.63 e[0m
e[38;5;6mfluentd e[38;5;5m01:31:52.64 e[0me[1mWelcome to the Bitnami fluentd containere[0m
e[38;5;6mfluentd e[38;5;5m01:31:52.64 e[0mSubscribe to project updates by watching e[1mhttps://github.com/bitnami/containerse[0m
e[38;5;6mfluentd e[38;5;5m01:31:52.64 e[0mSubmit issues and feature requests at e[1mhttps://github.com/bitnami/containers/issuese[0m
e[38;5;6mfluentd e[38;5;5m01:31:52.64 e[0m
e[38;5;6mfluentd e[38;5;5m01:31:52.64 e[0me[38;5;2mINFO e[0m ==> ** Starting Fluentd setup **
find: '/docker-entrypoint-initdb.d/': No such file or directory
e[38;5;6mfluentd e[38;5;5m01:31:52.66 e[0me[38;5;2mINFO e[0m ==> No custom scripts in /docker-entrypoint-initdb.d
e[38;5;6mfluentd e[38;5;5m01:31:52.66 e[0me[38;5;2mINFO e[0m ==> ** Fluentd setup finished! **

e[38;5;6mfluentd e[38;5;5m01:31:52.67 e[0me[38;5;2mINFO e[0m ==> ** Starting Fluentd **
2023-03-30 01:31:55 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-03-30 01:31:55 +0000 [info]: parsing config file is succeeded path="/opt/bitnami/fluentd/conf/fluentd.conf"
2023-03-30 01:31:55 +0000 [info]: gem 'fluentd' version '1.15.3'
2023-03-30 01:31:55 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2023-03-30 01:31:55 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.14'
2023-03-30 01:31:55 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.2.4'
2023-03-30 01:31:55 +0000 [info]: gem 'fluent-plugin-grafana-loki' version '1.2.20'
2023-03-30 01:31:55 +0000 [info]: gem 'fluent-plugin-kafka' version '0.18.1'
2023-03-30 01:31:55 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '3.1.3'
2023-03-30 01:31:55 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2023-03-30 01:31:55 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.3'
2023-03-30 01:31:55 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2023-03-30 01:31:55 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2023-03-30 01:31:55 +0000 [info]: gem 'fluent-plugin-s3' version '1.7.2'
2023-03-30 01:31:55 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2023-03-30 01:31:57 +0000 [warn]: !! The environment variable 'K8S_NODE_NAME' is not set to the node name which can affect the API server and watch efficiency !!
2023-03-30 01:31:59 +0000 [warn]: define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2023-03-30 01:31:59 +0000 [info]: using configuration file: <ROOT>
  <match fluent.**>
    @type null
  </match>
  <source>
    @type http
    port 9880
  </source>
  <source>
    @type tail
    path "/var/log/containers/*-app*.log"
    pos_file "/opt/bitnami/fluentd/logs/buffers/fluentd-docker.pos"
    tag "kubernetes.*"
    read_from_head true
    format json
    <parse>
      @type json
      unmatched_lines 
    </parse>
  </source>
  <filter kubernetes.**>
    @type kubernetes_metadata
  </filter>
  <match fluentd.healthcheck>
    @type null
  </match>
  <match kubernetes.var.log.containers.**java-app**.log>
    @type elasticsearch
    include_tag_key true
    host "elasticsearch-master.efk-stack.svc.cluster.local"
    port 9200
    user "elastic"
    password xxxxxx
    scheme https
    ssl_verify false
    index_name "java-app-logs"
    <buffer>
      @type "file"
      path "/opt/bitnami/fluentd/logs/buffers/java-logs.buffer"
      flush_thread_count 2
      flush_interval 5s
    </buffer>
  </match>
  <match kubernetes.var.log.containers.**node-app**.log>
    @type elasticsearch
    include_tag_key true
    host "elasticsearch-master.efk-stack.svc.cluster.local"
    port 9200
    user "elastic"
    password xxxxxx
    scheme https
    ssl_verify false
    index_name "node-app-logs"
    <buffer>
      @type "file"
      path "/opt/bitnami/fluentd/logs/buffers/node-logs.buffer"
      flush_thread_count 2
      flush_interval 5s
    </buffer>
  </match>
</ROOT>
2023-03-30 01:31:59 +0000 [info]: starting fluentd-1.15.3 pid=1 ruby="3.1.3"
2023-03-30 01:31:59 +0000 [info]: spawn command to main:  cmdline=["/opt/bitnami/ruby/bin/ruby", "-Eascii-8bit:ascii-8bit", "/opt/bitnami/fluentd/bin/fluentd", "--config", "/opt/bitnami/fluentd/conf/fluentd.conf", "--plugin", "/opt/bitnami/fluentd/plugins", "--under-supervisor"]
2023-03-30 01:31:59 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-03-30 01:32:01 +0000 [info]: #0 init worker0 logger path=nil rotate_age=nil rotate_size=nil
2023-03-30 01:32:01 +0000 [info]: adding match pattern="fluent.**" type="null"
2023-03-30 01:32:01 +0000 [info]: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
2023-03-30 01:32:03 +0000 [warn]: #0 !! The environment variable 'K8S_NODE_NAME' is not set to the node name which can affect the API server and watch efficiency !!
2023-03-30 01:32:03 +0000 [info]: adding match pattern="fluentd.healthcheck" type="null"
2023-03-30 01:32:03 +0000 [info]: adding match pattern="kubernetes.var.log.containers.**java-app**.log" type="elasticsearch"
2023-03-30 01:32:04 +0000 [info]: adding match pattern="kubernetes.var.log.containers.**node-app**.log" type="elasticsearch"
2023-03-30 01:32:04 +0000 [info]: adding source type="http"
2023-03-30 01:32:04 +0000 [info]: adding source type="tail"
2023-03-30 01:32:04 +0000 [warn]: #0 define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2023-03-30 01:32:04 +0000 [info]: #0 starting fluentd worker pid=34 ppid=1 worker=0
2023-03-30 01:32:04 +0000 [info]: #0 following tail of /var/log/containers/java-app-549d7577fb-rfsxn_default_java-app-a03446ef06427d29774151ae62cdbad65c09eb9f80d862d383934fa860c35b5f.log
2023-03-30 01:32:04 +0000 [warn]: #0 pattern not matched: "2023-03-27T19:44:51.347839054Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-27T19:44:51.305+00:00\",\"logger\":\"com.example.App\",\"level\":\"INFO\",\"class\":\"org.springframework.boot.StartupInfoLogger\",\"method\":\"logStarting\",\"file\":\"StartupInfoLogger.java\",\"line\":55,\"thread\":\"main\",\"msg\":\"Starting App on java-app-549d7577fb-rfsxn with PID 1 (/usr/app/java-app-1.0-SNAPSHOT.jar started by root in /usr/app)\"}"
2023-03-30 01:32:04 +0000 [warn]: #0 pattern not matched: "2023-03-27T19:44:51.350653657Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-27T19:44:51.350+00:00\",\"logger\":\"com.example.App\",\"level\":\"INFO\",\"class\":\"org.springframework.boot.SpringApplication\",\"method\":\"logStartupProfileInfo\",\"file\":\"SpringApplication.java\",\"line\":651,\"thread\":\"main\",\"msg\":\"No active profile set, falling back to default profiles: default\"}"
2023-03-30 01:32:04 +0000 [warn]: #0 pattern not matched: "2023-03-27T19:44:53.747954226Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-27T19:44:53.741+00:00\",\"logger\":\"org.apache.coyote.http11.Http11NioProtocol\",\"level\":\"INFO\",\"class\":\"org.apache.juli.logging.DirectJDKLog\",\"method\":\"log\",\"file\":\"DirectJDKLog.java\",\"line\":173,\"thread\":\"main\",\"msg\":\"Initializing ProtocolHandler [\\\"http-nio-8080\\\"]\"}"
2023-03-30 01:32:04 +0000 [warn]: #0 pattern not matched: "2023-03-27T19:44:53.761271045Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-27T19:44:53.760+00:00\",\"logger\":\"org.apache.catalina.core.StandardService\",\"level\":\"INFO\",\"class\":\"org.apache.juli.logging.DirectJDKLog\",\"method\":\"log\",\"file\":\"DirectJDKLog.java\",\"line\":173,\"thread\":\"main\",\"msg\":\"Starting service [Tomcat]\"}"
2023-03-30 01:32:04 +0000 [warn]: #0 pattern not matched: "2023-03-27T19:44:53.766214777Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-27T19:44:53.765+00:00\",\"logger\":\"org.apache.catalina.core.StandardEngine\",\"level\":\"INFO\",\"class\":\"org.apache.juli.logging.DirectJDKLog\",\"method\":\"log\",\"file\":\"DirectJDKLog.java\",\"line\":173,\"thread\":\"main\",\"msg\":\"Starting Servlet engine: [Apache Tomcat/9.0.29]\"}"
2023-03-30 01:32:04 +0000 [warn]: #0 pattern not matched: "2023-03-27T19:44:53.95874059Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-27T19:44:53.957+00:00\",\"logger\":\"org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/]\",\"level\":\"INFO\",\"class\":\"org.apache.juli.logging.DirectJDKLog\",\"method\":\"log\",\"file\":\"DirectJDKLog.java\",\"line\":173,\"thread\":\"main\",\"msg\":\"Initializing Spring embedded WebApplicationContext\"}"
2023-03-30 01:32:04 +0000 [warn]: #0 pattern not matched: "2023-03-27T19:44:54.068131426Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-27T19:44:54.066+00:00\",\"logger\":\"com.example.App\",\"level\":\"INFO\",\"class\":\"com.example.App\",\"method\":\"init\",\"file\":\"App.java\",\"line\":23,\"thread\":\"main\",\"msg\":\"I am a Java app\"}"
2023-03-30 01:32:04 +0000 [warn]: #0 pattern not matched: "2023-03-27T19:44:54.068643678Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-27T19:44:54.068+00:00\",\"logger\":\"com.example.App\",\"level\":\"INFO\",\"class\":\"com.example.App\",\"method\":\"init\",\"file\":\"App.java\",\"line\":24,\"thread\":\"main\",\"msg\":\"Just logging stuff\"}"
2023-03-30 01:32:04 +0000 [warn]: #0 pattern not matched: "2023-03-27T19:44:54.074229843Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-27T19:44:54.071+00:00\",\"logger\":\"com.example.App\",\"level\":\"ERROR\",\"class\":\"com.example.App\",\"method\":\"init\",\"file\":\"App.java\",\"line\":29,\"thread\":\"main\",\"stack\":\"java.lang.NullPointerException: Ooh noes! Something bad happened\\n\\tat com.example.App.init(App.java:26)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\\n\\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\\n\\tat java.lang.reflect.Method.invoke(Method.java:498)\\n\\tat org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:389)\\n\\tat org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:333)\\n\\tat org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:157)\\n\\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:416)\\n\\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1788)\\n\\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:595)\\n\\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517)\\n\\tat org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323)\\n\\tat org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)\\n\\tat org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321)\\n\\tat org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)\\n\\tat org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:879)\\n\\tat org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:878)\\n\\tat org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)\\n\\tat org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141)\\n\\tat org.springframework.boot.SpringApplication.refresh(SpringApplication.java:747)\\n\\tat org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)\\n\\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:315)\\n\\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)\\n\\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:1215)\\n\\tat com.example.App.main(App.java:14)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\\n\\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\\n\\tat java.lang.reflect.Method.invoke(Method.java:498)\\n\\tat org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)\\n\\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:87)\\n\\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:51)\\n\\tat org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:52)\\n\",\"msg\":\"Error occurred!\"}"
2023-03-30 01:32:04 +0000 [warn]: #0 pattern not matched: "2023-03-27T19:44:54.585778441Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-27T19:44:54.585+00:00\",\"logger\":\"org.apache.coyote.http11.Http11NioProtocol\",\"level\":\"INFO\",\"class\":\"org.apache.juli.logging.DirectJDKLog\",\"method\":\"log\",\"file\":\"DirectJDKLog.java\",\"line\":173,\"thread\":\"main\",\"msg\":\"Starting ProtocolHandler [\\\"http-nio-8080\\\"]\"}"
2023-03-30 01:32:04 +0000 [warn]: #0 pattern not matched: "2023-03-27T19:44:54.638871056Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-27T19:44:54.638+00:00\",\"logger\":\"com.example.App\",\"level\":\"INFO\",\"class\":\"org.springframework.boot.StartupInfoLogger\",\"method\":\"logStarted\",\"file\":\"StartupInfoLogger.java\",\"line\":61,\"thread\":\"main\",\"msg\":\"Started App in 4.378 seconds (JVM running for 5.867)\"}"
2023-03-30 01:32:04 +0000 [info]: #0 following tail of /var/log/containers/node-app-5674575ddc-nqkqp_efk-stack_node-app-f3ee862f3f880734dfb747dfecb065b3450149a565a28a3340dfed3b4ef8a638.log
2023-03-30 01:32:04 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:59:26.486536084Z stdout F {\"level\":30,\"time\":\"2023-03-29T06:59:26.485Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-nqkqp\",\"msg\":\"hello elastic world\"}"
2023-03-30 01:32:04 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:59:26.486664535Z stdout F {\"level\":30,\"time\":\"2023-03-29T06:59:26.486Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-nqkqp\",\"msg\":\"This is some great stuff\"}"
2023-03-30 01:32:04 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:59:26.486675165Z stdout F {\"level\":30,\"time\":\"2023-03-29T06:59:26.486Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-nqkqp\",\"msg\":\"Some more entries for our logging\"}"
2023-03-30 01:32:04 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:59:26.486738685Z stdout F {\"level\":30,\"time\":\"2023-03-29T06:59:26.486Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-nqkqp\",\"msg\":\"another line\"}"
2023-03-30 01:32:04 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:59:26.486801365Z stdout F {\"level\":30,\"time\":\"2023-03-29T06:59:26.486Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-nqkqp\",\"msg\":\"This never stops\"}"
2023-03-30 01:32:04 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:59:26.486810545Z stdout F {\"level\":30,\"time\":\"2023-03-29T06:59:26.486Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-nqkqp\",\"msg\":\"Logging logging all the way\"}"
2023-03-30 01:32:04 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:59:26.486873885Z stdout F {\"level\":30,\"time\":\"2023-03-29T06:59:26.486Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-nqkqp\",\"msg\":\"I think this is enough\"}"
2023-03-30 01:32:04 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:59:26.486883896Z stdout F {\"level\":30,\"time\":\"2023-03-29T06:59:26.486Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-nqkqp\",\"msg\":\"nope, one more!\"}"
2023-03-30 01:32:04 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:59:26.491062815Z stdout F {\"level\":30,\"time\":\"2023-03-29T06:59:26.490Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-nqkqp\",\"msg\":\"app listening on port 3000!\"}"
2023-03-30 01:32:04 +0000 [info]: #0 fluentd worker is now running worker=0
5

3rd

e[38;5;6mfluentd e[38;5;5m01:32:16.40 e[0m
e[38;5;6mfluentd e[38;5;5m01:32:16.40 e[0me[1mWelcome to the Bitnami fluentd containere[0m
e[38;5;6mfluentd e[38;5;5m01:32:16.40 e[0mSubscribe to project updates by watching e[1mhttps://github.com/bitnami/containerse[0m
e[38;5;6mfluentd e[38;5;5m01:32:16.41 e[0mSubmit issues and feature requests at e[1mhttps://github.com/bitnami/containers/issuese[0m
e[38;5;6mfluentd e[38;5;5m01:32:16.41 e[0m
e[38;5;6mfluentd e[38;5;5m01:32:16.41 e[0me[38;5;2mINFO e[0m ==> ** Starting Fluentd setup **
find: '/docker-entrypoint-initdb.d/': No such file or directory
e[38;5;6mfluentd e[38;5;5m01:32:16.42 e[0me[38;5;2mINFO e[0m ==> No custom scripts in /docker-entrypoint-initdb.d
e[38;5;6mfluentd e[38;5;5m01:32:16.42 e[0me[38;5;2mINFO e[0m ==> ** Fluentd setup finished! **

e[38;5;6mfluentd e[38;5;5m01:32:16.42 e[0me[38;5;2mINFO e[0m ==> ** Starting Fluentd **
2023-03-30 01:32:17 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-03-30 01:32:18 +0000 [info]: parsing config file is succeeded path="/opt/bitnami/fluentd/conf/fluentd.conf"
2023-03-30 01:32:18 +0000 [info]: gem 'fluentd' version '1.15.3'
2023-03-30 01:32:18 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2023-03-30 01:32:18 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.14'
2023-03-30 01:32:18 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.2.4'
2023-03-30 01:32:18 +0000 [info]: gem 'fluent-plugin-grafana-loki' version '1.2.20'
2023-03-30 01:32:18 +0000 [info]: gem 'fluent-plugin-kafka' version '0.18.1'
2023-03-30 01:32:18 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '3.1.3'
2023-03-30 01:32:18 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2023-03-30 01:32:18 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.3'
2023-03-30 01:32:18 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2023-03-30 01:32:18 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2023-03-30 01:32:18 +0000 [info]: gem 'fluent-plugin-s3' version '1.7.2'
2023-03-30 01:32:18 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2023-03-30 01:32:19 +0000 [warn]: !! The environment variable 'K8S_NODE_NAME' is not set to the node name which can affect the API server and watch efficiency !!
2023-03-30 01:32:20 +0000 [warn]: define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2023-03-30 01:32:20 +0000 [info]: using configuration file: <ROOT>
  <match fluent.**>
    @type null
  </match>
  <source>
    @type http
    port 9880
  </source>
  <source>
    @type tail
    path "/var/log/containers/*-app*.log"
    pos_file "/opt/bitnami/fluentd/logs/buffers/fluentd-docker.pos"
    tag "kubernetes.*"
    read_from_head true
    format json
    <parse>
      @type json
      unmatched_lines 
    </parse>
  </source>
  <filter kubernetes.**>
    @type kubernetes_metadata
  </filter>
  <match fluentd.healthcheck>
    @type null
  </match>
  <match kubernetes.var.log.containers.**java-app**.log>
    @type elasticsearch
    include_tag_key true
    host "elasticsearch-master.efk-stack.svc.cluster.local"
    port 9200
    user "elastic"
    password xxxxxx
    scheme https
    ssl_verify false
    index_name "java-app-logs"
    <buffer>
      @type "file"
      path "/opt/bitnami/fluentd/logs/buffers/java-logs.buffer"
      flush_thread_count 2
      flush_interval 5s
    </buffer>
  </match>
  <match kubernetes.var.log.containers.**node-app**.log>
    @type elasticsearch
    include_tag_key true
    host "elasticsearch-master.efk-stack.svc.cluster.local"
    port 9200
    user "elastic"
    password xxxxxx
    scheme https
    ssl_verify false
    index_name "node-app-logs"
    <buffer>
      @type "file"
      path "/opt/bitnami/fluentd/logs/buffers/node-logs.buffer"
      flush_thread_count 2
      flush_interval 5s
    </buffer>
  </match>
</ROOT>
2023-03-30 01:32:20 +0000 [info]: starting fluentd-1.15.3 pid=1 ruby="3.1.3"
2023-03-30 01:32:20 +0000 [info]: spawn command to main:  cmdline=["/opt/bitnami/ruby/bin/ruby", "-Eascii-8bit:ascii-8bit", "/opt/bitnami/fluentd/bin/fluentd", "--config", "/opt/bitnami/fluentd/conf/fluentd.conf", "--plugin", "/opt/bitnami/fluentd/plugins", "--under-supervisor"]
2023-03-30 01:32:20 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-03-30 01:32:21 +0000 [info]: #0 init worker0 logger path=nil rotate_age=nil rotate_size=nil
2023-03-30 01:32:21 +0000 [info]: adding match pattern="fluent.**" type="null"
2023-03-30 01:32:21 +0000 [info]: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
2023-03-30 01:32:22 +0000 [warn]: #0 !! The environment variable 'K8S_NODE_NAME' is not set to the node name which can affect the API server and watch efficiency !!
2023-03-30 01:32:22 +0000 [info]: adding match pattern="fluentd.healthcheck" type="null"
2023-03-30 01:32:22 +0000 [info]: adding match pattern="kubernetes.var.log.containers.**java-app**.log" type="elasticsearch"
2023-03-30 01:32:23 +0000 [info]: adding match pattern="kubernetes.var.log.containers.**node-app**.log" type="elasticsearch"
2023-03-30 01:32:23 +0000 [info]: adding source type="http"
2023-03-30 01:32:23 +0000 [info]: adding source type="tail"
2023-03-30 01:32:23 +0000 [warn]: #0 define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2023-03-30 01:32:23 +0000 [info]: #0 starting fluentd worker pid=34 ppid=1 worker=0
2023-03-30 01:32:23 +0000 [info]: #0 fluentd worker is now running worker=0
6

Which version of Elasticsearch are you running?

7

I'm running 8.5.1, that is the latest current version of the official helm chart helm install elasticsearch

8

I also enabled @log_level debug in my fluentd ConfigMap. Let me share the logs that has more info

9

1st pod

e[38;5;6mfluentd e[38;5;5m05:12:13.92 e[0m
e[38;5;6mfluentd e[38;5;5m05:12:13.92 e[0me[1mWelcome to the Bitnami fluentd containere[0m
e[38;5;6mfluentd e[38;5;5m05:12:13.93 e[0mSubscribe to project updates by watching e[1mhttps://github.com/bitnami/containerse[0m
e[38;5;6mfluentd e[38;5;5m05:12:13.93 e[0mSubmit issues and feature requests at e[1mhttps://github.com/bitnami/containers/issuese[0m
e[38;5;6mfluentd e[38;5;5m05:12:13.93 e[0m
e[38;5;6mfluentd e[38;5;5m05:12:13.93 e[0me[38;5;2mINFO e[0m ==> ** Starting Fluentd setup **
find: '/docker-entrypoint-initdb.d/': No such file or directory
e[38;5;6mfluentd e[38;5;5m05:12:13.95 e[0me[38;5;2mINFO e[0m ==> No custom scripts in /docker-entrypoint-initdb.d
e[38;5;6mfluentd e[38;5;5m05:12:13.95 e[0me[38;5;2mINFO e[0m ==> ** Fluentd setup finished! **

e[38;5;6mfluentd e[38;5;5m05:12:13.97 e[0me[38;5;2mINFO e[0m ==> ** Starting Fluentd **
2023-03-30 05:12:16 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-03-30 05:12:16 +0000 [info]: parsing config file is succeeded path="/opt/bitnami/fluentd/conf/fluentd.conf"
2023-03-30 05:12:16 +0000 [info]: gem 'fluentd' version '1.15.3'
2023-03-30 05:12:16 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2023-03-30 05:12:16 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.14'
2023-03-30 05:12:16 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.2.4'
2023-03-30 05:12:16 +0000 [info]: gem 'fluent-plugin-grafana-loki' version '1.2.20'
2023-03-30 05:12:16 +0000 [info]: gem 'fluent-plugin-kafka' version '0.18.1'
2023-03-30 05:12:16 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '3.1.3'
2023-03-30 05:12:16 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2023-03-30 05:12:16 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.3'
2023-03-30 05:12:16 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2023-03-30 05:12:16 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2023-03-30 05:12:16 +0000 [info]: gem 'fluent-plugin-s3' version '1.7.2'
2023-03-30 05:12:16 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2023-03-30 05:12:18 +0000 [warn]: !! The environment variable 'K8S_NODE_NAME' is not set to the node name which can affect the API server and watch efficiency !!
2023-03-30 05:12:19 +0000 [debug]: [elasticsearch] 'host elasticsearch-master.efk-stack.svc.cluster.local' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: elasticsearch-master.efk-stack.svc.cluster.local' doesn't have tag placeholder
2023-03-30 05:12:19 +0000 [debug]: [elasticsearch] 'index_name java-app-logs' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: java-app-logs' doesn't have tag placeholder
2023-03-30 05:12:19 +0000 [debug]: [elasticsearch] 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2023-03-30 05:12:19 +0000 [debug]: [elasticsearch] 'logstash_prefix false' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: false' doesn't have tag placeholder
2023-03-30 05:12:19 +0000 [debug]: [elasticsearch] 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders, but chunk key 'time' is not configured
2023-03-30 05:12:19 +0000 [debug]: [elasticsearch] 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2023-03-30 05:12:19 +0000 [debug]: [elasticsearch] 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2023-03-30 05:12:19 +0000 [debug]: [elasticsearch] 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2023-03-30 05:12:19 +0000 [debug]: [elasticsearch] 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2023-03-30 05:12:19 +0000 [debug]: [elasticsearch] Need substitution: false
2023-03-30 05:12:19 +0000 [debug]: [elasticsearch] 'host_placeholder elasticsearch-master.efk-stack.svc.cluster.local' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: elasticsearch-master.efk-stack.svc.cluster.local' doesn't have tag placeholder
2023-03-30 05:12:19 +0000 [warn]: define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2023-03-30 05:12:19 +0000 [info]: using configuration file: <ROOT>
  <match fluent.**>
    @type null
  </match>
  <source>
    @type http
    port 9880
  </source>
  <source>
    @type tail
    path "/var/log/containers/*-app*.log"
    pos_file "/opt/bitnami/fluentd/logs/buffers/fluentd-docker.pos"
    tag "kubernetes.*"
    read_from_head true
    format json
    <parse>
      @type json
      unmatched_lines 
    </parse>
  </source>
  <filter kubernetes.**>
    @type kubernetes_metadata
  </filter>
  <match fluentd.healthcheck>
    @type null
  </match>
  <match **>
    @id elasticsearch
    @type elasticsearch
    @log_level "debug"
    include_tag_key true
    host "elasticsearch-master.efk-stack.svc.cluster.local"
    port 9200
    user "elastic"
    password xxxxxx
    index_name "java-app-logs"
    scheme https
    ssl_verify false
    logstash_prefix "false"
    logstash_format false
    <buffer>
      @type "file"
      path "/opt/bitnami/fluentd/logs/buffers/java-logs.buffer"
      flush_thread_count 2
      flush_interval 5s
    </buffer>
  </match>
</ROOT>
2023-03-30 05:12:19 +0000 [info]: starting fluentd-1.15.3 pid=1 ruby="3.1.3"
2023-03-30 05:12:19 +0000 [info]: spawn command to main:  cmdline=["/opt/bitnami/ruby/bin/ruby", "-Eascii-8bit:ascii-8bit", "/opt/bitnami/fluentd/bin/fluentd", "--config", "/opt/bitnami/fluentd/conf/fluentd.conf", "--plugin", "/opt/bitnami/fluentd/plugins", "--under-supervisor"]
2023-03-30 05:12:19 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-03-30 05:12:21 +0000 [info]: #0 init worker0 logger path=nil rotate_age=nil rotate_size=nil
2023-03-30 05:12:21 +0000 [info]: adding match pattern="fluent.**" type="null"
2023-03-30 05:12:22 +0000 [info]: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
2023-03-30 05:12:23 +0000 [warn]: #0 !! The environment variable 'K8S_NODE_NAME' is not set to the node name which can affect the API server and watch efficiency !!
2023-03-30 05:12:23 +0000 [info]: adding match pattern="fluentd.healthcheck" type="null"
2023-03-30 05:12:23 +0000 [info]: adding match pattern="**" type="elasticsearch"
2023-03-30 05:12:24 +0000 [debug]: #0 [elasticsearch] 'host elasticsearch-master.efk-stack.svc.cluster.local' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: elasticsearch-master.efk-stack.svc.cluster.local' doesn't have tag placeholder
2023-03-30 05:12:24 +0000 [debug]: #0 [elasticsearch] 'index_name java-app-logs' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: java-app-logs' doesn't have tag placeholder
2023-03-30 05:12:24 +0000 [debug]: #0 [elasticsearch] 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2023-03-30 05:12:24 +0000 [debug]: #0 [elasticsearch] 'logstash_prefix false' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: false' doesn't have tag placeholder
2023-03-30 05:12:24 +0000 [debug]: #0 [elasticsearch] 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders, but chunk key 'time' is not configured
2023-03-30 05:12:24 +0000 [debug]: #0 [elasticsearch] 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2023-03-30 05:12:24 +0000 [debug]: #0 [elasticsearch] 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2023-03-30 05:12:24 +0000 [debug]: #0 [elasticsearch] 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2023-03-30 05:12:24 +0000 [debug]: #0 [elasticsearch] 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2023-03-30 05:12:24 +0000 [debug]: #0 [elasticsearch] Need substitution: false
2023-03-30 05:12:24 +0000 [debug]: #0 [elasticsearch] 'host_placeholder elasticsearch-master.efk-stack.svc.cluster.local' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: elasticsearch-master.efk-stack.svc.cluster.local' doesn't have tag placeholder
2023-03-30 05:12:25 +0000 [debug]: #0 [elasticsearch] Detected ES 8.x or above: This parameter has no effect.
2023-03-30 05:12:25 +0000 [info]: adding source type="http"
2023-03-30 05:12:25 +0000 [info]: adding source type="tail"
2023-03-30 05:12:25 +0000 [warn]: #0 define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2023-03-30 05:12:25 +0000 [info]: #0 starting fluentd worker pid=34 ppid=1 worker=0
2023-03-30 05:12:25 +0000 [debug]: #0 [elasticsearch] buffer started instance=51620 stage_size=0 queue_size=0
2023-03-30 05:12:25 +0000 [debug]: #0 [elasticsearch] flush_thread actually running
2023-03-30 05:12:25 +0000 [debug]: #0 [elasticsearch] flush_thread actually running
2023-03-30 05:12:25 +0000 [debug]: #0 [elasticsearch] enqueue_thread actually running
2023-03-30 05:12:25 +0000 [info]: #0 following tail of /var/log/containers/java-app-549d7577fb-rfsxn_default_java-app-a03446ef06427d29774151ae62cdbad65c09eb9f80d862d383934fa860c35b5f.log
2023-03-30 05:12:25 +0000 [warn]: #0 pattern not matched: "2023-03-27T19:44:51.347839054Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-27T19:44:51.305+00:00\",\"logger\":\"com.example.App\",\"level\":\"INFO\",\"class\":\"org.springframework.boot.StartupInfoLogger\",\"method\":\"logStarting\",\"file\":\"StartupInfoLogger.java\",\"line\":55,\"thread\":\"main\",\"msg\":\"Starting App on java-app-549d7577fb-rfsxn with PID 1 (/usr/app/java-app-1.0-SNAPSHOT.jar started by root in /usr/app)\"}"
2023-03-30 05:12:25 +0000 [warn]: #0 pattern not matched: "2023-03-27T19:44:51.350653657Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-27T19:44:51.350+00:00\",\"logger\":\"com.example.App\",\"level\":\"INFO\",\"class\":\"org.springframework.boot.SpringApplication\",\"method\":\"logStartupProfileInfo\",\"file\":\"SpringApplication.java\",\"line\":651,\"thread\":\"main\",\"msg\":\"No active profile set, falling back to default profiles: default\"}"
2023-03-30 05:12:25 +0000 [warn]: #0 pattern not matched: "2023-03-27T19:44:53.747954226Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-27T19:44:53.741+00:00\",\"logger\":\"org.apache.coyote.http11.Http11NioProtocol\",\"level\":\"INFO\",\"class\":\"org.apache.juli.logging.DirectJDKLog\",\"method\":\"log\",\"file\":\"DirectJDKLog.java\",\"line\":173,\"thread\":\"main\",\"msg\":\"Initializing ProtocolHandler [\\\"http-nio-8080\\\"]\"}"
2023-03-30 05:12:25 +0000 [warn]: #0 pattern not matched: "2023-03-27T19:44:53.761271045Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-27T19:44:53.760+00:00\",\"logger\":\"org.apache.catalina.core.StandardService\",\"level\":\"INFO\",\"class\":\"org.apache.juli.logging.DirectJDKLog\",\"method\":\"log\",\"file\":\"DirectJDKLog.java\",\"line\":173,\"thread\":\"main\",\"msg\":\"Starting service [Tomcat]\"}"
2023-03-30 05:12:25 +0000 [warn]: #0 pattern not matched: "2023-03-27T19:44:53.766214777Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-27T19:44:53.765+00:00\",\"logger\":\"org.apache.catalina.core.StandardEngine\",\"level\":\"INFO\",\"class\":\"org.apache.juli.logging.DirectJDKLog\",\"method\":\"log\",\"file\":\"DirectJDKLog.java\",\"line\":173,\"thread\":\"main\",\"msg\":\"Starting Servlet engine: [Apache Tomcat/9.0.29]\"}"
2023-03-30 05:12:25 +0000 [warn]: #0 pattern not matched: "2023-03-27T19:44:53.95874059Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-27T19:44:53.957+00:00\",\"logger\":\"org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/]\",\"level\":\"INFO\",\"class\":\"org.apache.juli.logging.DirectJDKLog\",\"method\":\"log\",\"file\":\"DirectJDKLog.java\",\"line\":173,\"thread\":\"main\",\"msg\":\"Initializing Spring embedded WebApplicationContext\"}"
2023-03-30 05:12:25 +0000 [warn]: #0 pattern not matched: "2023-03-27T19:44:54.068131426Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-27T19:44:54.066+00:00\",\"logger\":\"com.example.App\",\"level\":\"INFO\",\"class\":\"com.example.App\",\"method\":\"init\",\"file\":\"App.java\",\"line\":23,\"thread\":\"main\",\"msg\":\"I am a Java app\"}"
2023-03-30 05:12:25 +0000 [warn]: #0 pattern not matched: "2023-03-27T19:44:54.068643678Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-27T19:44:54.068+00:00\",\"logger\":\"com.example.App\",\"level\":\"INFO\",\"class\":\"com.example.App\",\"method\":\"init\",\"file\":\"App.java\",\"line\":24,\"thread\":\"main\",\"msg\":\"Just logging stuff\"}"
2023-03-30 05:12:25 +0000 [warn]: #0 pattern not matched: "2023-03-27T19:44:54.074229843Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-27T19:44:54.071+00:00\",\"logger\":\"com.example.App\",\"level\":\"ERROR\",\"class\":\"com.example.App\",\"method\":\"init\",\"file\":\"App.java\",\"line\":29,\"thread\":\"main\",\"stack\":\"java.lang.NullPointerException: Ooh noes! Something bad happened\\n\\tat com.example.App.init(App.java:26)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\\n\\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\\n\\tat java.lang.reflect.Method.invoke(Method.java:498)\\n\\tat org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:389)\\n\\tat org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:333)\\n\\tat org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:157)\\n\\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:416)\\n\\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1788)\\n\\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:595)\\n\\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517)\\n\\tat org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323)\\n\\tat org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)\\n\\tat org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321)\\n\\tat org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)\\n\\tat org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:879)\\n\\tat org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:878)\\n\\tat org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)\\n\\tat org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141)\\n\\tat org.springframework.boot.SpringApplication.refresh(SpringApplication.java:747)\\n\\tat org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)\\n\\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:315)\\n\\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)\\n\\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:1215)\\n\\tat com.example.App.main(App.java:14)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\\n\\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\\n\\tat java.lang.reflect.Method.invoke(Method.java:498)\\n\\tat org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)\\n\\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:87)\\n\\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:51)\\n\\tat org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:52)\\n\",\"msg\":\"Error occurred!\"}"
2023-03-30 05:12:25 +0000 [warn]: #0 pattern not matched: "2023-03-27T19:44:54.585778441Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-27T19:44:54.585+00:00\",\"logger\":\"org.apache.coyote.http11.Http11NioProtocol\",\"level\":\"INFO\",\"class\":\"org.apache.juli.logging.DirectJDKLog\",\"method\":\"log\",\"file\":\"DirectJDKLog.java\",\"line\":173,\"thread\":\"main\",\"msg\":\"Starting ProtocolHandler [\\\"http-nio-8080\\\"]\"}"
2023-03-30 05:12:25 +0000 [warn]: #0 pattern not matched: "2023-03-27T19:44:54.638871056Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-27T19:44:54.638+00:00\",\"logger\":\"com.example.App\",\"level\":\"INFO\",\"class\":\"org.springframework.boot.StartupInfoLogger\",\"method\":\"logStarted\",\"file\":\"StartupInfoLogger.java\",\"line\":61,\"thread\":\"main\",\"msg\":\"Started App in 4.378 seconds (JVM running for 5.867)\"}"
2023-03-30 05:12:25 +0000 [info]: #0 following tail of /var/log/containers/node-app-5674575ddc-nqkqp_efk-stack_node-app-f3ee862f3f880734dfb747dfecb065b3450149a565a28a3340dfed3b4ef8a638.log
2023-03-30 05:12:25 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:59:26.486536084Z stdout F {\"level\":30,\"time\":\"2023-03-29T06:59:26.485Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-nqkqp\",\"msg\":\"hello elastic world\"}"
2023-03-30 05:12:25 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:59:26.486664535Z stdout F {\"level\":30,\"time\":\"2023-03-29T06:59:26.486Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-nqkqp\",\"msg\":\"This is some great stuff\"}"
2023-03-30 05:12:25 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:59:26.486675165Z stdout F {\"level\":30,\"time\":\"2023-03-29T06:59:26.486Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-nqkqp\",\"msg\":\"Some more entries for our logging\"}"
2023-03-30 05:12:25 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:59:26.486738685Z stdout F {\"level\":30,\"time\":\"2023-03-29T06:59:26.486Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-nqkqp\",\"msg\":\"another line\"}"
2023-03-30 05:12:25 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:59:26.486801365Z stdout F {\"level\":30,\"time\":\"2023-03-29T06:59:26.486Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-nqkqp\",\"msg\":\"This never stops\"}"
2023-03-30 05:12:25 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:59:26.486810545Z stdout F {\"level\":30,\"time\":\"2023-03-29T06:59:26.486Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-nqkqp\",\"msg\":\"Logging logging all the way\"}"
2023-03-30 05:12:25 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:59:26.486873885Z stdout F {\"level\":30,\"time\":\"2023-03-29T06:59:26.486Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-nqkqp\",\"msg\":\"I think this is enough\"}"
2023-03-30 05:12:25 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:59:26.486883896Z stdout F {\"level\":30,\"time\":\"2023-03-29T06:59:26.486Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-nqkqp\",\"msg\":\"nope, one more!\"}"
2023-03-30 05:12:25 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:59:26.491062815Z stdout F {\"level\":30,\"time\":\"2023-03-29T06:59:26.490Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-nqkqp\",\"msg\":\"app listening on port 3000!\"}"
2023-03-30 05:12:25 +0000 [info]: #0 fluentd worker is now running worker=0
2023-03-30 05:38:23 +0000 [info]: #0 410 Gone encountered. Restarting pod watch to reset resource versions.410 Gone
2023-03-30 05:38:23 +0000 [info]: 410 Gone encountered. Restarting pod watch to reset resource versions.410 Gone
10

2nd pod

e[38;5;6mfluentd e[38;5;5m05:11:47.97 e[0m
e[38;5;6mfluentd e[38;5;5m05:11:47.97 e[0me[1mWelcome to the Bitnami fluentd containere[0m
e[38;5;6mfluentd e[38;5;5m05:11:47.97 e[0mSubscribe to project updates by watching e[1mhttps://github.com/bitnami/containerse[0m
e[38;5;6mfluentd e[38;5;5m05:11:47.97 e[0mSubmit issues and feature requests at e[1mhttps://github.com/bitnami/containers/issuese[0m
e[38;5;6mfluentd e[38;5;5m05:11:47.97 e[0m
e[38;5;6mfluentd e[38;5;5m05:11:47.97 e[0me[38;5;2mINFO e[0m ==> ** Starting Fluentd setup **
find: '/docker-entrypoint-initdb.d/': No such file or directory
e[38;5;6mfluentd e[38;5;5m05:11:47.98 e[0me[38;5;2mINFO e[0m ==> No custom scripts in /docker-entrypoint-initdb.d
e[38;5;6mfluentd e[38;5;5m05:11:47.98 e[0me[38;5;2mINFO e[0m ==> ** Fluentd setup finished! **

e[38;5;6mfluentd e[38;5;5m05:11:47.99 e[0me[38;5;2mINFO e[0m ==> ** Starting Fluentd **
2023-03-30 05:11:49 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-03-30 05:11:49 +0000 [info]: parsing config file is succeeded path="/opt/bitnami/fluentd/conf/fluentd.conf"
2023-03-30 05:11:49 +0000 [info]: gem 'fluentd' version '1.15.3'
2023-03-30 05:11:49 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2023-03-30 05:11:49 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.14'
2023-03-30 05:11:49 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.2.4'
2023-03-30 05:11:49 +0000 [info]: gem 'fluent-plugin-grafana-loki' version '1.2.20'
2023-03-30 05:11:49 +0000 [info]: gem 'fluent-plugin-kafka' version '0.18.1'
2023-03-30 05:11:49 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '3.1.3'
2023-03-30 05:11:49 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2023-03-30 05:11:49 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.3'
2023-03-30 05:11:49 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2023-03-30 05:11:49 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2023-03-30 05:11:49 +0000 [info]: gem 'fluent-plugin-s3' version '1.7.2'
2023-03-30 05:11:49 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2023-03-30 05:11:50 +0000 [warn]: !! The environment variable 'K8S_NODE_NAME' is not set to the node name which can affect the API server and watch efficiency !!
2023-03-30 05:11:51 +0000 [debug]: [elasticsearch] 'host elasticsearch-master.efk-stack.svc.cluster.local' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: elasticsearch-master.efk-stack.svc.cluster.local' doesn't have tag placeholder
2023-03-30 05:11:51 +0000 [debug]: [elasticsearch] 'index_name java-app-logs' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: java-app-logs' doesn't have tag placeholder
2023-03-30 05:11:51 +0000 [debug]: [elasticsearch] 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2023-03-30 05:11:51 +0000 [debug]: [elasticsearch] 'logstash_prefix false' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: false' doesn't have tag placeholder
2023-03-30 05:11:51 +0000 [debug]: [elasticsearch] 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders, but chunk key 'time' is not configured
2023-03-30 05:11:51 +0000 [debug]: [elasticsearch] 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2023-03-30 05:11:51 +0000 [debug]: [elasticsearch] 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2023-03-30 05:11:51 +0000 [debug]: [elasticsearch] 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2023-03-30 05:11:51 +0000 [debug]: [elasticsearch] 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2023-03-30 05:11:51 +0000 [debug]: [elasticsearch] Need substitution: false
2023-03-30 05:11:51 +0000 [debug]: [elasticsearch] 'host_placeholder elasticsearch-master.efk-stack.svc.cluster.local' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: elasticsearch-master.efk-stack.svc.cluster.local' doesn't have tag placeholder
2023-03-30 05:11:51 +0000 [warn]: define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2023-03-30 05:11:51 +0000 [info]: using configuration file: <ROOT>
  <match fluent.**>
    @type null
  </match>
  <source>
    @type http
    port 9880
  </source>
  <source>
    @type tail
    path "/var/log/containers/*-app*.log"
    pos_file "/opt/bitnami/fluentd/logs/buffers/fluentd-docker.pos"
    tag "kubernetes.*"
    read_from_head true
    format json
    <parse>
      @type json
      unmatched_lines 
    </parse>
  </source>
  <filter kubernetes.**>
    @type kubernetes_metadata
  </filter>
  <match fluentd.healthcheck>
    @type null
  </match>
  <match **>
    @id elasticsearch
    @type elasticsearch
    @log_level "debug"
    include_tag_key true
    host "elasticsearch-master.efk-stack.svc.cluster.local"
    port 9200
    user "elastic"
    password xxxxxx
    index_name "java-app-logs"
    scheme https
    ssl_verify false
    logstash_prefix "false"
    logstash_format false
    <buffer>
      @type "file"
      path "/opt/bitnami/fluentd/logs/buffers/java-logs.buffer"
      flush_thread_count 2
      flush_interval 5s
    </buffer>
  </match>
</ROOT>
2023-03-30 05:11:51 +0000 [info]: starting fluentd-1.15.3 pid=1 ruby="3.1.3"
2023-03-30 05:11:51 +0000 [info]: spawn command to main:  cmdline=["/opt/bitnami/ruby/bin/ruby", "-Eascii-8bit:ascii-8bit", "/opt/bitnami/fluentd/bin/fluentd", "--config", "/opt/bitnami/fluentd/conf/fluentd.conf", "--plugin", "/opt/bitnami/fluentd/plugins", "--under-supervisor"]
2023-03-30 05:11:51 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-03-30 05:11:53 +0000 [info]: #0 init worker0 logger path=nil rotate_age=nil rotate_size=nil
2023-03-30 05:11:53 +0000 [info]: adding match pattern="fluent.**" type="null"
2023-03-30 05:11:53 +0000 [info]: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
2023-03-30 05:11:54 +0000 [warn]: #0 !! The environment variable 'K8S_NODE_NAME' is not set to the node name which can affect the API server and watch efficiency !!
2023-03-30 05:11:54 +0000 [info]: adding match pattern="fluentd.healthcheck" type="null"
2023-03-30 05:11:54 +0000 [info]: adding match pattern="**" type="elasticsearch"
2023-03-30 05:11:55 +0000 [debug]: #0 [elasticsearch] 'host elasticsearch-master.efk-stack.svc.cluster.local' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: elasticsearch-master.efk-stack.svc.cluster.local' doesn't have tag placeholder
2023-03-30 05:11:55 +0000 [debug]: #0 [elasticsearch] 'index_name java-app-logs' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: java-app-logs' doesn't have tag placeholder
2023-03-30 05:11:55 +0000 [debug]: #0 [elasticsearch] 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2023-03-30 05:11:55 +0000 [debug]: #0 [elasticsearch] 'logstash_prefix false' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: false' doesn't have tag placeholder
2023-03-30 05:11:55 +0000 [debug]: #0 [elasticsearch] 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders, but chunk key 'time' is not configured
2023-03-30 05:11:55 +0000 [debug]: #0 [elasticsearch] 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2023-03-30 05:11:55 +0000 [debug]: #0 [elasticsearch] 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2023-03-30 05:11:55 +0000 [debug]: #0 [elasticsearch] 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2023-03-30 05:11:55 +0000 [debug]: #0 [elasticsearch] 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2023-03-30 05:11:55 +0000 [debug]: #0 [elasticsearch] Need substitution: false
2023-03-30 05:11:55 +0000 [debug]: #0 [elasticsearch] 'host_placeholder elasticsearch-master.efk-stack.svc.cluster.local' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: elasticsearch-master.efk-stack.svc.cluster.local' doesn't have tag placeholder
2023-03-30 05:11:55 +0000 [debug]: #0 [elasticsearch] Detected ES 8.x or above: This parameter has no effect.
2023-03-30 05:11:55 +0000 [info]: adding source type="http"
2023-03-30 05:11:55 +0000 [info]: adding source type="tail"
2023-03-30 05:11:55 +0000 [warn]: #0 define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2023-03-30 05:11:55 +0000 [info]: #0 starting fluentd worker pid=34 ppid=1 worker=0
2023-03-30 05:11:55 +0000 [debug]: #0 [elasticsearch] buffer started instance=51620 stage_size=0 queue_size=0
2023-03-30 05:11:55 +0000 [debug]: #0 [elasticsearch] flush_thread actually running
2023-03-30 05:11:55 +0000 [info]: #0 fluentd worker is now running worker=0
2023-03-30 05:11:55 +0000 [debug]: #0 [elasticsearch] enqueue_thread actually running
2023-03-30 05:11:55 +0000 [debug]: #0 [elasticsearch] flush_thread actually running
2023-03-30 05:38:23 +0000 [info]: #0 410 Gone encountered. Restarting pod watch to reset resource versions.410 Gone
2023-03-30 05:38:23 +0000 [info]: 410 Gone encountered. Restarting pod watch to reset resource versions.410 Gone
11

3rd pod

e[38;5;6mfluentd e[38;5;5m05:12:00.83 e[0m
e[38;5;6mfluentd e[38;5;5m05:12:00.84 e[0me[1mWelcome to the Bitnami fluentd containere[0m
e[38;5;6mfluentd e[38;5;5m05:12:00.84 e[0mSubscribe to project updates by watching e[1mhttps://github.com/bitnami/containerse[0m
e[38;5;6mfluentd e[38;5;5m05:12:00.84 e[0mSubmit issues and feature requests at e[1mhttps://github.com/bitnami/containers/issuese[0m
e[38;5;6mfluentd e[38;5;5m05:12:00.84 e[0m
e[38;5;6mfluentd e[38;5;5m05:12:00.84 e[0me[38;5;2mINFO e[0m ==> ** Starting Fluentd setup **
find: '/docker-entrypoint-initdb.d/': No such file or directory
e[38;5;6mfluentd e[38;5;5m05:12:00.85 e[0me[38;5;2mINFO e[0m ==> No custom scripts in /docker-entrypoint-initdb.d
e[38;5;6mfluentd e[38;5;5m05:12:00.85 e[0me[38;5;2mINFO e[0m ==> ** Fluentd setup finished! **

e[38;5;6mfluentd e[38;5;5m05:12:00.86 e[0me[38;5;2mINFO e[0m ==> ** Starting Fluentd **
2023-03-30 05:12:02 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-03-30 05:12:02 +0000 [info]: parsing config file is succeeded path="/opt/bitnami/fluentd/conf/fluentd.conf"
2023-03-30 05:12:02 +0000 [info]: gem 'fluentd' version '1.15.3'
2023-03-30 05:12:02 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2023-03-30 05:12:02 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.14'
2023-03-30 05:12:02 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.2.4'
2023-03-30 05:12:02 +0000 [info]: gem 'fluent-plugin-grafana-loki' version '1.2.20'
2023-03-30 05:12:02 +0000 [info]: gem 'fluent-plugin-kafka' version '0.18.1'
2023-03-30 05:12:02 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '3.1.3'
2023-03-30 05:12:02 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2023-03-30 05:12:02 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.3'
2023-03-30 05:12:02 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2023-03-30 05:12:02 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2023-03-30 05:12:02 +0000 [info]: gem 'fluent-plugin-s3' version '1.7.2'
2023-03-30 05:12:02 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2023-03-30 05:12:03 +0000 [warn]: !! The environment variable 'K8S_NODE_NAME' is not set to the node name which can affect the API server and watch efficiency !!
2023-03-30 05:12:04 +0000 [debug]: [elasticsearch] 'host elasticsearch-master.efk-stack.svc.cluster.local' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: elasticsearch-master.efk-stack.svc.cluster.local' doesn't have tag placeholder
2023-03-30 05:12:04 +0000 [debug]: [elasticsearch] 'index_name java-app-logs' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: java-app-logs' doesn't have tag placeholder
2023-03-30 05:12:04 +0000 [debug]: [elasticsearch] 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2023-03-30 05:12:04 +0000 [debug]: [elasticsearch] 'logstash_prefix false' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: false' doesn't have tag placeholder
2023-03-30 05:12:04 +0000 [debug]: [elasticsearch] 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders, but chunk key 'time' is not configured
2023-03-30 05:12:04 +0000 [debug]: [elasticsearch] 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2023-03-30 05:12:04 +0000 [debug]: [elasticsearch] 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2023-03-30 05:12:04 +0000 [debug]: [elasticsearch] 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2023-03-30 05:12:04 +0000 [debug]: [elasticsearch] 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2023-03-30 05:12:04 +0000 [debug]: [elasticsearch] Need substitution: false
2023-03-30 05:12:04 +0000 [debug]: [elasticsearch] 'host_placeholder elasticsearch-master.efk-stack.svc.cluster.local' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: elasticsearch-master.efk-stack.svc.cluster.local' doesn't have tag placeholder
2023-03-30 05:12:04 +0000 [warn]: define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2023-03-30 05:12:04 +0000 [info]: using configuration file: <ROOT>
  <match fluent.**>
    @type null
  </match>
  <source>
    @type http
    port 9880
  </source>
  <source>
    @type tail
    path "/var/log/containers/*-app*.log"
    pos_file "/opt/bitnami/fluentd/logs/buffers/fluentd-docker.pos"
    tag "kubernetes.*"
    read_from_head true
    format json
    <parse>
      @type json
      unmatched_lines 
    </parse>
  </source>
  <filter kubernetes.**>
    @type kubernetes_metadata
  </filter>
  <match fluentd.healthcheck>
    @type null
  </match>
  <match **>
    @id elasticsearch
    @type elasticsearch
    @log_level "debug"
    include_tag_key true
    host "elasticsearch-master.efk-stack.svc.cluster.local"
    port 9200
    user "elastic"
    password xxxxxx
    index_name "java-app-logs"
    scheme https
    ssl_verify false
    logstash_prefix "false"
    logstash_format false
    <buffer>
      @type "file"
      path "/opt/bitnami/fluentd/logs/buffers/java-logs.buffer"
      flush_thread_count 2
      flush_interval 5s
    </buffer>
  </match>
</ROOT>
2023-03-30 05:12:04 +0000 [info]: starting fluentd-1.15.3 pid=1 ruby="3.1.3"
2023-03-30 05:12:04 +0000 [info]: spawn command to main:  cmdline=["/opt/bitnami/ruby/bin/ruby", "-Eascii-8bit:ascii-8bit", "/opt/bitnami/fluentd/bin/fluentd", "--config", "/opt/bitnami/fluentd/conf/fluentd.conf", "--plugin", "/opt/bitnami/fluentd/plugins", "--under-supervisor"]
2023-03-30 05:12:04 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-03-30 05:12:06 +0000 [info]: #0 init worker0 logger path=nil rotate_age=nil rotate_size=nil
2023-03-30 05:12:06 +0000 [info]: adding match pattern="fluent.**" type="null"
2023-03-30 05:12:06 +0000 [info]: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
2023-03-30 05:12:07 +0000 [warn]: #0 !! The environment variable 'K8S_NODE_NAME' is not set to the node name which can affect the API server and watch efficiency !!
2023-03-30 05:12:07 +0000 [info]: adding match pattern="fluentd.healthcheck" type="null"
2023-03-30 05:12:07 +0000 [info]: adding match pattern="**" type="elasticsearch"
2023-03-30 05:12:08 +0000 [debug]: #0 [elasticsearch] 'host elasticsearch-master.efk-stack.svc.cluster.local' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: elasticsearch-master.efk-stack.svc.cluster.local' doesn't have tag placeholder
2023-03-30 05:12:08 +0000 [debug]: #0 [elasticsearch] 'index_name java-app-logs' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: java-app-logs' doesn't have tag placeholder
2023-03-30 05:12:08 +0000 [debug]: #0 [elasticsearch] 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2023-03-30 05:12:08 +0000 [debug]: #0 [elasticsearch] 'logstash_prefix false' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: false' doesn't have tag placeholder
2023-03-30 05:12:08 +0000 [debug]: #0 [elasticsearch] 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders, but chunk key 'time' is not configured
2023-03-30 05:12:08 +0000 [debug]: #0 [elasticsearch] 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2023-03-30 05:12:08 +0000 [debug]: #0 [elasticsearch] 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2023-03-30 05:12:08 +0000 [debug]: #0 [elasticsearch] 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2023-03-30 05:12:08 +0000 [debug]: #0 [elasticsearch] 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2023-03-30 05:12:08 +0000 [debug]: #0 [elasticsearch] Need substitution: false
2023-03-30 05:12:08 +0000 [debug]: #0 [elasticsearch] 'host_placeholder elasticsearch-master.efk-stack.svc.cluster.local' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: elasticsearch-master.efk-stack.svc.cluster.local' doesn't have tag placeholder
2023-03-30 05:12:08 +0000 [debug]: #0 [elasticsearch] Detected ES 8.x or above: This parameter has no effect.
2023-03-30 05:12:08 +0000 [info]: adding source type="http"
2023-03-30 05:12:08 +0000 [info]: adding source type="tail"
2023-03-30 05:12:08 +0000 [warn]: #0 define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2023-03-30 05:12:08 +0000 [info]: #0 starting fluentd worker pid=34 ppid=1 worker=0
2023-03-30 05:12:08 +0000 [debug]: #0 [elasticsearch] buffer started instance=51620 stage_size=0 queue_size=0
2023-03-30 05:12:08 +0000 [debug]: #0 [elasticsearch] flush_thread actually running
2023-03-30 05:12:08 +0000 [debug]: #0 [elasticsearch] flush_thread actually running
2023-03-30 05:12:08 +0000 [debug]: #0 [elasticsearch] enqueue_thread actually running
2023-03-30 05:12:08 +0000 [info]: #0 following tail of /var/log/containers/java-app-549d7577fb-9fsbx_efk-stack_java-app-dd19de943b0669f0f5f6f313f315c9cc6402687a21a8cd5cb5178b12c4fe9597.log
2023-03-30 05:12:08 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:58:55.953726094Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-29T06:58:55.917+00:00\",\"logger\":\"com.example.App\",\"level\":\"INFO\",\"class\":\"org.springframework.boot.StartupInfoLogger\",\"method\":\"logStarting\",\"file\":\"StartupInfoLogger.java\",\"line\":55,\"thread\":\"main\",\"msg\":\"Starting App on java-app-549d7577fb-9fsbx with PID 1 (/usr/app/java-app-1.0-SNAPSHOT.jar started by root in /usr/app)\"}"
2023-03-30 05:12:08 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:58:55.960623084Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-29T06:58:55.960+00:00\",\"logger\":\"com.example.App\",\"level\":\"INFO\",\"class\":\"org.springframework.boot.SpringApplication\",\"method\":\"logStartupProfileInfo\",\"file\":\"SpringApplication.java\",\"line\":651,\"thread\":\"main\",\"msg\":\"No active profile set, falling back to default profiles: default\"}"
2023-03-30 05:12:08 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:58:57.567979296Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-29T06:58:57.567+00:00\",\"logger\":\"org.apache.coyote.http11.Http11NioProtocol\",\"level\":\"INFO\",\"class\":\"org.apache.juli.logging.DirectJDKLog\",\"method\":\"log\",\"file\":\"DirectJDKLog.java\",\"line\":173,\"thread\":\"main\",\"msg\":\"Initializing ProtocolHandler [\\\"http-nio-8080\\\"]\"}"
2023-03-30 05:12:08 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:58:57.569236491Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-29T06:58:57.568+00:00\",\"logger\":\"org.apache.catalina.core.StandardService\",\"level\":\"INFO\",\"class\":\"org.apache.juli.logging.DirectJDKLog\",\"method\":\"log\",\"file\":\"DirectJDKLog.java\",\"line\":173,\"thread\":\"main\",\"msg\":\"Starting service [Tomcat]\"}"
2023-03-30 05:12:08 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:58:57.56993596Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-29T06:58:57.569+00:00\",\"logger\":\"org.apache.catalina.core.StandardEngine\",\"level\":\"INFO\",\"class\":\"org.apache.juli.logging.DirectJDKLog\",\"method\":\"log\",\"file\":\"DirectJDKLog.java\",\"line\":173,\"thread\":\"main\",\"msg\":\"Starting Servlet engine: [Apache Tomcat/9.0.29]\"}"
2023-03-30 05:12:08 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:58:57.663176606Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-29T06:58:57.662+00:00\",\"logger\":\"org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/]\",\"level\":\"INFO\",\"class\":\"org.apache.juli.logging.DirectJDKLog\",\"method\":\"log\",\"file\":\"DirectJDKLog.java\",\"line\":173,\"thread\":\"main\",\"msg\":\"Initializing Spring embedded WebApplicationContext\"}"
2023-03-30 05:12:08 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:58:57.732736979Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-29T06:58:57.732+00:00\",\"logger\":\"com.example.App\",\"level\":\"INFO\",\"class\":\"com.example.App\",\"method\":\"init\",\"file\":\"App.java\",\"line\":23,\"thread\":\"main\",\"msg\":\"I am a Java app\"}"
2023-03-30 05:12:08 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:58:57.73312415Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-29T06:58:57.732+00:00\",\"logger\":\"com.example.App\",\"level\":\"INFO\",\"class\":\"com.example.App\",\"method\":\"init\",\"file\":\"App.java\",\"line\":24,\"thread\":\"main\",\"msg\":\"Just logging stuff\"}"
2023-03-30 05:12:08 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:58:57.737325255Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-29T06:58:57.735+00:00\",\"logger\":\"com.example.App\",\"level\":\"ERROR\",\"class\":\"com.example.App\",\"method\":\"init\",\"file\":\"App.java\",\"line\":29,\"thread\":\"main\",\"stack\":\"java.lang.NullPointerException: Ooh noes! Something bad happened\\n\\tat com.example.App.init(App.java:26)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\\n\\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\\n\\tat java.lang.reflect.Method.invoke(Method.java:498)\\n\\tat org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:389)\\n\\tat org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:333)\\n\\tat org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:157)\\n\\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:416)\\n\\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1788)\\n\\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:595)\\n\\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517)\\n\\tat org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323)\\n\\tat org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)\\n\\tat org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321)\\n\\tat org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)\\n\\tat org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:879)\\n\\tat org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:878)\\n\\tat org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)\\n\\tat org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141)\\n\\tat org.springframework.boot.SpringApplication.refresh(SpringApplication.java:747)\\n\\tat org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)\\n\\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:315)\\n\\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)\\n\\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:1215)\\n\\tat com.example.App.main(App.java:14)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\\n\\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\\n\\tat java.lang.reflect.Method.invoke(Method.java:498)\\n\\tat org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)\\n\\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:87)\\n\\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:51)\\n\\tat org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:52)\\n\",\"msg\":\"Error occurred!\"}"
2023-03-30 05:12:08 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:58:58.128968328Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-29T06:58:58.128+00:00\",\"logger\":\"org.apache.coyote.http11.Http11NioProtocol\",\"level\":\"INFO\",\"class\":\"org.apache.juli.logging.DirectJDKLog\",\"method\":\"log\",\"file\":\"DirectJDKLog.java\",\"line\":173,\"thread\":\"main\",\"msg\":\"Starting ProtocolHandler [\\\"http-nio-8080\\\"]\"}"
2023-03-30 05:12:08 +0000 [warn]: #0 pattern not matched: "2023-03-29T06:58:58.162547261Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-29T06:58:58.162+00:00\",\"logger\":\"com.example.App\",\"level\":\"INFO\",\"class\":\"org.springframework.boot.StartupInfoLogger\",\"method\":\"logStarted\",\"file\":\"StartupInfoLogger.java\",\"line\":61,\"thread\":\"main\",\"msg\":\"Started App in 3.037 seconds (JVM running for 4.196)\"}"
2023-03-30 05:12:08 +0000 [info]: #0 following tail of /var/log/containers/node-app-5674575ddc-b8nsj_default_node-app-7fda84f4efe1999ea0646602cb3ddb344b5d51bcf1d3d78b4b750bc3ba62cf03.log
2023-03-30 05:12:08 +0000 [warn]: #0 pattern not matched: "2023-03-27T21:04:37.343184371Z stdout F {\"level\":30,\"time\":\"2023-03-27T21:04:37.342Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-b8nsj\",\"msg\":\"hello elastic world\"}"
2023-03-30 05:12:08 +0000 [warn]: #0 pattern not matched: "2023-03-27T21:04:37.343369806Z stdout F {\"level\":30,\"time\":\"2023-03-27T21:04:37.343Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-b8nsj\",\"msg\":\"This is some great stuff\"}"
2023-03-30 05:12:08 +0000 [warn]: #0 pattern not matched: "2023-03-27T21:04:37.343416778Z stdout F {\"level\":30,\"time\":\"2023-03-27T21:04:37.343Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-b8nsj\",\"msg\":\"Some more entries for our logging\"}"
2023-03-30 05:12:08 +0000 [warn]: #0 pattern not matched: "2023-03-27T21:04:37.343456669Z stdout F {\"level\":30,\"time\":\"2023-03-27T21:04:37.343Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-b8nsj\",\"msg\":\"another line\"}"
2023-03-30 05:12:08 +0000 [warn]: #0 pattern not matched: "2023-03-27T21:04:37.343543261Z stdout F {\"level\":30,\"time\":\"2023-03-27T21:04:37.343Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-b8nsj\",\"msg\":\"This never stops\"}"
2023-03-30 05:12:08 +0000 [warn]: #0 pattern not matched: "2023-03-27T21:04:37.343583712Z stdout F {\"level\":30,\"time\":\"2023-03-27T21:04:37.343Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-b8nsj\",\"msg\":\"Logging logging all the way\"}"
2023-03-30 05:12:08 +0000 [warn]: #0 pattern not matched: "2023-03-27T21:04:37.343622373Z stdout F {\"level\":30,\"time\":\"2023-03-27T21:04:37.343Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-b8nsj\",\"msg\":\"I think this is enough\"}"
2023-03-30 05:12:08 +0000 [warn]: #0 pattern not matched: "2023-03-27T21:04:37.343660364Z stdout F {\"level\":30,\"time\":\"2023-03-27T21:04:37.343Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-b8nsj\",\"msg\":\"nope, one more!\"}"
2023-03-30 05:12:08 +0000 [warn]: #0 pattern not matched: "2023-03-27T21:04:37.346210555Z stdout F {\"level\":30,\"time\":\"2023-03-27T21:04:37.346Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-b8nsj\",\"msg\":\"app listening on port 3000!\"}"
2023-03-30 05:12:08 +0000 [info]: #0 fluentd worker is now running worker=0
2023-03-30 05:19:08 +0000 [info]: #0 following tail of /var/log/containers/java-app-549d7577fb-mr5s7_efk-stack-2_java-app-142a45c7f6c9a84bb2ce0796101aaaabdb07199964b74af278955a1b8e80cc7e.log
2023-03-30 05:19:08 +0000 [warn]: #0 pattern not matched: "2023-03-30T05:18:17.6268538Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-30T05:18:17.587+00:00\",\"logger\":\"com.example.App\",\"level\":\"INFO\",\"class\":\"org.springframework.boot.StartupInfoLogger\",\"method\":\"logStarting\",\"file\":\"StartupInfoLogger.java\",\"line\":55,\"thread\":\"main\",\"msg\":\"Starting App on java-app-549d7577fb-mr5s7 with PID 1 (/usr/app/java-app-1.0-SNAPSHOT.jar started by root in /usr/app)\"}"
2023-03-30 05:19:08 +0000 [warn]: #0 pattern not matched: "2023-03-30T05:18:17.631103016Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-30T05:18:17.630+00:00\",\"logger\":\"com.example.App\",\"level\":\"INFO\",\"class\":\"org.springframework.boot.SpringApplication\",\"method\":\"logStartupProfileInfo\",\"file\":\"SpringApplication.java\",\"line\":651,\"thread\":\"main\",\"msg\":\"No active profile set, falling back to default profiles: default\"}"
2023-03-30 05:19:08 +0000 [warn]: #0 pattern not matched: "2023-03-30T05:18:19.612358451Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-30T05:18:19.611+00:00\",\"logger\":\"org.apache.coyote.http11.Http11NioProtocol\",\"level\":\"INFO\",\"class\":\"org.apache.juli.logging.DirectJDKLog\",\"method\":\"log\",\"file\":\"DirectJDKLog.java\",\"line\":173,\"thread\":\"main\",\"msg\":\"Initializing ProtocolHandler [\\\"http-nio-8080\\\"]\"}"
2023-03-30 05:19:08 +0000 [warn]: #0 pattern not matched: "2023-03-30T05:18:19.613416781Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-30T05:18:19.613+00:00\",\"logger\":\"org.apache.catalina.core.StandardService\",\"level\":\"INFO\",\"class\":\"org.apache.juli.logging.DirectJDKLog\",\"method\":\"log\",\"file\":\"DirectJDKLog.java\",\"line\":173,\"thread\":\"main\",\"msg\":\"Starting service [Tomcat]\"}"
2023-03-30 05:19:08 +0000 [warn]: #0 pattern not matched: "2023-03-30T05:18:19.613991046Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-30T05:18:19.613+00:00\",\"logger\":\"org.apache.catalina.core.StandardEngine\",\"level\":\"INFO\",\"class\":\"org.apache.juli.logging.DirectJDKLog\",\"method\":\"log\",\"file\":\"DirectJDKLog.java\",\"line\":173,\"thread\":\"main\",\"msg\":\"Starting Servlet engine: [Apache Tomcat/9.0.29]\"}"
2023-03-30 05:19:08 +0000 [warn]: #0 pattern not matched: "2023-03-30T05:18:19.699244769Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-30T05:18:19.698+00:00\",\"logger\":\"org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/]\",\"level\":\"INFO\",\"class\":\"org.apache.juli.logging.DirectJDKLog\",\"method\":\"log\",\"file\":\"DirectJDKLog.java\",\"line\":173,\"thread\":\"main\",\"msg\":\"Initializing Spring embedded WebApplicationContext\"}"
2023-03-30 05:19:08 +0000 [warn]: #0 pattern not matched: "2023-03-30T05:18:19.765253424Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-30T05:18:19.764+00:00\",\"logger\":\"com.example.App\",\"level\":\"INFO\",\"class\":\"com.example.App\",\"method\":\"init\",\"file\":\"App.java\",\"line\":23,\"thread\":\"main\",\"msg\":\"I am a Java app\"}"
2023-03-30 05:19:08 +0000 [warn]: #0 pattern not matched: "2023-03-30T05:18:19.76548494Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-30T05:18:19.765+00:00\",\"logger\":\"com.example.App\",\"level\":\"INFO\",\"class\":\"com.example.App\",\"method\":\"init\",\"file\":\"App.java\",\"line\":24,\"thread\":\"main\",\"msg\":\"Just logging stuff\"}"
2023-03-30 05:19:08 +0000 [warn]: #0 pattern not matched: "2023-03-30T05:18:19.769440549Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-30T05:18:19.767+00:00\",\"logger\":\"com.example.App\",\"level\":\"ERROR\",\"class\":\"com.example.App\",\"method\":\"init\",\"file\":\"App.java\",\"line\":29,\"thread\":\"main\",\"stack\":\"java.lang.NullPointerException: Ooh noes! Something bad happened\\n\\tat com.example.App.init(App.java:26)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\\n\\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\\n\\tat java.lang.reflect.Method.invoke(Method.java:498)\\n\\tat org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:389)\\n\\tat org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:333)\\n\\tat org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:157)\\n\\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:416)\\n\\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1788)\\n\\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:595)\\n\\tat org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517)\\n\\tat org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323)\\n\\tat org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)\\n\\tat org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321)\\n\\tat org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)\\n\\tat org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:879)\\n\\tat org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:878)\\n\\tat org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)\\n\\tat org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141)\\n\\tat org.springframework.boot.SpringApplication.refresh(SpringApplication.java:747)\\n\\tat org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)\\n\\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:315)\\n\\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)\\n\\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:1215)\\n\\tat com.example.App.main(App.java:14)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\\n\\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\\n\\tat java.lang.reflect.Method.invoke(Method.java:498)\\n\\tat org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)\\n\\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:87)\\n\\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:51)\\n\\tat org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:52)\\n\",\"msg\":\"Error occurred!\"}"
2023-03-30 05:19:08 +0000 [warn]: #0 pattern not matched: "2023-03-30T05:18:20.134517314Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-30T05:18:20.133+00:00\",\"logger\":\"org.apache.coyote.http11.Http11NioProtocol\",\"level\":\"INFO\",\"class\":\"org.apache.juli.logging.DirectJDKLog\",\"method\":\"log\",\"file\":\"DirectJDKLog.java\",\"line\":173,\"thread\":\"main\",\"msg\":\"Starting ProtocolHandler [\\\"http-nio-8080\\\"]\"}"
2023-03-30 05:19:08 +0000 [warn]: #0 pattern not matched: "2023-03-30T05:18:20.174866598Z stdout F {\"app\":\"JavaApp\",\"time\":\"2023-03-30T05:18:20.174+00:00\",\"logger\":\"com.example.App\",\"level\":\"INFO\",\"class\":\"org.springframework.boot.StartupInfoLogger\",\"method\":\"logStarted\",\"file\":\"StartupInfoLogger.java\",\"line\":61,\"thread\":\"main\",\"msg\":\"Started App in 3.415 seconds (JVM running for 4.658)\"}"
2023-03-30 05:19:08 +0000 [info]: #0 following tail of /var/log/containers/node-app-5674575ddc-jrsnm_efk-stack-2_node-app-e317b95d1536c6597ff713e1dd048b9d0e65a63840a6ddbab51d451f2db70dcf.log
2023-03-30 05:19:08 +0000 [warn]: #0 pattern not matched: "2023-03-30T05:18:45.011015191Z stdout F {\"level\":30,\"time\":\"2023-03-30T05:18:45.010Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-jrsnm\",\"msg\":\"hello elastic world\"}"
2023-03-30 05:19:08 +0000 [warn]: #0 pattern not matched: "2023-03-30T05:18:45.011328631Z stdout F {\"level\":30,\"time\":\"2023-03-30T05:18:45.011Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-jrsnm\",\"msg\":\"This is some great stuff\"}"
2023-03-30 05:19:08 +0000 [warn]: #0 pattern not matched: "2023-03-30T05:18:45.011432654Z stdout F {\"level\":30,\"time\":\"2023-03-30T05:18:45.011Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-jrsnm\",\"msg\":\"Some more entries for our logging\"}"
2023-03-30 05:19:08 +0000 [warn]: #0 pattern not matched: "2023-03-30T05:18:45.011513196Z stdout F {\"level\":30,\"time\":\"2023-03-30T05:18:45.011Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-jrsnm\",\"msg\":\"another line\"}"
2023-03-30 05:19:08 +0000 [warn]: #0 pattern not matched: "2023-03-30T05:18:45.011637499Z stdout F {\"level\":30,\"time\":\"2023-03-30T05:18:45.011Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-jrsnm\",\"msg\":\"This never stops\"}"
2023-03-30 05:19:08 +0000 [warn]: #0 pattern not matched: "2023-03-30T05:18:45.011903276Z stdout F {\"level\":30,\"time\":\"2023-03-30T05:18:45.011Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-jrsnm\",\"msg\":\"Logging logging all the way\"}"
2023-03-30 05:19:08 +0000 [warn]: #0 pattern not matched: "2023-03-30T05:18:45.011912466Z stdout F {\"level\":30,\"time\":\"2023-03-30T05:18:45.011Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-jrsnm\",\"msg\":\"I think this is enough\"}"
2023-03-30 05:19:08 +0000 [warn]: #0 pattern not matched: "2023-03-30T05:18:45.011967918Z stdout F {\"level\":30,\"time\":\"2023-03-30T05:18:45.011Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-jrsnm\",\"msg\":\"nope, one more!\"}"
2023-03-30 05:19:08 +0000 [warn]: #0 pattern not matched: "2023-03-30T05:18:45.015542666Z stdout F {\"level\":30,\"time\":\"2023-03-30T05:18:45.015Z\",\"pid\":1,\"hostname\":\"node-app-5674575ddc-jrsnm\",\"msg\":\"app listening on port 3000!\"}"
2023-03-30 05:38:23 +0000 [info]: #0 410 Gone encountered. Restarting pod watch to reset resource versions.410 Gone
2023-03-30 05:38:23 +0000 [info]: 410 Gone encountered. Restarting pod watch to reset resource versions.410 Gone
12

hey @warkolm , have you been able to take a look?

13

Yep but I don't know fluentd so I can't really comment.

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.