EFK Stack on Kubernetes - Collecting logs from default namespace

Elastic Stack Elasticsearch
1

Hi, I am using EFK stack on Kubernetes, I want to configure fluentd to collect logs from one specific namespace, the default namespace.
This is my fleuntd config file:

<label @FLUENT_LOG>
      <match fluent.**>
        @type null
      </match>
    </label>

    <source>
      @type tail
      path /var/log/containers/*.log
      pos_file /var/log/app.log.pos
      tag kubernetes.*
      read_from_head true
      <parse>
        @type json
        time_format %Y-%m-%dT%H:%M:%S.%NZ
      </parse>
    </source>

    <filter kubernetes.**>
        @type kubernetes_metadata
    </filter>

    <filter kubernetes.**>
       @type grep
         <exclude>
            key log
            pattern (.\[notice]\.*|^[ \\\/\(\)\*\|_]+(?!.*[a-zA-Z0-9]).*$|^\s*$|.*GET*|.*POST*)
         </exclude>
         <regexp>
            key kubernetes.namespace_name
            pattern /^default$/
         </regexp>
    </filter>
    <match ** >
       @type elasticsearch_dynamic
       @log_level info
       include_tag_key true
       host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
       port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
       user "#{ENV['FLUENT_ELASTICSEARCH_USER']}"
       password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD']}"
       scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}"
       ssl_verify "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'true'}"
       reload_connections true
       logstash_format true
       logstash_prefix logstash
       <buffer>
           @type file
           path /var/log/fluentd-buffers/kubernetes.system.buffer
           flush_mode interval
           retry_type exponential_backoff
           flush_thread_count 2
           flush_interval 5s
           retry_forever true
           retry_max_interval 30
           chunk_limit_size 2M
           queue_limit_length 32
           overflow_action block
       </buffer>
    </match>

The way I tired to do that is by adding:

         <regexp>
            key kubernetes.namespace_name
            pattern /^default$/
         </regexp>

but is not working this way, fluentd is not collecting any logs.

I don't know if the question is for here but please help me if you know the answer.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.