# JSON log parse to separate fields

**URL:** <https://discuss.elastic.co/t/json-log-parse-to-separate-fields/303511>\
**Category:** Elasticsearch\
**Created:** [April 28, 2022, 1:08pm UTC](https://discuss.elastic.co/t/json-log-parse-to-separate-fields/303511 "2022-04-28T13:08:18Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dmitriy\_Esin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dmitriy_esin/32/95725_2.png) [@Dmitriy\_Esin](https://discuss.elastic.co/u/Dmitriy_Esin)\
**Post date:** [April 28, 2022, 1:08pm UTC](https://discuss.elastic.co/t/json-log-parse-to-separate-fields/303511/1 "2022-04-28T13:08:18Z")

</div>

Hi all!  
I have a question about implementing JSON log parsing to separated fields in Kibana.  
I use fluend to get logs from my k8s cluster.  
See the config:

```auto
  containers.input.conf: |-
    <source>
      @id fluentd-containers.log
      @type tail
      path /var/log/containers/*.log
      pos_file /var/log/es-containers.log.pos
      tag raw.kubernetes.*
      read_from_head true
      <parse>
        @type multi_format
        <pattern>
          format json
          time_key time
          time_format %Y-%m-%dT%H:%M:%S.%NZ
        </pattern>
        <pattern>
          format /^(?<time>.+) (?<stream>stdout|stderr) [^]* (?<log>.*)$/
          time_format %Y-%m-%dT%H:%M:%S.%N%:z
        </pattern>
      </parse>
    </source>
    # Detect exceptions in the log output and forward them as one log entry.
    <match raw.kubernetes.**>
      @id raw.kubernetes
      @type detect_exceptions
      remove_tag_prefix raw
      message log
      stream stream
      multiline_flush_interval 5
      max_bytes 500000
      max_lines 1000
    </match>

```

```auto
  system.input.conf: |-
    <source>
      @id systemd.log
      @type systemd
      tag systemd
      read_from_head true
      <storage>
        @type local
        persistent true
        path /var/log/systemd.log.pos
      </storage>
      <entry>
        field_map {"MESSAGE": "log", "_PID": ["process", "pid"], "_CMDLINE": "process", "_COMM": "cmd"}
        field_map_strict false
        fields_strip_underscores true
        fields_lowercase true
      </entry>
    </source>
    # Example:
    # I1118 21:26:53.975789 6 proxier.go:1096] Port "nodePort for kube-system/default-http-backend:http" (:31429/tcp) was open before and is still needed
    <source>
      @id kube-proxy.log
      @type tail
      format multiline
      multiline_flush_interval 5s
      format_firstline /^\w\d{4}/
      format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<log>.*)/
      time_format %m%d %H:%M:%S.%N
      path /var/log/kube-proxy.log
      pos_file /var/log/es-kube-proxy.log.pos
      tag kubeproxy
      read_from_head true
    </source>

```

```auto
output.conf: |-
    <filter kubeproxy>
      @type record_transformer
      enable_ruby
      <record>
        hostname ${ENV["HOSTNAME"]}
      </record>
    </filter>
    
    <filter **>
      @type prometheus
      <metric>
        type counter
        name fluentd_input_status_num_records_total
        desc Total number of log entries generated by either application containers or system components
      </metric>
    </filter>
    <filter kubernetes.**>
      @type kubernetes_metadata
    </filter>
    <match **>
      @id elasticsearch
      @type elasticsearch
      @log_level info
      include_tag_key true
      host ${HOST}
      port 9200
      scheme https
      ssl_verify false
      ssl_version TLSv1_2
      user ${USER}
      password ${PASSWORD}
      logstash_format true
      logstash_prefix ${INDEX_NAME}
      <buffer>
        @type file
        path /var/log/fluentd-buffers/kubernetes.system.buffer
        flush_mode interval
        retry_type exponential_backoff
        flush_thread_count 2
        flush_interval 5s
        retry_forever
        retry_max_interval 30
        chunk_limit_size 2M
        queue_limit_length 8
        overflow_action block
      </buffer>
    </match>

```

How I receive all logs and JSON logs at the current time (see the highlighted fields within the JSON log string - these need to be parsed out into individual fields in the index):

 ![Screenshot 2022-04-28 at 15.59.14](https://us1.discourse-cdn.com/elastic/original/3X/9/d/9dbe55f8b3f3a9c1908eb1014a8ea30973381211.jpeg)

How it should be:

 ![Screenshot 2022-04-28 at 16.00.26](https://us1.discourse-cdn.com/elastic/original/3X/a/e/ae1a2e83660b09e4220ab19f1a09a45ce0037565.jpeg)  
 ![Screenshot 2022-04-28 at 16.00.59](https://us1.discourse-cdn.com/elastic/original/3X/b/4/b4bbe72a932acc370cb1e1e9a2d4119c80e7d1ac.png)  
 ![Screenshot 2022-04-28 at 16.01.14](https://us1.discourse-cdn.com/elastic/original/3X/c/9/c9790494ffe70dc9fff31731aa561cd9507be50d.png)

But I also have non a JSON type logs in this index and nothing should be broken.  
I know that I can use Ingest Pipeline for filebeat, but in my case, I need to create something the same for fluentd.  
Does anyone have ideas?  
I will be glad for any help from you!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 26, 2022, 1:09pm UTC](https://discuss.elastic.co/t/json-log-parse-to-separate-fields/303511/2 "2022-05-26T13:09:10Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.