# Parse json in Log field to get individual fields for visualization

**URL:** <https://discuss.elastic.co/t/parse-json-in-log-field-to-get-individual-fields-for-visualization/291244>\
**Category:** Logstash\
**Created:** [December 8, 2021, 4:34pm UTC](https://discuss.elastic.co/t/parse-json-in-log-field-to-get-individual-fields-for-visualization/291244 "2021-12-08T16:34:01Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![girija](https://avatars.discourse-cdn.com/v4/letter/g/96bed5/32.png) [@girija](https://discuss.elastic.co/u/girija)\
**Post date:** [December 8, 2021, 4:34pm UTC](https://discuss.elastic.co/t/parse-json-in-log-field-to-get-individual-fields-for-visualization/291244/1 "2021-12-08T16:34:01Z")

</div>

Hello  
I am new to Kibana and have difficulty reading the logs for visualization.  
I have pasted a sample of how my log field looks.  
I need to read the values inside the log for creating visualizations in Kibana, like where env is DEV, or where transactionId is xyz, or where payload contains a certain value. Could someone help me with how this is done? Thank you!

Pasting sample log field here:

```auto
2021-12-08T14:02:49.899 INFO [bwEngThread:In-Memory Process Worker-1] c.t.b.p.g.L.E.LogMessage - {"env":"<mark>DEV</mark>","appName":"GenLogs","transactionID":"449d8241-392f-4877-a5ea-dddeebed3c29","timestamp":"1638972169775","srcApplication":"EPIC","operation":"testLog","type":"INFO","message":"New message logged.","payload":"<timer:TimerOutputSchema xmlns:timer=\"http://tns.tibco.com/bw/activity/timer/xsd/output\"><Now>1638972169328</Now><Hour>2</Hour><Minute>2</Minute><Second>49</Second><Week>50</Week><Month>12</Month><Year>2021</Year><Date>2021-12-08</Date><Time>2:02:49 PM</Time><DayOfMonth>8</DayOfMonth></timer:TimerOutputSchema>"}

```

 ![kibana](https://us1.discourse-cdn.com/elastic/original/3X/1/e/1e3bbf32dd71230cb723c695c7d1ee4c0b5db626.png)

---

<div class="post-metadata">

**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\
**Post date:** [December 8, 2021, 4:54pm UTC](https://discuss.elastic.co/t/parse-json-in-log-field-to-get-individual-fields-for-visualization/291244/2 "2021-12-08T16:54:12Z")

</div>

You will need to use ingest processors like [grok](https://www.elastic.co/guide/en/elasticsearch/reference/current/grok-processor.html) or [dissect](https://www.elastic.co/guide/en/elasticsearch/reference/current/dissect-processor.html) and then [json](https://www.elastic.co/guide/en/elasticsearch/reference/current/json-processor.html) to parse it all out. If you are ingesting through Logstash you can also do it there.

If you can paste your entire `log` message in text format then someone might be able to help you out configuring it. Should be text, not a screenshot.

---

<div class="post-metadata">

**Author:** ![girija](https://avatars.discourse-cdn.com/v4/letter/g/96bed5/32.png) [@girija](https://discuss.elastic.co/u/girija)\
**Post date:** [December 8, 2021, 6:05pm UTC](https://discuss.elastic.co/t/parse-json-in-log-field-to-get-individual-fields-for-visualization/291244/3 "2021-12-08T18:05:02Z")

</div>

Thank you for replying. I have updated my post with the log details now.

I am using fluent-bit and logstash to ingest.

---

<div class="post-metadata">

**Author:** ![AquaX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aquax/32/92006_2.png) [@AquaX](https://discuss.elastic.co/u/AquaX)\
**Post date:** [December 8, 2021, 7:44pm UTC](https://discuss.elastic.co/t/parse-json-in-log-field-to-get-individual-fields-for-visualization/291244/4 "2021-12-08T19:44:15Z")

</div>

> [@girija](#):
>
> ```auto
> 2021-12-08T14:02:49.899 INFO [bwEngThread:In-Memory Process Worker-1] c.t.b.p.g.L.E.LogMessage - {"env":"<mark>DEV</mark>","appName":"GenLogs","transactionID":"449d8241-392f-4877-a5ea-dddeebed3c29","timestamp":"1638972169775","srcApplication":"EPIC","operation":"testLog","type":"INFO","message":"New message logged.","payload":"<timer:TimerOutputSchema xmlns:timer=\"http://tns.tibco.com/bw/activity/timer/xsd/output\"><Now>1638972169328</Now><Hour>2</Hour><Minute>2</Minute><Second>49</Second><Week>50</Week><Month>12</Month><Year>2021</Year><Date>2021-12-08</Date><Time>2:02:49 PM</Time><DayOfMonth>8</DayOfMonth></timer:TimerOutputSchema>"}
> 
> ```

Try this for a [grok](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html) pattern:  
`%{DATE}T%{TIME} %{NOTSPACE:log_level} \[%{DATA:worker}\] %{DATA:message_type} - %{GREEDYDATA:json_message}`

You should end up with something like this:

```auto
{
  "DATE": [
    [
      "21-12-08"
    ]
  ],
  "DATE_US": [
    [
      null
    ]
  ],
  "MONTHNUM": [
    [
      null,
      "12"
    ]
  ],
  "MONTHDAY": [
    [
      null,
      "21"
    ]
  ],
  "YEAR": [
    [
      null,
      "08"
    ]
  ],
  "DATE_EU": [
    [
      "21-12-08"
    ]
  ],
  "TIME": [
    [
      "14:02:49.899"
    ]
  ],
  "HOUR": [
    [
      "14"
    ]
  ],
  "MINUTE": [
    [
      "02"
    ]
  ],
  "SECOND": [
    [
      "49.899"
    ]
  ],
  "log_level": [
    [
      "INFO"
    ]
  ],
  "worker": [
    [
      "bwEngThread:In-Memory Process Worker-1"
    ]
  ],
  "message_type": [
    [
      "c.t.b.p.g.L.E.LogMessage"
    ]
  ],
  "json_message": [
    [
      "{"env":"<mark>DEV</mark>","appName":"GenLogs","transactionID":"449d8241-392f-4877-a5ea-dddeebed3c29","timestamp":"1638972169775","srcApplication":"EPIC","operation":"testLog","type":"INFO","message":"New message logged.","payload":"<timer:TimerOutputSchema xmlns:timer=\\"http://tns.tibco.com/bw/activity/timer/xsd/output\\"><Now>1638972169328</Now><Hour>2</Hour><Minute>2</Minute><Second>49</Second><Week>50</Week><Month>12</Month><Year>2021</Year><Date>2021-12-08</Date><Time>2:02:49 PM</Time><DayOfMonth>8</DayOfMonth></timer:TimerOutputSchema>"}"
    ]
  ]
}

```

Then you can send the json\_message field through a [JSON filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html) and break that apart even further so that you can get individual fields in the root of your json object.

Hope this helps!

---

<div class="post-metadata">

**Author:** ![girija](https://avatars.discourse-cdn.com/v4/letter/g/96bed5/32.png) [@girija](https://discuss.elastic.co/u/girija)\
**Post date:** [December 9, 2021, 5:56am UTC](https://discuss.elastic.co/t/parse-json-in-log-field-to-get-individual-fields-for-visualization/291244/5 "2021-12-09T05:56:33Z")

</div>

Thank you so much for taking time out for helping me. I will give this a try.

---

<div class="post-metadata">

**Author:** ![girija](https://avatars.discourse-cdn.com/v4/letter/g/96bed5/32.png) [@girija](https://discuss.elastic.co/u/girija)\
**Post date:** [December 9, 2021, 9:56am UTC](https://discuss.elastic.co/t/parse-json-in-log-field-to-get-individual-fields-for-visualization/291244/6 "2021-12-09T09:56:40Z")

</div>

Hi Andres,

I had to remove logstash from the stack as customer wants only fluentbit and elastic.  
I read that grok does not work with fluentbit. IS there any other way to achieve what I want with just fluentbit and elastic?

Thank you

---

<div class="post-metadata">

**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\
**Post date:** [December 9, 2021, 11:04am UTC](https://discuss.elastic.co/t/parse-json-in-log-field-to-get-individual-fields-for-visualization/291244/7 "2021-12-09T11:04:49Z")

</div>

You can use [Ingest Processors](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html) in Elastic. Here is a simulation so you can see how it's done. Run in Dev Tools in Kibana.

```auto
POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "description": "...",
    "processors": [
      {
        "grok": {
          "field": "message",
          "patterns": [
            """
              %{DATE}T%{TIME} %{NOTSPACE:log_level} \[%{DATA:worker}\] %{DATA:message_type} - %{GREEDYDATA:json_message}
            """
          ]
        }
      },
      {
        "json": {
          "field": "json_message",
          "add_to_root": true
        }
      },
      {
        "remove": {
          "field": [
            "message",
            "json_message"
          ]
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "message": """
          2021-12-08T14:02:49.899 INFO [bwEngThread:In-Memory Process Worker-1] c.t.b.p.g.L.E.LogMessage - {"env":"<mark>DEV</mark>","appName":"GenLogs","transactionID":"449d8241-392f-4877-a5ea-dddeebed3c29","timestamp":"1638972169775","srcApplication":"EPIC","operation":"testLog","type":"INFO","message":"New message logged.","payload":"<timer:TimerOutputSchema xmlns:timer=\"http://tns.tibco.com/bw/activity/timer/xsd/output\"><Now>1638972169328</Now><Hour>2</Hour><Minute>2</Minute><Second>49</Second><Week>50</Week><Month>12</Month><Year>2021</Year><Date>2021-12-08</Date><Time>2:02:49 PM</Time><DayOfMonth>8</DayOfMonth></timer:TimerOutputSchema>"}
        """
      }
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![girija](https://avatars.discourse-cdn.com/v4/letter/g/96bed5/32.png) [@girija](https://discuss.elastic.co/u/girija)\
**Post date:** [December 9, 2021, 12:24pm UTC](https://discuss.elastic.co/t/parse-json-in-log-field-to-get-individual-fields-for-visualization/291244/8 "2021-12-09T12:24:05Z")

</div>

Thank you

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 6, 2022, 12:24pm UTC](https://discuss.elastic.co/t/parse-json-in-log-field-to-get-individual-fields-for-visualization/291244/9 "2022-01-06T12:24:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.