# Parsing message with plain text and json

**URL:** <https://discuss.elastic.co/t/parsing-message-with-plain-text-and-json/172261>\
**Category:** Logstash\
**Created:** [March 14, 2019, 7:07am UTC](https://discuss.elastic.co/t/parsing-message-with-plain-text-and-json/172261 "2019-03-14T07:07:09Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![sunilmchaudhari](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunilmchaudhari/32/9475_2.png) [@sunilmchaudhari](https://discuss.elastic.co/u/sunilmchaudhari)\
**Post date:** [March 14, 2019, 7:07am UTC](https://discuss.elastic.co/t/parsing-message-with-plain-text-and-json/172261/1 "2019-03-14T07:07:09Z")

</div>

Hello,  
I have to filter one log which is mix of plain text and the json message.  
So the sample is  
[ERROR] ABCD posting failed! { StatusCodeError: 422 - {"error":{"message":"Duplicated ticket","status\_code":422}}

Can anyone guide me what filters I can use for this case and how?  
One way I have is I can write grok pattern for whole message but I don't think that's good solution.

BR,  
Sunil.

---

<div class="post-metadata">

**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\
**Post date:** [March 14, 2019, 8:34am UTC](https://discuss.elastic.co/t/parsing-message-with-plain-text-and-json/172261/2 "2019-03-14T08:34:10Z")

</div>

You could possibly use a [grok](https://www.elastic.co/guide/en/logstash/6.6/plugins-filters-grok.html) or a [dissect](https://www.elastic.co/guide/en/logstash/6.6/plugins-filters-dissect.html) filter for that.

I don't know dissect very well so I will give a grok example

> [@sunilmchaudhari](#):
>
> [ERROR] ABCD posting failed! { StatusCodeError: 422 - {"error":{"message":"Duplicated ticket","status\_code":422}}

Using the following grok pattern

```
\[%{DATA:level}\]%{SPACE}%{DATA:msg}\{%{SPACE}%{DATA:status}%{SPACE}-%{SPACE}%{DATA:json}\}$

```

Would result in

```
{
  "msg": "ABCD posting failed! ",
  "level": "ERROR",
  "json": "{\"error\":{\"message\":\"Duplicated ticket\",\"status_code\":422}",
  "status": "StatusCodeError: 422"
}

```

The you can use a [json](https://www.elastic.co/guide/en/logstash/6.6/plugins-filters-json.html) filter on the json field above to further parse it.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 14, 2019, 1:49pm UTC](https://discuss.elastic.co/t/parsing-message-with-plain-text-and-json/172261/3 "2019-03-14T13:49:55Z")

</div>

An example using dissect would be

```
    dissect { mapping => { "message" => "[%{level}] %{errorMessage} { %{} - %{json}" } }
    json { source => "json" }
```

---

<div class="post-metadata">

**Author:** ![sunilmchaudhari](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunilmchaudhari/32/9475_2.png) [@sunilmchaudhari](https://discuss.elastic.co/u/sunilmchaudhari)\
**Post date:** [March 15, 2019, 2:25am UTC](https://discuss.elastic.co/t/parsing-message-with-plain-text-and-json/172261/4 "2019-03-15T02:25:27Z")

</div>

Thanks @A_B,  
Above sample worked fine with this.  
However I have another sample which is bit different.  
**Input**

> [ERROR] @verifyFavourite: { status: 403, code: 'LIMIT\_REACHED', title: 'Maximum numbers of favourites reached' }

**Groke pattern I wrote for this**

\[%{DATA:level}\]%{SPACE}%{DATA:errorDescription}\:%{SPACE}%{GREEDYDATA:errorJsonMessage}\}$  
Somehow escaping slash removed in the post when I quoted the text, I don't know why. So put extra slashes.

above grok pattern gives me below json.

> ```
> {
> "level": [
> [
> "ERROR"
> ]
> ],
> "SPACE": [
> [
> " ",
> " "
> ]
> ],
> "errorDescription": [
> [
> "@verifyFavourite"
> ]
> ],
> "errorJsonMessage": [
> [
> "{ status: 403, code: 'LIMIT_REACHED', title: 'Maximum numbers of favourites reached' "
> ]
> ]
> }
> 
> ```

Note the single quote around value in json 'LIMIT\_REACHED'.

However when I give same data to logstash config using std input, using echo command. It gives me below json. and got jsonparsefailure for the json field.

**Command:**

> echo '[ERROR] @verifyFavourite: { status: 403, code: 'LIMIT\_REACHED', title: 'Maximum numbers of favourites reached' }' |bin/logstash -f logstash\_ec2\_grok.conf --path.data /esarchive/test\_environment/

**Output:**

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/e/ee6205f2e782ab59b5ac2fa501f160abb38b6dc5.png)

In this case it gives json without single quotes. LIMIT\_REACHED  
Why is this difference in output from grok debugger and the console output?

BR,  
Sunil.

---

<div class="post-metadata">

**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\
**Post date:** [March 15, 2019, 9:52am UTC](https://discuss.elastic.co/t/parsing-message-with-plain-text-and-json/172261/5 "2019-03-15T09:52:49Z")

</div>

> [@sunilmchaudhari](#):
>
> { status: 403, code: 'LIMIT\_REACHED', title: 'Maximum numbers of favourites reached' }

I don't think that is valid JSON. E.g. `jq` on my machine does not like it.

That aside, if you have more than one line format in a log file you have to handle that in some way... Staying with `grok` you can make one über pattern that handles all scenarios (with conditional and/or optional fields) or make one named pattern per line type and match the lines against any of them

To elaborate a bit  
Make a pattern file in e.g. `/etc/logstash/patterns/my_log_patterns` with content (just grabbing the patterns from above)

```auto
MY_PATTERN_01 \[%{DATA:level}\]%{SPACE}%{DATA:msg}\{%{SPACE}%{DATA:status}%{SPACE}-%{SPACE}%{DATA:json}\}$
MY_PATTERN_02 \[%{DATA:level}\]%{SPACE}%{DATA:errorDescription}\:%{SPACE}%{GREEDYDATA:errorJsonMessage}\}$

COMBINED_PATTERNS (%{MY_PATTERN_01}|%{MY_PATTERN_02})

```

Then you can do

```auto
match => { "message" => "%{COMBINED_PATTERNS}" }

```

---

<div class="post-metadata">

**Author:** ![sunilmchaudhari](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunilmchaudhari/32/9475_2.png) [@sunilmchaudhari](https://discuss.elastic.co/u/sunilmchaudhari)\
**Post date:** [March 15, 2019, 11:36am UTC](https://discuss.elastic.co/t/parsing-message-with-plain-text-and-json/172261/6 "2019-03-15T11:36:14Z")

</div>

Hi @A_B,  
Thanks for this approach.  
I am struggling with jsonParse failure.

This is my input:

> { expedia: '{"status":{"code":500,"description":"Internal server error"},"data":"Cannot read property 'toFixed' of undefined"}',status: 'PAYMENT\_FAILED' }

Grok Pattern:

> %{GREEDYDATA:expediaDetails}$

When I use json filter for expediaDetails, it fails.

expediaDetails Output of logstash:

> "expediaDetails" =\> "{ expedia: status:code:500,status: PAYMENT\_FAILED } { expedia: status:description:Internal server error,status: PAYMENT\_FAILED } { expedia: data:Cannot read property \'toFixed\' of undefined,status: PAYMENT\_FAILED } "

Input is the nested json and the output is very different than the output in grokedebugger.  
from the logstash output, I am not able to put the values in new fields. For example I tried below approach. Both are not working.

1. 

> ruby {  
> code =\> 'event.set("statusCode1", event.get("[expediaDetails][expedia][status][code]"))'  
> }

1. 

> mutate { add\_field =\> {  
> "statusCode" =\> "%{[expediaDetails][expedia][status][code]}"  
> }

I dont know how to put these values in particular field.  
What I understand is, in logstash output, its printing message like an array of values.  
So first it prints **expedia:status:code:500** , later **expedia:status:description** and so on. But actual input structure is different.  
Please help on this.

BR,  
Sunil.

---

<div class="post-metadata">

**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\
**Post date:** [March 15, 2019, 11:59am UTC](https://discuss.elastic.co/t/parsing-message-with-plain-text-and-json/172261/7 "2019-03-15T11:59:52Z")

</div>

> [@sunilmchaudhari](#):
>
> ',status: 'PAYMENT\_FAILED' }

You go to the end of the line but the last part in not JSON. Only the part between the single ticks. i.e.

> [@sunilmchaudhari](#):
>
> {"status":{"code":500,"description":"Internal server error"},"data":"Cannot read property 'toFixed' of undefined"}

So, I would expect JSON parse failures...

---

<div class="post-metadata">

**Author:** ![sunilmchaudhari](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunilmchaudhari/32/9475_2.png) [@sunilmchaudhari](https://discuss.elastic.co/u/sunilmchaudhari)\
**Post date:** [March 15, 2019, 1:01pm UTC](https://discuss.elastic.co/t/parsing-message-with-plain-text-and-json/172261/8 "2019-03-15T13:01:47Z")

</div>

@A_B  
Sorry, Can you please explain bit more. I couldn't understand.  
Also please suggest the solution.

---

<div class="post-metadata">

**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\
**Post date:** [March 15, 2019, 2:00pm UTC](https://discuss.elastic.co/t/parsing-message-with-plain-text-and-json/172261/9 "2019-03-15T14:00:01Z")

</div>

E.g. [https://stackoverflow.com/questions/4162642/single-vs-double-quotes-in-json/34855065](https://stackoverflow.com/questions/4162642/single-vs-double-quotes-in-json/34855065)

_JSON requires double quotes for its strings._

Which means that the start and end of the following is not valid JSON

```
{ expedia: '{"status":{"code":500,"description":"Internal server error"},"data":"Cannot read property 'toFixed' of undefined"}',status: 'PAYMENT_FAILED' }

```

Only the middle part that is _within_ the single quotes. I have learned this the hard way 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 12, 2019, 2:03pm UTC](https://discuss.elastic.co/t/parsing-message-with-plain-text-and-json/172261/10 "2019-04-12T14:03:44Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.