# Multiline json over TCP is not parsed

**URL:** <https://discuss.elastic.co/t/multiline-json-over-tcp-is-not-parsed/249438>\
**Category:** Logstash\
**Created:** [September 22, 2020, 3:11am UTC](https://discuss.elastic.co/t/multiline-json-over-tcp-is-not-parsed/249438 "2020-09-22T03:11:54Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![cristi.mitroi](https://avatars.discourse-cdn.com/v4/letter/c/e79b87/32.png) [@cristi.mitroi](https://discuss.elastic.co/u/cristi.mitroi)\
**Post date:** [September 22, 2020, 3:11am UTC](https://discuss.elastic.co/t/multiline-json-over-tcp-is-not-parsed/249438/1 "2020-09-22T03:11:54Z")

</div>

I have the following multiline json:  
"  
{"businessId":"335","technicId":"41180883","teller":"progr\_1 ","pid":"0024444930","timestamp":"16:03:31.0664","sursa":"test.mbv","nr\_linie":"391","mesaj":"MESSAGE PUT TIME IS 16033104"}  
{"businessId":"335","technicId":"41180884","teller":"progr\_1 ","pid":"0024444930","timestamp":"16:03:31.0665","sursa":"test.mbk","nr\_linie":"392","mesaj":"MESSAGE PUT DATE IS 20200902"}  
{"businessId":"335","technicId":"11808853","teller":"progr\_1 ","pid":"0024444930","timestamp":"16:03:31.0665","sursa":"test.mbx","nr\_linie":"401","mesaj":"ended with reason code 000000000 "}  
"  
and the following logstash conf:

input {  
tcp {  
port =\> 9601  
codec =\> multiline {  
pattern =\> "^{"  
negate =\> true  
what =\> previous  
}  
}  
}

filter {  
json {  
source =\> "message"  
}  
}

output {  
elasticsearch {  
hosts =\> ["[http://localhost:9200](http://localhost:9200)"]  
index =\> "next\_index"  
}  
stdout {  
codec =\> rubydebug  
}  
}

and I get the following error:

[WARN] 2020-09-22 05:42:17.480 [[main]\>worker7] json - Error parsing json {:source=\>"message", :raw=\>"52 \<13\>1 2020-09-22T05:41:45+03:00 bndikdev}, - - - \n263 \<13\>1 2020-09-22T05:41:45+03:00 bndikdev{"businessId" - - - "335","technicId":"41180883","teller":"progr\_1 ","pid":"0024444930","timestamp":"16:03:31.0897","sursa":"opnAAmq1.cbl","nr\_linie":"940","mesaj":"TECHNICID = 41180883 "},\n232 \<13\>1 2020-09-22T05:41:45+03:00 bndikdev{"businessId" - - - "335","technicId":"41180883","teller":"progr\_1 ","pid":"0024444930","timestamp":"16:03:31.0897","sursa":"opnAAmq1.cbl","nr\_linie":"941","mesaj":"MSGBRCH = 1000"},\n259 \<13\>1 2020-09-22T05:41:45+03:00 bndikdev {"businessId" - - - "335","technicId":"41180884","teller":"progr\_1 ","pid":"0024444930","timestamp":"16:03:31.0897","sursa":"opnAAmq1.cbl","nr\_linie":"942","mesaj":"MSGTYPE = OPNAA "}", :exception=\>#\<LogStash::Json::ParserError: Unexpected character ('\<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')  
at [Source: (byte)"52 \<13\>1 2020-09-22T05:41:45+03:00 bndikdev}, - - -  
263 \<13\>1 2020-09-22T05:41:45+03:00 bndikdev{"businessId" - - - "335","technicId":"41180884","teller":"progr\_1 ","pid":"0024444930","timestamp":"16:03:31.0897","sursa":"opnAAmq1.cbl","nr\_linie":"940","mesaj":"TECHNICID = 11808853 "},  
232 \<13\>1 2020-09-22T05:41:45+03:00 bndikdev{"businessId" - - - "335","technicId":"11808853","teller":"progr\_1 ","pid":"0024444930","timestamp":"16:03:31.0897","sursa":""[truncated 320 bytes]; line: 1, column: 5]\>}

I also tried to parse the multiline json using the following conf:  
input {  
tcp {  
port =\> 9601  
codec =\> plain  
}  
}

filter {  
json {  
source =\> "message"  
}  
}

output {  
elasticsearch {  
hosts =\> ["[http://localhost:9200](http://localhost:9200)"]  
index =\> "next\_index"  
}  
stdout {  
codec =\> rubydebug  
}  
}  
and I still get json parse failure:  
[WARN] 2020-09-22 03:59:44.495 [[main]\>worker0] json - Error parsing json {:source=\>"message", :raw=\>"250 \<13\>1 2020-09-22T03:59:44+03:00 bndikdev {"businessId" - - - "335","technicId":"641180883","teller":"progr\_1 ","pid":"0024444930","timestamp":"16:03:31.0856","sursa":"opnAAmq1.cbl","nr\_linie":"370","mesaj":"alternative nespecificate",", :exception=\>#\<LogStash::Json::ParserError: Unexpected character ('\<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')  
at [Source: (byte)"250 \<13\>1 2020-09-22T03:59:44+03:00 bndikdev {"businessId" - - - "335","technicId":"641180883","teller":"progr\_1 ","pid":"0024444930","timestamp":"16:03:31.0856","sursa":"opnAAmq1.cbl","nr\_linie":"370","mesaj":" alternative nespecificate","; line: 1, column: 6]\>}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 22, 2020, 2:45pm UTC](https://discuss.elastic.co/t/multiline-json-over-tcp-is-not-parsed/249438/2 "2020-09-22T14:45:18Z")

</div>

Your data does not appear to have the format you think it has. It has syslog headers prepended to it. You will need to remove those.

---

<div class="post-metadata">

**Author:** ![cristi.mitroi](https://avatars.discourse-cdn.com/v4/letter/c/e79b87/32.png) [@cristi.mitroi](https://discuss.elastic.co/u/cristi.mitroi)\
**Post date:** [September 23, 2020, 4:26am UTC](https://discuss.elastic.co/t/multiline-json-over-tcp-is-not-parsed/249438/3 "2020-09-23T04:26:51Z")

</div>

It’s true because I receive the data from an AIX 7.1 that uses syslog-ng to send logs to the ELK server. How can i remove those headers ?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 23, 2020, 1:20pm UTC](https://discuss.elastic.co/t/multiline-json-over-tcp-is-not-parsed/249438/4 "2020-09-23T13:20:28Z")

</div>

> [@cristi.mitroi](#):
>
> 250 \<13\>1 2020-09-22T03:59:44+03:00 bndikdev {"businessId" - - - "335","techn

Use dissect

```
dissect { mapping => { "message" => "%{someNumber} <%{syslogPri}> %{timestamp} %{hostname} %{jsonData}" } }

```

then pass [jsonData] to the json filter.

---

<div class="post-metadata">

**Author:** ![cristi.mitroi](https://avatars.discourse-cdn.com/v4/letter/c/e79b87/32.png) [@cristi.mitroi](https://discuss.elastic.co/u/cristi.mitroi)\
**Post date:** [September 23, 2020, 6:00pm UTC](https://discuss.elastic.co/t/multiline-json-over-tcp-is-not-parsed/249438/5 "2020-09-23T18:00:51Z")

</div>

As instructed I added dissect:  
filter {

dissect {  
mapping =\> {  
"message" =\> "%{someNumber} \<%{syslogPri}\>1 %{timestamp} %{hostname} %{jsonData}"  
}  
}

json {  
source =\> "jsonData"  
}

mutate {  
remove\_field =\> ["someNumber"]  
}  
}  
I used the following json : {"messageType": "cagw","uniqueID": "8e760b6b7c4e937ebb7063746dd945ecee782cac","sessionID": "6c43b719e0e457784a0a6d5dfef96a82"}  
And I checked it with [https://jsonlint.com/](https://jsonlint.com/) and it was correct  
Now I get a json parse failure:  
[WARN] 2020-09-23 21:00:08.244 [[main]\>worker2] json - Error parsing json {:source=\>"jsonData", :raw=\>"{"messageType" - - - "cagw","uniqueID": "8e760b6b7c4e937ebb7063746dd945ecee782cac","sessionID": "6c43b719e0e457784a0a6d5dfef96a82"}", :exception=\>#\<LogStash::Json::ParserError: Unexpected character ('-' (code 45)): was expecting a colon to separate field name and value  
at [Source: (byte)"{"messageType" - - - "cagw","uniqueID": "8e760b6b7c4e937ebb7063746dd945ecee782cac","sessionID": "6c43b719e0e457784a0a6d5dfef96a82"}"; line: 1, column: 17]\>}  
{  
"message" =\> "174 \<13\>1 2020-09-23T21:00:08+03:00 andibkdev {"messageType" - - - "cagw","uniqueID": "8e760b6b7c4e937ebb7063746dd945ecee782cac","sessionID": "6c43b719e0e457784a0a6d5dfef96a82"}",  
"jsonData" =\> "{"messageType" - - - "cagw","uniqueID": "8e760b6b7c4e937ebb7063746dd945ecee782cac","sessionID": "6c43b719e0e457784a0a6d5dfef96a82"}",  
"hostname" =\> "andibkdev",  
"timestamp" =\> "2020-09-23T21:00:08+03:00",  
"@timestamp" =\> 2020-09-23T18:00:08.141Z,  
"host" =\> "10.188.208.34",  
"port" =\> 36362,  
"tags" =\> [  
[0] "\_jsonparsefailure"  
],  
"@version" =\> "1",  
"syslogPri" =\> "13"  
}

---

<div class="post-metadata">

**Author:** ![cristi.mitroi](https://avatars.discourse-cdn.com/v4/letter/c/e79b87/32.png) [@cristi.mitroi](https://discuss.elastic.co/u/cristi.mitroi)\
**Post date:** [September 23, 2020, 6:02pm UTC](https://discuss.elastic.co/t/multiline-json-over-tcp-is-not-parsed/249438/6 "2020-09-23T18:02:51Z")

</div>

> [@cristi.mitroi](#):
>
> "jsonData" =\> "{"messageType" - - - "cagw"

I see that the value of the json gets in elastic with "---" although it's not like that in the original json

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 23, 2020, 6:40pm UTC](https://discuss.elastic.co/t/multiline-json-over-tcp-is-not-parsed/249438/7 "2020-09-23T18:40:42Z")

</div>

> [@cristi.mitroi](#):
>
> bndikdev{"businessId" - - - "335"

Looking at your first post I see you have the same mutation in the JSON there. I have no suggestions on what could be causing that.

If you are unable to hunt it down I guess you could

```
mutate { gsub => ["jsonData", " - - -", ":"] }

```

---

<div class="post-metadata">

**Author:** ![cristi.mitroi](https://avatars.discourse-cdn.com/v4/letter/c/e79b87/32.png) [@cristi.mitroi](https://discuss.elastic.co/u/cristi.mitroi)\
**Post date:** [September 24, 2020, 2:56pm UTC](https://discuss.elastic.co/t/multiline-json-over-tcp-is-not-parsed/249438/8 "2020-09-24T14:56:40Z")

</div>

Hi Badger,

Thank you for you advices, my issues got fixed.

Cheers!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 22, 2020, 2:56pm UTC](https://discuss.elastic.co/t/multiline-json-over-tcp-is-not-parsed/249438/9 "2020-10-22T14:56:48Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.