# Parse single line multi object json with logstash

**URL:** <https://discuss.elastic.co/t/parse-single-line-multi-object-json-with-logstash/207608>\
**Category:** Logstash\
**Created:** [November 13, 2019, 1:15am UTC](https://discuss.elastic.co/t/parse-single-line-multi-object-json-with-logstash/207608 "2019-11-13T01:15:10Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![sili.redwork](https://avatars.discourse-cdn.com/v4/letter/s/9e8a1a/32.png) [@sili.redwork](https://discuss.elastic.co/u/sili.redwork)\
**Post date:** [November 13, 2019, 1:15am UTC](https://discuss.elastic.co/t/parse-single-line-multi-object-json-with-logstash/207608/1 "2019-11-13T01:15:11Z")

</div>

Hello,

We are sending collecting cloudwatch logs to central log account's S3 bucket using Cloudwatch --\> Log Destination --\> Kinesis --\> S3

S3 file has multiple json objects in single line, below is the sample format

{"messageType": "DATA\_MESSAGE","owner": "owner-id","logGroup": "log-group","logStream": "log-stream","subscriptionFilters": ["Destination"],"logEvents": [{"id": "event-id","timestamp": 1573519068908,"message": "{}"},{"id": "event-id","timestamp": 1573519068908,"message": "{}"}]}{"messageType": "DATA\_MESSAGE","owner": "owner-id","logGroup": "log-group","logStream": "log-stream","subscriptionFilters": ["Destination"],"logEvents": [{"id": "event-id","timestamp": 1573518985345,"message": "{}"}]}

With below logstash configuration I can parse first json object but not subsequent objects

input {  
file {  
path =\> "sample\_masked\_log"  
codec =\> "json"  
type =\> "Log"  
sincedb\_path =\> "/tmp/sincedb"  
start\_position =\> "beginning"  
}  
}  
filter {  
json {  
source =\> "message"  
}  
split {  
field =\> "[logEvents]"  
}  
mutate {  
add\_field =\> {  
"log-event-id" =\> "%{[logEvents][id]}"  
"log-event-message" =\> "%{[logEvents][message]}"  
"log-event-time" =\> "%{[logEvents][timestamp]}"  
}  
remove\_field =\> ["[message]" ]  
remove\_field =\> ["[host]" ]  
remove\_field =\> ["[path]" ]  
remove\_field =\> ["[logEvents]" ]  
remove\_field =\> ["[subscriptionFilters]" ]  
remove\_field =\> ["[messageType]" ]  
}  
}  
output {  
stdout { codec =\> rubydebug }  
}

If I have log in multiple lines - one json object per line as below then I can get all events.

{"messageType": "DATA\_MESSAGE","owner": "owner-id","logGroup": "log-group","logStream": "log-stream","subscriptionFilters": ["Destination"],"logEvents": [{"id": "event-id","timestamp": 1573519068908,"message": "{}"},{"id": "event-id","timestamp": 1573519068908,"message": "{}"}]}  
{"messageType": "DATA\_MESSAGE","owner": "owner-id","logGroup": "log-group","logStream": "log-stream","subscriptionFilters": ["Destination"],"logEvents": [{"id": "event-id","timestamp": 1573518985345,"message": "{}"}]}

As I cannot change the source, Is there way to parse single line multi object json with logstash?

Thanks in advance for your any suggestions.

---

<div class="post-metadata">

**Author:** ![manud](https://avatars.discourse-cdn.com/v4/letter/m/edb3f5/32.png) [@manud](https://discuss.elastic.co/u/manud)\
**Post date:** [November 13, 2019, 2:10am UTC](https://discuss.elastic.co/t/parse-single-line-multi-object-json-with-logstash/207608/2 "2019-11-13T02:10:12Z")

</div>

You could try using gsub to add a newline char between the JSON objects then use the split plugin. This assumes that the objects appear in succession with no characters between them and that the `}{` sequence doesn't appear in any of the JSON values.

```auto
filter {
    mutate {
        gsub => ["message", "\}\{", "}\n{"]
    }
    split {
        field => "message"
    }
}

```

* * *

EDIT

Couldn't get the above to work but it does seem to work if you actually add a line in to the config like so:

```auto
filter {
    mutate {
        gsub => [ "message", "\}\{", "}
{" ]
    }
    split {
        field => "message"
    }
}

```

---

<div class="post-metadata">

**Author:** ![sili.redwork](https://avatars.discourse-cdn.com/v4/letter/s/9e8a1a/32.png) [@sili.redwork](https://discuss.elastic.co/u/sili.redwork)\
**Post date:** [November 13, 2019, 5:41am UTC](https://discuss.elastic.co/t/parse-single-line-multi-object-json-with-logstash/207608/3 "2019-11-13T05:41:52Z")

</div>

Thank you Manu,

**Tried your suggestion but no luck.**

**with below config**  
input {  
file {  
path =\> "/Users/sreddapani001/Desktop/Elasticsearch/SampleLogs/sample\_masked\_log"  
codec =\> "json"  
type =\> "Log"  
sincedb\_path =\> "/tmp/sincedb"  
start\_position =\> "beginning"  
}  
}  
filter {  
json {  
source =\> "message"  
}  
# mutate {  
# gsub =\> [ "message", "}{", "}  
#{" ]  
# }  
# split {  
# field =\> "message"  
# }  
split {  
field =\> "[logEvents]"  
}  
mutate {  
add\_field =\> {  
"log-event-id" =\> "%{[logEvents][id]}"  
"log-event-message" =\> "%{[logEvents][message]}"  
"log-event-time" =\> "%{[logEvents][timestamp]}"  
}  
remove\_field =\> ["[message]" ]  
remove\_field =\> ["[host]" ]  
remove\_field =\> ["[path]" ]  
remove\_field =\> ["[logEvents]" ]  
remove\_field =\> ["[subscriptionFilters]" ]  
remove\_field =\> ["[messageType]" ]  
}  
}  
output {  
stdout { codec =\> rubydebug }  
}

**I am getting two events in first json block as**

{  
"type" =\> "Log",  
"log-event-id" =\> "event-id",  
"log-event-message" =\> "{}",  
"owner" =\> "owner-id",  
"logGroup" =\> "log-group",  
"log-event-time" =\> "1573519068908",  
"logStream" =\> "log-stream",  
"@version" =\> "1",  
"@timestamp" =\> 2019-11-13T05:34:39.765Z  
}  
{  
"type" =\> "Log",  
"log-event-id" =\> "event-id",  
"log-event-message" =\> "{}",  
"owner" =\> "owner-id",  
"logGroup" =\> "log-group",  
"log-event-time" =\> "1573519068908",  
"logStream" =\> "log-stream",  
"@version" =\> "1",  
"@timestamp" =\> 2019-11-13T05:34:39.765Z  
}

**But If I uncomment your suggestion section in above configuration, I am still getting two events with "\_split\_type\_failure"**

{  
"@version" =\> "1",  
"log-event-id" =\> "event-id",  
"logStream" =\> "log-stream",  
"log-event-message" =\> "{}",  
"@timestamp" =\> 2019-11-13T05:37:44.604Z,  
"logGroup" =\> "log-group",  
"type" =\> "Log",  
"tags" =\> [  
[0] "\_split\_type\_failure"  
],  
"log-event-time" =\> "1573519068908",  
"owner" =\> "owner-id"  
}  
{  
"@version" =\> "1",  
"log-event-id" =\> "event-id",  
"logStream" =\> "log-stream",  
"log-event-message" =\> "{}",  
"@timestamp" =\> 2019-11-13T05:37:44.604Z,  
"logGroup" =\> "log-group",  
"type" =\> "Log",  
"tags" =\> [  
[0] "\_split\_type\_failure"  
],  
"log-event-time" =\> "1573519068908",  
"owner" =\> "owner-id"  
}

---

<div class="post-metadata">

**Author:** ![manud](https://avatars.discourse-cdn.com/v4/letter/m/edb3f5/32.png) [@manud](https://discuss.elastic.co/u/manud)\
**Post date:** [November 13, 2019, 7:07am UTC](https://discuss.elastic.co/t/parse-single-line-multi-object-json-with-logstash/207608/4 "2019-11-13T07:07:07Z")

</div>

Move my suggestion to before the json filter:

```auto
filter {
    mutate {
        gsub => [ "message", "}{", "}
{" ]
    }
    split {
        field => "message"
    }
    json {
        source => "message"
    }
    split {
        field => "[logEvents]"
    }
    mutate {
        add_field => {
            "log-event-id" => "%{[logEvents][id]}"
            "log-event-message" => "%{[logEvents][message]}"
            "log-event-time" => "%{[logEvents][timestamp]}"
        }
        remove_field => [ 
            "[message]",
            "[host]",
            "[path]",
            "[logEvents]",
            "[subscriptionFilters]",
            "[messageType]"
        ]
    }
}

```

---

<div class="post-metadata">

**Author:** ![sili.redwork](https://avatars.discourse-cdn.com/v4/letter/s/9e8a1a/32.png) [@sili.redwork](https://discuss.elastic.co/u/sili.redwork)\
**Post date:** [November 13, 2019, 6:43pm UTC](https://discuss.elastic.co/t/parse-single-line-multi-object-json-with-logstash/207608/5 "2019-11-13T18:43:07Z")

</div>

Thanks Manud, Still no joy.

**Configuration**

> ```
> input {
> file {
> path => "sample_masked_log"
> codec => "json"
> type => "Log"
> sincedb_path => "/tmp/sincedb"
> start_position => "beginning"
> }
> }
> filter {
> mutate {
> gsub => [ "message", "}{", "}
> {" ]
> }
> split {
> field => "message"
> }
> json {
> source => "message"
> }
> 
> split {
> field => "[logEvents]"
> }
> mutate {
> add_field => {
> "log-event-id" => "%{[logEvents][id]}"
> "log-event-message" => "%{[logEvents][message]}"
> "log-event-time" => "%{[logEvents][timestamp]}"
> }
> remove_field => [
> "[message]",
> "[host]",
> "[path]",
> "[logEvents]",
> "[subscriptionFilters]",
> "[messageType]"
> ]
> }
> }
> output {
> stdout { codec => rubydebug }
> }
> 
> ```

**Content of sample\_masked\_log file**

> {"messageType": "DATA\_MESSAGE","owner": "owner-id","logGroup": "log-group","logStream": "log-stream","subscriptionFilters": ["Destination"],"logEvents": [{"id": "event-id","timestamp": 1573519068908,"message": "{}"},{"id": "event-id","timestamp": 1573519068908,"message": "{}"}]}{"messageType": "DATA\_MESSAGE","owner": "owner-id","logGroup": "log-group","logStream": "log-stream","subscriptionFilters": ["Destination"],"logEvents": [{"id": "event-id","timestamp": 1573518985345,"message": "{}"}]}

**Output**

> {  
> "@timestamp" =\> 2019-11-13T18:34:20.892Z,  
> "logGroup" =\> "log-group",  
> "type" =\> "Log",  
> "log-event-time" =\> "1573519068908",  
> "@version" =\> "1",  
> "tags" =\> [  
> [0] "\_split\_type\_failure"  
> ],  
> "logStream" =\> "log-stream",  
> "log-event-message" =\> "{}",  
> "owner" =\> "owner-id",  
> "log-event-id" =\> "event-id"  
> }  
> {  
> "@timestamp" =\> 2019-11-13T18:34:20.892Z,  
> "logGroup" =\> "log-group",  
> "type" =\> "Log",  
> "log-event-time" =\> "1573519068908",  
> "@version" =\> "1",  
> "tags" =\> [  
> [0] "\_split\_type\_failure"  
> ],  
> "logStream" =\> "log-stream",  
> "log-event-message" =\> "{}",  
> "owner" =\> "owner-id",  
> "log-event-id" =\> "event-id"  
> }

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 13, 2019, 7:25pm UTC](https://discuss.elastic.co/t/parse-single-line-multi-object-json-with-logstash/207608/6 "2019-11-13T19:25:42Z")

</div>

If you remove all of your filters and use

```
output { stdout { codec => rubydebug } }

```

What does an event look like?

---

<div class="post-metadata">

**Author:** ![sili.redwork](https://avatars.discourse-cdn.com/v4/letter/s/9e8a1a/32.png) [@sili.redwork](https://discuss.elastic.co/u/sili.redwork)\
**Post date:** [November 13, 2019, 8:19pm UTC](https://discuss.elastic.co/t/parse-single-line-multi-object-json-with-logstash/207608/7 "2019-11-13T20:19:45Z")

</div>

Hi Badger,

**Removed all filters, with below config**

> input {  
> file {  
> path =\> "/Users/sreddapani001/Desktop/Elasticsearch/SampleLogs/sample\_masked\_log"  
> codec =\> "json"  
> type =\> "Log"  
> sincedb\_path =\> "/tmp/sincedb"  
> start\_position =\> "beginning"  
> }  
> }  
> output {  
> stdout { codec =\> rubydebug }  
> }

**Output**

> {  
> "subscriptionFilters" =\> [  
> [0] "Destination"  
> ],  
> "@version" =\> "1",  
> "owner" =\> "owner-id",  
> "logGroup" =\> "log-group",  
> "path" =\> "/Users/sreddapani001/Desktop/Elasticsearch/SampleLogs/sample\_masked\_log",  
> "host" =\> "AU\_C02T92HMG8WN",  
> "type" =\> "Log",  
> "logEvents" =\> [  
> [0] {  
> "timestamp" =\> 1573519068908,  
> "id" =\> "event-id",  
> "message" =\> "{}"  
> },  
> [1] {  
> "timestamp" =\> 1573519068908,  
> "id" =\> "event-id",  
> "message" =\> "{}"  
> }  
> ],  
> "logStream" =\> "log-stream",  
> "messageType" =\> "DATA\_MESSAGE",  
> "@timestamp" =\> 2019-11-13T20:17:42.077Z  
> }

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 13, 2019, 9:03pm UTC](https://discuss.elastic.co/t/parse-single-line-multi-object-json-with-logstash/207608/8 "2019-11-13T21:03:51Z")

</div>

The codec parsed your JSON. You should be able to

```
split { field => "logEvents" }

```

if you want to.

---

<div class="post-metadata">

**Author:** ![sili.redwork](https://avatars.discourse-cdn.com/v4/letter/s/9e8a1a/32.png) [@sili.redwork](https://discuss.elastic.co/u/sili.redwork)\
**Post date:** [November 13, 2019, 11:54pm UTC](https://discuss.elastic.co/t/parse-single-line-multi-object-json-with-logstash/207608/9 "2019-11-13T23:54:09Z")

</div>

Hi Badger,

source file has three LogEvents - highlighted with **bold**

> {"messageType": "DATA\_MESSAGE","owner": "owner-id","logGroup": "log-group","logStream": "log-stream","subscriptionFilters": ["Destination"],"logEvents": [**{"id": "event-id","timestamp": 1573519068908,"message": "{}"}** , **{"id": "event-id","timestamp": 1573519068908,"message": "{}"}**]}{"messageType": "DATA\_MESSAGE","owner": "owner-id","logGroup": "log-group","logStream": "log-stream","subscriptionFilters": ["Destination"],"logEvents": [**{"id": "event-id","timestamp": 1573518985345,"message": "{}"}**]}

But I am getting output only two messages - those are part of first json block, parser is not looking into second json block.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 14, 2019, 12:37am UTC](https://discuss.elastic.co/t/parse-single-line-multi-object-json-with-logstash/207608/10 "2019-11-14T00:37:50Z")

</div>

> [@sili.redwork](#):
>
> But I am getting output only two messages - those are part of first json block, parser is not looking into second json block.

Actually I only get one, which is the last one. I believe this occurs because it is parsing the JSON into a hash, and if it sees a duplicate key it simply overwrites the first hash entry.

I suggest you post a new question where you are more explicit about there being duplicate logEvents keys in a single JSON object. I did not get that from the question and I don't think Manu did either.

---

<div class="post-metadata">

**Author:** ![sili.redwork](https://avatars.discourse-cdn.com/v4/letter/s/9e8a1a/32.png) [@sili.redwork](https://discuss.elastic.co/u/sili.redwork)\
**Post date:** [November 14, 2019, 1:24am UTC](https://discuss.elastic.co/t/parse-single-line-multi-object-json-with-logstash/207608/11 "2019-11-14T01:24:29Z")

</div>

Thanks Badger, I don't believe this is related to duplicate logevents.

I have added some randomness to source(in bold), I still have same issue.

> {"messageType": "DATA\_MESSAGE","owner": "owner-id","logGroup": "log-group","logStream": "log-stream","subscriptionFilters": ["Destination"],"logEvents": [{"id": "event-id","timestamp": 1573519068 **901** ,"message": **"{"event":"1.0"}"** },{"id": "event-id","timestamp": 1573519068908,"message": **"{"event":"2.0"}"** }]}{"messageType": "DATA\_MESSAGE","owner": "owner-id","logGroup": "log-group","logStream": "log-stream","subscriptionFilters": ["Destination"],"logEvents": [{"id": "event-id","timestamp": 1573518985345,"message": **"{"event":"3.0"}"** }]}

---

<div class="post-metadata">

**Author:** ![manud](https://avatars.discourse-cdn.com/v4/letter/m/edb3f5/32.png) [@manud](https://discuss.elastic.co/u/manud)\
**Post date:** [November 14, 2019, 8:17pm UTC](https://discuss.elastic.co/t/parse-single-line-multi-object-json-with-logstash/207608/12 "2019-11-14T20:17:54Z")

</div>

I missed that you had the JSON codec in effect. I tested my solution without the JSON codec as we want to split apart the distinct JSON objects before any deserialisation. You get a \_split\_type\_failure because there's no message field to split on if you parse with the codec.

If you remove the codec so that at the start of the `filter` block gets all objects as a single line string under the message field, then the gsub action will make it a multiline field and split should work correct.

I think @Badger is correct in saying that the JSON codec is parsing each of the objects and overwriting the fields each time.

---

<div class="post-metadata">

**Author:** ![sili.redwork](https://avatars.discourse-cdn.com/v4/letter/s/9e8a1a/32.png) [@sili.redwork](https://discuss.elastic.co/u/sili.redwork)\
**Post date:** [November 14, 2019, 11:59pm UTC](https://discuss.elastic.co/t/parse-single-line-multi-object-json-with-logstash/207608/13 "2019-11-14T23:59:44Z")

</div>

Thank you both @manud and @Badger,

Yes it was json codec in input section messing it up. I managed to make it work with below config

```
input {
	file {
		path => "/Users/sreddapani001/Desktop/Elasticsearch/SampleLogs/sample_masked_log"
# codec => "json"
# type => "Log"
		sincedb_path => "/tmp/sincedb"
		start_position => "beginning"
	}
}

filter {
	mutate {
        gsub => [ "message", "}{", "}
{" ]
    	}
    	split {
        	field => "message"
    	}

	json {
		source => "message"
	}	

	split {
		field => "[logEvents]"
	}	
	mutate {
		add_field => {
			"log-event-id" => "%{[logEvents][id]}"
			"log-event-message" => "%{[logEvents][message]}"
			"log-event-time" => "%{[logEvents][timestamp]}"
		}
		remove_field => [ 
			"[host]",
			"[path]",
			"[message]",
			"[logEvents]",
			"[subscriptionFilters]",
			"[messageType]" 
		]
	}
}

output {
stdout { codec => rubydebug }
}

```

I can see three events in output

```
{
             "@version" => "1",
           "@timestamp" => 2019-11-14T23:53:07.517Z,
             "logGroup" => "log-group",
            "logStream" => "log-stream",
    "log-event-message" => "{\"eventVersion\":\"1.05\"}",
       "log-event-time" => "1573519068901",
         "log-event-id" => "event-id",
                "owner" => "owner-id"
}
{
             "@version" => "1",
           "@timestamp" => 2019-11-14T23:53:07.517Z,
             "logGroup" => "log-group",
            "logStream" => "log-stream",
    "log-event-message" => "{\"eventVersion\":\"2.05\"}",
       "log-event-time" => "1573519068908",
         "log-event-id" => "event-id",
                "owner" => "owner-id"
}
{
             "@version" => "1",
           "@timestamp" => 2019-11-14T23:53:07.517Z,
             "logGroup" => "log-group",
            "logStream" => "log-stream",
    "log-event-message" => "{\"eventVersion\":\"3.05\"}",
       "log-event-time" => "1573518985345",
         "log-event-id" => "event-id",
                "owner" => "owner-id"
}

```

Thank you again both of you much appreciated for your help.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 12, 2019, 11:59pm UTC](https://discuss.elastic.co/t/parse-single-line-multi-object-json-with-logstash/207608/14 "2019-12-12T23:59:47Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.