How to parse jason in logstash

Elastic Stack Logstash
1

Hi All,

I am able to parse simple json file which is having one doc in one line
ex:
{"name":"name1",address1:"address1"}
{"name2":"name2", "address2" : "address2"}

but If I am having json format like below how can I parse that

[{"firstname":"name1","fastname":"name2"}][{"address1":"address1","address2":"address2"}]

any idea ?

Thanks :smiley:

2

I'd try grokking the two [] sets as json1 and json2 and then combine them after that, if you want them in a single message that is.

3

Thanks @warkolm
But the problem is when I am trying to parse the simple json with once doc/line then its working fine..but when json file having two or more doc/line
like :
[{"name":"name1",address1:"address1"}][{"name":"name1",address1:"address1"}]
It is reading the first json and without reading the second json doc , it is directly coming to the second line..

I am using simple config file

input
{
file
{
path => "G:/data/sample.json"
codec => "json"
start_position => "beginning"
sincedb_path => "G:/data/data1.sincedb"
}
}
filter
{
json {

source => "message"
}

}
output
{
stdout { codec => rubydebug }
}

4

Yes, we understand the problem. And our suggestion is that you use a grok filter to split the line in two fields that each can be parsed as valid JSON.

filter {
  grok {
    match => [
      "message",
      "^\[%{GREEDYDATA:json1}\]\[%{GREEDYDATA:json2}\]$"
    ]
  }
  json {
    source => "json1"]
    remove_field => ["json1"]
  }
  json {
    source => "json2"]
    remove_field => ["json2"]
  }
}

This assumes that the sequence "][" never appears inside the JSON data. It also only works for lines with two arrays with a single JSON object each.

5

Thank you @mangnusbaeck

one query is that : I have a json file having around 1000 records, at the end it does not contain new line character , so Logstash is not reading anything till the time I am explicitly putting an "ENTER (key)" there.
how can I overcome with this problem ?

Thanks

6

I don't think there's a way around that, at least not if you want the stateful file reading (i.e. it can be interrupted and will continue where it left off) that you get with the file input. If you can use the exec input you can just cat the file and add an extra newline.

7

Hi Magnus,

Is there any way that , while reading the file I can put end line character from logstash only ?
Because it would be very difficult to put end line character in file manually in production environment .

Thanks

8

I have nothing to add to what I wrote before.