File input JSON with multiline codec plugin

Elastic Stack Logstash
1

Hello,

I'm simply trying to parse my JSON fileds into events.

The JSON file look like this:

[
{
"field1": "value1",
"field2": "value2",
"field3": "value3",
"field4": "value4",
"field5": "value5",
"field6": "value6"
},
{
"field1": "value1",
"field2": "value2",
"field3": "value3",
"field4": "value4",
"field5": "value5",
"field6": "value6"
}
]

I would like to parse it to have one event of the 6 fields, with their value.

So I did some research and find that I have to use the Multiline codec plugin with the pattern option. I tried a lot of pattern but I have a parse error everytime.

[WARN ][logstash.filters.json ] Error parsing json {:source=>"message", :raw=>" },\r", :exception=>#<LogStash::Json::ParserError:"; line: 1, column: 0])r '}': expected ']' (for root starting at [Source: (byte)" },

My input look like this:

input {
file {
codec => multiline {
pattern => "^\s\s\s\s"
negate => true
what => previous
max_lines => 20000
}
path => "/etc/logstash/conf.d/adcsi/MyJson.json"
type => "json"
sincedb_path => "/dev/null"
start_position => "beginning"
}

My filter:

filter {
if [type] == "json" {
json {
source => "message"
}
}
}

Any help would be appreciate !

2

Take a look at this.

3

Thank you for your response, I will try this today and inform you if it worked !

Have a great day

4

Hello Badger,

Your solution doesn't seem to work.

Here is the error I get :

[WARN ][logstash.filters.split ] Only String and Array types are splittable. field:someField is of type = NilClass

Here is my input:

file {
     codec => multiline { 
     pattern => "^Spalanzani" 
	 negate => true 
	 what => previous 
	 auto_flush_interval => 1 
	 multiline_tag => "" 
	 }
    path => "/etc/logstash/conf.d/adcsi/Users.json"
    type => "adcsi-users"
    sincedb_path => "/dev/null"
    start_position => "beginning"
  }

Here is my filter for this json:

filter {
if [type] == "adcsi-users"{
json {
source => "message"
target => "someField"
remove_field => [ "message" ]
}
split {
field => "someField"
}
}
}

Do I need to change the format of my Json to make a JSON Lines ? Where every line is an object ? If so, I have 2000+ lines to change, is there any other solution that doesn't need to reformat my json ?

Thank you for your help.

Best regards

5

That is telling you that [someField] does not exist, which means your json filter did not parse [message].

6

Yes... That's the problem...

It did not solve my problem, I really don't understand what I have to do to parse my json correctly...

7

I haven't find a solution for my Json.

My solution was to convert my JSON into a CSV file. Now works properly...

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.