How to parse mix json logs

#21

Test the format before parsing...

    dissect { mapping => { "message" => "%{ts} %{+ts} | %{restOfLine}" } }
    if [restOfLine] =~ /{.*}/ {
        json { source => "restOfLine" }
    } else if [restOfLine] =~ /\[.*\]/ {
        mutate { gsub => [ "restOfLine", "^\[", "", "restOfLine", "\]$", "" ] }
        csv { source => "restOfLine" }
    } else {
        # Handle other format
    }
(Amol) #22

Thank you so much for your kind help its working.but i am getting mutiple fields named column in kibana why this is so,could you please explain me.i am adding screen short of it here

if it is not necessary could we remove this.

#23

I assumed you wanted to parse that as a csv. If you do not supply the column names the csv will generate them.

(Amol) #24

ok got it,Thank you Bader.

(Amol) #25

hi Badger,
Now I wanted to parse this as csv and want to provide column names for each value which are separated by comma here

V4,0833364F,533.330,0,0,533.330,0,0,0,0,-0.849,0,0,-0.849,628.064,0,0,628.064,432.013,
431.847,433.645,430.547,249.423,247.824,251.089,249.355,1.055,0,0,3.164,49.975,38982176.000,0
,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,19/02/03,23:52:1234:

Thank you please help

#26

OK, so use the columns option on the csv filter.

(Amol) #27

Yes I can but the problem is values are not fixed in size they may vary.how do I create cloumn names dynamically as per the number values seperated by comma.

Or do I map only few values with same number of columns like I want to map first two values and last one value which is in date formats,and it should come up with date data type in kibana

#28

If you require a different number of columns for different events then you could use a conditional to decide which csv filter to use. Something like

if [message] =~ /.*,.*,.*,.*,.*/
    csv {} # events with 5 fields
else if [message] =~ /.*,.*,.*,.*/
    csv {} # events with 4 fields
[...]
(system) closed #29

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.