Dynamic Field Parsing

Elastic Stack Logstash
1

Hello,
I have a csv file which has a few fields that have quoted dynamic entries. I used the csv module to break out the fields and now I am trying to account for the embedded subfields. There are two different scenarios:

#1 The first integer in the subfield defines how many times the remaining subfieds will repeat

Example:

"MSD" => "1,X,Y,Z"
or

"MSD" => "2,X1,Y1,Z1,X2,Y2,Z2"

Desired output:
"MSD" => {
"NS" => 1,
"INPUT_1" => X1,
"OUTPUT_1" => Y1,
"MEASURE_1" => Z1,
}
"MSD" => {
"NS" => 2,
"INPUT_1" => X1,
"OUTPUT_1" => Y1,
"MEASURE_1" => Z1,
"INPUT_2" => X2,
"OUTPUT_2" => Y2,
"MEASURE_2" => Z2
}

#2 The First Field provides the total number of remaining fields.

Example:
"RD" => "3,A,B,C"
or
"RD" => "4,A,B,C,D"

Desired output: (All fields will have specific names, I know there is a maximum of 20)

"RD" => {
"count" => "x"
"Static_field"=>A
...
"Another_Statically_named_fieldX => "x"
}

Thanks for any push in the right direction.

2

For the second use case you can use a csv filter

    mutate { add_field => { "RD" => "4,A,B,C,D" } }
    csv {
        source => "RD"
        target => "RDx"
        columns => [ "count", "static1", "static2", "static3", "static4", "static5", "static6" ]
        skip_empty_columns => true
    }
    if "_csvparsefailure" not in [tags] {
        mutate { rename => { "RDx" => "RD" } }
    }

For the first one I think you need ruby

    mutate { split => { "MSD" => "," } }
    ruby {
        code => '
            msd = event.remove("MSD")
            h = {}
            h["NS"] = msd.shift
            index = 1
            while msd.length > 0
                a = msd.shift(3)
                h["INPUT_#{index}"] = a[0]
                h["OUTPUT#{index}"] = a[1]
                h["MEASURE#{index}"] = a[2]
                index += 1
            end
            event.set("MSD", h)
        '
    }
3

Thank you!!! This is perfect.

4

I have one more field in the output that has a variable length based on the first and second embedded sub-fields.

The first sub-field contains the count for the number of fields after the second sub-field and the second sub-field is the count of how many times the first count will be repeated.

I know that the max number of sub-field in the loop is seven.

example 1:

Fieldname_x => "3,1,A1,B1,C1"

example 2:
Fieldname_x => "3,2,A1,B1,C1,A2,B2,C3"

Example 3 (just to make sure I am clear):
Fieldname_x => "5,2,A1,B1,C1,D1,E1,A2,B2,C3,D1,E1"

Desired output:
"Fieldname" => {
"Fieldcount" => 3
"loopcount" => 2
"nameA_1" => A1
"nameB_1" => B1
"nameC_1" => C1
"nameA_2" => A2
"nameB_2" => B2
"nameC_2" => C2
}

This is what I have tried and I am getting an error "Ruby exception occurred: undefined method `shift'"

I am sure I need another loop, but I am no good at Ruby.

    ruby {
      code => '
            test = event.remove("Fieldname_x")
            h = {}
            h["Fieldcount"] = test.shift
            h["loopcount"] = test.shift
            index = 1
            while index <= h["loopcount"].to_i        #had to change scalar to integer
            h["nameA_#{index}"] = test.shift
            h["nameB_#{index}"] = test.shift
            h["nameC_#{index}"] = test.shift
            h["nameD_#{index}"] = test.shift
            h["nameE_#{index}"] = test.shift
            h["nameF_#{index}"] = test.shift
            h["nameG_#{index}"] = test.shift
            index +=1
            end
            event.set("Fieldname", h)
            '
            }

Thanks for any help!!!

5

Once you add mutate+split to make Fieldname_x an array that code works. That said, I would write it like this:

mutate { split => { "Fieldname_x" => "," } }
ruby {
    code => '
        test = event.remove("Fieldname_x")
        h = {}
        h["Fieldcount"] = test.shift.to_i
        h["loopcount"] = test.shift.to_i
        for i in 1..h["loopcount"]
            for j in 1..h["Fieldcount"]
                c = " ABCDEFGHIJKLMNOPQRSTUVWXYZ"[j]
                h["name#{c}_#{i}"] = test.shift
            end
        end
        event.set("Fieldname", h)
    '
}
6

Thank you again Badger. This was a huge help!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.