# Dynamic Field Parsing

**URL:** <https://discuss.elastic.co/t/dynamic-field-parsing/180225>\
**Category:** Logstash\
**Created:** [May 8, 2019, 5:18pm UTC](https://discuss.elastic.co/t/dynamic-field-parsing/180225 "2019-05-08T17:18:07Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![kdotson](https://avatars.discourse-cdn.com/v4/letter/k/ed655f/32.png) [@kdotson](https://discuss.elastic.co/u/kdotson)\
**Post date:** [May 8, 2019, 5:18pm UTC](https://discuss.elastic.co/t/dynamic-field-parsing/180225/1 "2019-05-08T17:18:08Z")

</div>

Hello,  
I have a csv file which has a few fields that have quoted dynamic entries. I used the csv module to break out the fields and now I am trying to account for the embedded subfields. There are two different scenarios:

#1 The first integer in the subfield defines how many times the remaining subfieds will repeat

Example:

"MSD" =\> "1,X,Y,Z"  
or

"MSD" =\> "2,X1,Y1,Z1,X2,Y2,Z2"

Desired output:  
"MSD" =\> {  
"NS" =\> 1,  
"INPUT\_1" =\> X1,  
"OUTPUT\_1" =\> Y1,  
"MEASURE\_1" =\> Z1,  
}  
"MSD" =\> {  
"NS" =\> 2,  
"INPUT\_1" =\> X1,  
"OUTPUT\_1" =\> Y1,  
"MEASURE\_1" =\> Z1,   
"INPUT\_2" =\> X2,  
"OUTPUT\_2" =\> Y2,  
"MEASURE\_2" =\> Z2  
}

#2 The First Field provides the total number of remaining fields.

Example:  
"RD" =\> "3,A,B,C"  
or  
"RD" =\> "4,A,B,C,D"

Desired output: (All fields will have specific names, I know there is a maximum of 20)

"RD" =\> {  
"count" =\> "x"  
"Static\_field"=\>A  
...  
"Another\_Statically\_named\_fieldX =\> "x"  
}

Thanks for any push in the right direction.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 8, 2019, 6:09pm UTC](https://discuss.elastic.co/t/dynamic-field-parsing/180225/2 "2019-05-08T18:09:50Z")

</div>

> [@kdotson](#):
>
> "RD" =\> "4,A,B,C,D"

For the second use case you can use a csv filter

```
    mutate { add_field => { "RD" => "4,A,B,C,D" } }
    csv {
        source => "RD"
        target => "RDx"
        columns => ["count", "static1", "static2", "static3", "static4", "static5", "static6"]
        skip_empty_columns => true
    }
    if "_csvparsefailure" not in [tags] {
        mutate { rename => { "RDx" => "RD" } }
    }

```

For the first one I think you need ruby

```
    mutate { split => { "MSD" => "," } }
    ruby {
        code => '
            msd = event.remove("MSD")
            h = {}
            h["NS"] = msd.shift
            index = 1
            while msd.length > 0
                a = msd.shift(3)
                h["INPUT_#{index}"] = a[0]
                h["OUTPUT#{index}"] = a[1]
                h["MEASURE#{index}"] = a[2]
                index += 1
            end
            event.set("MSD", h)
        '
    }

```

---

<div class="post-metadata">

**Author:** ![kdotson](https://avatars.discourse-cdn.com/v4/letter/k/ed655f/32.png) [@kdotson](https://discuss.elastic.co/u/kdotson)\
**Post date:** [May 8, 2019, 7:38pm UTC](https://discuss.elastic.co/t/dynamic-field-parsing/180225/3 "2019-05-08T19:38:19Z")

</div>

Thank you!!! This is perfect.

---

<div class="post-metadata">

**Author:** ![kdotson](https://avatars.discourse-cdn.com/v4/letter/k/ed655f/32.png) [@kdotson](https://discuss.elastic.co/u/kdotson)\
**Post date:** [May 10, 2019, 11:01pm UTC](https://discuss.elastic.co/t/dynamic-field-parsing/180225/4 "2019-05-10T23:01:32Z")

</div>

I have one more field in the output that has a variable length based on the first and second embedded sub-fields.

The first sub-field contains the count for the number of fields after the second sub-field and the second sub-field is the count of how many times the first count will be repeated.

I know that the max number of sub-field in the loop is seven.

example 1:

Fieldname\_x =\> "3,1,A1,B1,C1"

example 2:  
Fieldname\_x =\> "3,2,A1,B1,C1,A2,B2,C3"

Example 3 (just to make sure I am clear):  
Fieldname\_x =\> "5,2,A1,B1,C1,D1,E1,A2,B2,C3,D1,E1"

Desired output:  
"Fieldname" =\> {  
"Fieldcount" =\> 3  
"loopcount" =\> 2  
"nameA\_1" =\> A1  
"nameB\_1" =\> B1  
"nameC\_1" =\> C1  
"nameA\_2" =\> A2  
"nameB\_2" =\> B2  
"nameC\_2" =\> C2  
}

This is what I have tried and I am getting an error "Ruby exception occurred: undefined method `shift'"

I am sure I need another loop, but I am no good at Ruby.

```
    ruby {
      code => '
            test = event.remove("Fieldname_x")
            h = {}
            h["Fieldcount"] = test.shift
            h["loopcount"] = test.shift
            index = 1
            while index <= h["loopcount"].to_i #had to change scalar to integer
            h["nameA_#{index}"] = test.shift
            h["nameB_#{index}"] = test.shift
            h["nameC_#{index}"] = test.shift
            h["nameD_#{index}"] = test.shift
            h["nameE_#{index}"] = test.shift
            h["nameF_#{index}"] = test.shift
            h["nameG_#{index}"] = test.shift
            index +=1
            end
            event.set("Fieldname", h)
            '
            }

```

Thanks for any help!!!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 11, 2019, 9:26pm UTC](https://discuss.elastic.co/t/dynamic-field-parsing/180225/5 "2019-05-11T21:26:11Z")

</div>

Once you add mutate+split to make Fieldname\_x an array that code works. That said, I would write it like this:

```
mutate { split => { "Fieldname_x" => "," } }
ruby {
    code => '
        test = event.remove("Fieldname_x")
        h = {}
        h["Fieldcount"] = test.shift.to_i
        h["loopcount"] = test.shift.to_i
        for i in 1..h["loopcount"]
            for j in 1..h["Fieldcount"]
                c = " ABCDEFGHIJKLMNOPQRSTUVWXYZ"[j]
                h["name#{c}_#{i}"] = test.shift
            end
        end
        event.set("Fieldname", h)
    '
}
```

---

<div class="post-metadata">

**Author:** ![kdotson](https://avatars.discourse-cdn.com/v4/letter/k/ed655f/32.png) [@kdotson](https://discuss.elastic.co/u/kdotson)\
**Post date:** [May 13, 2019, 3:07pm UTC](https://discuss.elastic.co/t/dynamic-field-parsing/180225/6 "2019-05-13T15:07:33Z")

</div>

Thank you again Badger. This was a huge help!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 10, 2019, 3:07pm UTC](https://discuss.elastic.co/t/dynamic-field-parsing/180225/7 "2019-06-10T15:07:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.