Logstash dynamically create field name

Elastic Stack Logstash
1

this is my input
{"dataflow":[{"gate": "v204.06", "attrmath":"v206.07"}]}

I want output like
dataflow.m1: gate
dataflow.m1_version: v204.06
dataflow.m2: attrmath
dataflow.m2_version: v206.07

I want this m1, m2, m3 .... dynamically created as this might have 20+ such k/v pair no exact number

how do I do this. I am trying this

if [dataflow] {
   ruby {
     code => '
         event.get("[dataflow][0]").each { |k,v|
           event.set("dataflow.m", k)
           event.set("dataflow.m_version", v)
         }
         event.remove("dataflow")
   '
   }
}

but how do I make m to m1, m2 m3.....

2

I would try something like

if [dataflow] {
   ruby {
     code => '
         index = 1
         event.get("[dataflow][0]").each { |k,v|
           event.set("dataflow.m#{index}", k)
           event.set("dataflow.m#{index}_version", v)
           index += 1
         }
         event.remove("dataflow")
   '
   }
}
1 Like
3

someone just change this format of message, now I have array but fixed length

dataflow":[{"name":"out","version":"v211.01","Instances":1346560,"elapsed":"0.23"},{"name":"attributes","version":"v207.05","Instances":1346560,elapsed":"0.06"}]

now I have to loop through this array and this k,v pair

output should be
dataflow.m1=out
dataflow.m1_versin=v211.01
dataflow.m1_instance=1346560
dataflow.m1_elapsed=0.23

dataflow.m2=attributes .........

as you can see I want only first key as value, all other key are not usable.

sorry for confusing out whole thing.

4

now I want to rotate in this dataflow[0], dataflow[1], dataflow[2] to the end of array
and further loop inside for each k,v pair

if [dataflow] {
   ruby {
     code => '
         [dataflow].each_with_index { |value,index|
            event.get(value).each { |k, v|
               event.set("dataflow_m#{index}_#{k}", v)
            }
         }
         event.remove("dataflow")
   '
   }
}

this should give me something like
dataflow.m0_name=out
dataflow.m0_version=v211.01
dataflow.m0_instance=1346560
dataflow.m0_elapsed=0.23

but it does not. gives ruby exception.

5

What is the exception?

6

sorry I meant to say it was not assigning value.

here is new code that I have comeup with lot of testing
I just simplify this. so I will get

dataflow.m0_name=out
dataflow.m0_version=v211.01
dataflow.m0_instance=1346560
dataflow.m0_elapsed=0.23

if [dataflow] {
       ruby {
         code => '
             event.get("dataflow").each { |item|    ---> work till here, as I can print item which is whole array
                  index=1
                  event.get("item").each { |k, v|
                       event.set("dataflow_m#{index}_#{k}", v)
                }
                index += 1 
             }
       '
       }
    }

something I am missing which is not creating new field.

7

You do not have a field called item, so this will not work.

8

it actual works. here is test. but anotyher loop with k,v is not working

if [dataflow] {
      ruby {
         code => '
             index=1
             event.get("dataflow").each { |item|
                event.set("flag_#{index}", item["name"])
                index += 1
             }
       '
       }
    }

    output
             "flag_2" => "attributes",
             "flag_1" => "out",
9

@Badger

I try this and divided this, as test and I got what I need. but I do not know how many array element will I have in dataflow, hence I can't delete them or run iterate on second for loop

any other recommendation?

#### this code is dividing array in to { key,value) pairs
         ruby {
             code => '
                 event.get("dataflow").each_with_index { |val,index |
                     event.set("whatisthis_#{index}", val)
                 }
           '
           }

### this one creating new field depending on key and value
        ruby {
           code => '
              for i in 0..1
                 event.get("whatisthis_#{i}").each { |k, v|
                    event.set("dataflow_m#{i}_#{k}", v)
                 }
                event.remove("whatisthis_#{i}")
             end
           '
        }
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.