How to add field to every element from an array with logstash ruby

Elastic Stack Logstash
1

Hi

I have an array such like this:

"step": [
      "> MESSAGE AUTOMATE 03/08/2020 18:39:01 : POUR LA JOURNEE DU 20200731 L'ETAPE UNIP A PRIS 00H00",
      "> MESSAGE AUTOMATE 03/08/2020 18:39:01 : POUR LA JOURNEE DU 20200731 L'ETAPE UNIC A PRIS 00H00",
      "> MESSAGE AUTOMATE 03/08/2020 18:39:01 : POUR LA JOURNEE DU 20200731 L'ETAPE CHGFI1 A PRIS 00H00",
      "> MESSAGE AUTOMATE 03/08/2020 18:39:01 : POUR LA JOURNEE DU 20200731 L'ETAPE RCOU A PRIS 00H00",
      "> MESSAGE AUTOMATE 03/08/2020 18:42:35 : POUR LA JOURNEE DU 20200731 L'ETAPE RCAP A PRIS 00H12",
      "> MESSAGE AUTOMATE 03/08/2020 18:42:35 : POUR LA JOURNEE DU 20200731 L'ETAPE SDEC A PRIS 00H00",
      "> MESSAGE AUTOMATE 03/08/2020 18:50:51 : POUR LA JOURNEE DU 20200731 L'ETAPE TRI A PRIS 00H08",
      "> MESSAGE AUTOMATE 03/08/2020 18:50:51 : POUR LA JOURNEE DU 20200731 L'ETAPE RADC A PRIS 00H00",
      "> MESSAGE AUTOMATE 03/08/2020 18:50:51 : POUR LA JOURNEE DU 20200731 L'ETAPE AFFI A PRIS 00H00",
      "> MESSAGE AUTOMATE 03/08/2020 18:50:51 : POUR LA JOURNEE DU 20200731 L'ETAPE RSFI A PRIS 00H00",
      "> MESSAGE AUTOMATE 03/08/2020 18:50:51 : POUR LA JOURNEE DU 20200731 L'ETAPE CHGFI3 A PRIS 00H00",
      "> MESSAGE AUTOMATE 03/08/2020 18:51:02 : POUR LA JOURNEE DU 20200731 L'ETAPE RCIM A PRIS 00H01",
      "> MESSAGE AUTOMATE 03/08/2020 18:51:02 : POUR LA JOURNEE DU 20200731 L'ETAPE AFFC A PRIS 00H00"
    ]

my requirement is add field to every element from this array because i want to apply a grok filter to every line

this my logstash.cong :

filter {



kv {
    source => "message"
    field_split => "\n"
 }





ruby {
       code =>   
                  "etape =[]
                  
                   
                  
                  
                 taches = event.get('/soft/rilprdaglm/scripts/wfc_automate_genere_G_WFL10.sos ')
                  taches.each {  | item  | 
                  if item.include? ('ETAPE')
                       
                       etape.push(item)
                     

                    
                  end

                 event.set('step',etape) 


                    
          } 
 "
                  

                  
                  

                 
    }

how can i add a field to every item ? please help me because i have no solution, i'm actually blocked

2

It is really unclear what you want to do, so let me make a couple of general points.

Firstly, you can tell grok to match against an array. If we start off with a file containing

{
"step": [
      "> MESSAGE AUTOMATE 03/08/2020 18:39:01 : POUR LA JOURNEE DU 20200731 L'ETAPE UNIP A PRIS 00H00",
      "random text",
      "> MESSAGE AUTOMATE 03/08/2020 18:51:02 : POUR LA JOURNEE DU 20200731 L'ETAPE AFFC A PRIS 00H00"
    ]
}

Then we can consume the file and parse it

input {
    file {
        path => "/home/user/test.txt" 
        sincedb_path => "/dev/null" 
        start_position => beginning 
        codec => multiline { pattern => "^Spalanzani" negate => true what => previous auto_flush_interval => 1 multiline_tag => "" }
    }
}
filter { json { source => "message" remove_field => [ "message" ] } }

This gets us an event that contains this field

      "step" => [
    [0] "> MESSAGE AUTOMATE 03/08/2020 18:39:01 : POUR LA JOURNEE DU 20200731 L'ETAPE UNIP A PRIS 00H00",
    [1] "random text",
    [2] "> MESSAGE AUTOMATE 03/08/2020 18:51:02 : POUR LA JOURNEE DU 20200731 L'ETAPE AFFC A PRIS 00H00"
],

a grok against that

    grok {
        pattern_definitions => { "TIMESTAMP" => "%{MONTHNUM}/%{MONTHDAY}/%{YEAR} %{TIME}" }
        match => { "step" => "%{TIMESTAMP:timestamp}" }
    }

produces this

 "timestamp" => [
    [0] "03/08/2020 18:39:01",
    [1] "03/08/2020 18:51:02"
]

Secondly, a more compact to remove elements from an array based on a regexp is

code => '
    step = event.get("step")
    if step.is_a? Array
        event.set("step", step.delete_if { |x| x !~ /ETAPE/ })
    end
'

which will change the [step] field to

      "step" => [
    [0] "> MESSAGE AUTOMATE 03/08/2020 18:39:01 : POUR LA JOURNEE DU 20200731 L'ETAPE UNIP A PRIS 00H00",
    [1] "> MESSAGE AUTOMATE 03/08/2020 18:51:02 : POUR LA JOURNEE DU 20200731 L'ETAPE AFFC A PRIS 00H00"
],

It is possible you want to use a split filter to create a new event for each entry in the array.

I am unclear where '/soft/rilprdaglm/scripts/wfc_automate_genere_G_WFL10.sos ' comes from. Do you really have a field called that?

3

Thank you for your answer ! i see that you didn't understand my requirement i already have step field which contains a table of messages.I want to parse the table and add a field to every line of the table for example : "step 1 " : "MESSAGE AUTOMATE 03/08/2020 18:39:01 : POUR LA JOURNEE DU 20200731 L'ETAPE UNIP A PRIS 00H00"
you mentionned that i can use a [split filter] to create a new event for each entry in the array. how can i do it please
i wish that i make it clear to you hope that you can help me

4

OK, still not clear on the requirement, but it sounds like you want to convert each member of the array into a hash. I have not tested this code, but if you start with

  "step" => [
[0] "> MESSAGE AUTOMATE 03/08/2020 18:39:01 : POUR LA JOURNEE DU 20200731 L'ETAPE UNIP A PRIS 00H00",
[1] "> MESSAGE AUTOMATE 03/08/2020 18:51:02 : POUR LA JOURNEE DU 20200731 L'ETAPE AFFC A PRIS 00H00"
],

then the following ruby filter will do that

ruby {
    code => '
        step = event.get("step")
        if step.is_a? Array
            newStep = []
            step.each_index { |x|
                newStep << { "step#{x}" => step[x] }
            }
            event.set("step", newStep)
        end
      '
}

that would result in

  "step" => [
[0] { "step0" => "> MESSAGE AUTOMATE 03/08/2020 18:39:01 : POUR LA JOURNEE DU 20200731 L'ETAPE UNIP A PRIS 00H00" },
[1] { "step1" => "> MESSAGE AUTOMATE 03/08/2020 18:51:02 : POUR LA JOURNEE DU 20200731 L'ETAPE AFFC A PRIS 00H00" }
],

With a split filter you could use

split { field => "step" }

which would give you two events, one containing

  "step" => "> MESSAGE AUTOMATE 03/08/2020 18:39:01 : POUR LA JOURNEE DU 20200731 L'ETAPE UNIP A PRIS 00H00"

and the other containing

"step" => "> MESSAGE AUTOMATE 03/08/2020 18:51:02 : POUR LA JOURNEE DU 20200731 L'ETAPE AFFC A PRIS 00H00"

If you try to combine both you would get the first event containing

  "step" => { "step0" => "> MESSAGE AUTOMATE 03/08/2020 18:39:01 : POUR LA JOURNEE DU 20200731 L'ETAPE UNIP A PRIS 00H00" } }

But in that case in some document you would have to reference [step][step0], and in another [step][step1]. A different field name for every event from the same initial event. I cannot imagine a use-case where that would be desireable.

5

Thank you a lot it helps me to solve my problem

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.