Set default value for a field in Logstash if the field is null or if it doesn't exist

[adrianolimit]

Hello,

I would like to know if it would be possible to set default value for a field in Logstash if the field is null or if it doesn't exist.

In my example, I have a "dead_letter_queue" input where I can have one, two or three fields :

A | B | C
A | B
A | C
...

Now, I want to write a "dead_letter_queue" file where I will write (for the example above) :

A|B|C
A|B|
A||C

To do that, I created this config file :

input {
  dead_letter_queue {
    path => "/usr/share/logstash/data/dead_letter_queue" 
    pipeline_id => "main" 
    commit_offsets => true
  }
}
output {
  file {
    path => "/path_to_my_file/dead_letter_queue.log"
    codec => line {
      format => "%{A}|%{B}|%{C}"
    }
  }
}

However, the output file looks like this :

A|B|C
A|B|%{C}
A|%{B}|C

I could check each field in a filter and if the field doesn't exist I could create it with the value "", but it will not be very optimal if I have 20 fields or more (I will have 20 "if" condifitions).

My question is : Is it possible to set a default value in Logstash if the field doesn't exist or if its value is null? Something like this?
format => "%{A:default_value}|%{B:default_value}|%{C:default_value}"

Thank you in advance for your help.

[paz]

There is no easy blanket way to do so in Logstash to my knowledge, but you can probably do that with some custom Ruby code.

Do you want to print out only available fields, or include all possible fields regardless if they have a value or not (and substitute nulls with an empty string)? Both are easily feasible, e.g. the latter option can be done by this:

    ruby {
        init => "
            $fields = ['A','B','C','D']
        "
        code => "
            $fields.each {|k| event.set(k,'') if event.get(k) == nil}
        "
    }

[adrianolimit]

Thank you for your solution, it's a good option for my use case. The perfect option would be to set the default value (if the field doesn't exist or it is null) in the output section like I described above, but I think it's not possible right now, maybe in a future release

Thank you again for your help and your solution