Conditional for empty field


(Lukas Tilch) #1

Hello I'm trying to parse a date in a field which sometimes contains no data

I'm getting this error log when I try to parse it.
[2017-06-01T16:43:23,036][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["update", {:_id=>"7345936", :_index=>"logstash-issue", :_type=>"issue", :_routing=>nil, :_retry_on_conflict=>1}, 2017-06-01T14:43:18.681Z Lukass-MBP.fritz.box %{message}], :response=>{"update"=>{"_index"=>"logstash-issue", "_type"=>"issue", "_id"=>"7345936", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [PlannedCompletionDate]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"\""}}}}}

That leads to the Data not being indexed. So I thought of checking beforehand if the field contains anything.

i've tried conditionals like
if !([PlannedCompletionDate]=~/.+/) { date {...} }

if [PlannedCompletionDate] != ~/.+/ { date {...} }

if [PlannedCompletionDate] != "" { date {...} }

none of which worked at all.

any Ideas?
Thanks in advance


(Magnus B├Ąck) #2

The last conditional should work. Please use a stdout { codec => rubydebug } output instead of your elasticsearch output so we can see exactly what your event looks like.

But the problem here isn't that the date filter is getting told to parse an empty string, it's that Elasticsearch is given an empty string. Perhaps you should delete the PlannedCompletionDate field if it's empty?


(Lukas Tilch) #3

thank you,

that was in fact why elasticsearch threw the error.

rubydebug dind't show any errors, but I didn't quite make the connection that the Format might be the problem for ES

solved it like this.

thanks again
:slight_smile:

if [PlannedCompletionDate] != "" {
    date {
      match => ["PlannedCompletionDate", "MM/dd/YYYY"]
      target => "PlannedCompletionDate"
    }
    }

    if [PlannedCompletionDate] == "" {
      mutate{
      remove_field => ["PlannedCompletionDate"]
    }
    }

(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.