Conditional for empty field

Hello I'm trying to parse a date in a field which sometimes contains no data

I'm getting this error log when I try to parse it.
[2017-06-01T16:43:23,036][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["update", {:_id=>"7345936", :_index=>"logstash-issue", :_type=>"issue", :_routing=>nil, :_retry_on_conflict=>1}, 2017-06-01T14:43:18.681Z %{message}], :response=>{"update"=>{"_index"=>"logstash-issue", "_type"=>"issue", "_id"=>"7345936", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [PlannedCompletionDate]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"\""}}}}}

That leads to the Data not being indexed. So I thought of checking beforehand if the field contains anything.

i've tried conditionals like
if !([PlannedCompletionDate]=~/.+/) { date {...} }

if [PlannedCompletionDate] != ~/.+/ { date {...} }

if [PlannedCompletionDate] != "" { date {...} }

none of which worked at all.

any Ideas?
Thanks in advance

The last conditional should work. Please use a stdout { codec => rubydebug } output instead of your elasticsearch output so we can see exactly what your event looks like.

But the problem here isn't that the date filter is getting told to parse an empty string, it's that Elasticsearch is given an empty string. Perhaps you should delete the PlannedCompletionDate field if it's empty?


thank you,

that was in fact why elasticsearch threw the error.

rubydebug dind't show any errors, but I didn't quite make the connection that the Format might be the problem for ES

solved it like this.

thanks again

if [PlannedCompletionDate] != "" {
    date {
      match => ["PlannedCompletionDate", "MM/dd/YYYY"]
      target => "PlannedCompletionDate"

    if [PlannedCompletionDate] == "" {
      remove_field => ["PlannedCompletionDate"]
1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.