Drop field based on pattern

Elastic Stack Logstash
1

Hi, I have some field names that begin with the string "sub" that I want Logstash to drop.

Can I do something like:

if [sub*] {
    drop {}
}
2

Use a prune filter with the blacklist_names option.

1 Like
3

I tried doing the following to drop the "last-clear" field and any field that is blank:

prune {
    blacklist_names => ["last-clear", " "]
}

The "last-clear" has dropped, but I keep getting this WARN message:

[WARN ] 2021-06-22 20:58:20.397 [[qa-mx-oc]>worker0] elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"qa-mx-2021.06.22", :routing=>nil, :_type=>"_doc"}, #LogStash::Event:0x2d94acac], :response=>{"index"=>{"_index"=>"qa-mx-2021.06.22", "_type"=>"_doc", "_id"=>"m6WENXoBVd9j9yI8SwBf", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"field name cannot be an empty string"}}}}}

I tried enabling debug but can't correlate this error message to the debug messages. I tried grepping the _id value from the debug logs but didn't get anything.

How can I find out what field is causing the WARN message?

4

"" is a valid field name in JSON, but a lot of things object to empty names. It looks like elasticsearch is one of them.

prune { blacklist_names => [ "" ] }

does not work. I think it is deleting every field on the event and then the empty event is discarded.

mutate { remove_field => [ "" ] }

does work.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.