Logstash filter out log lines

Elastic Stack Logstash
1

Hi,
I have json logs that I want to filter based on the message field.

{"@timestamp":"2022-04-05T01:20:50.917+00:00","severity":"INFO","service":"service","pid":"18245","thread":"http-nio-8102-exec-1","class":"class","message":"[ELK] some log data"}

Above is the json object that is being parsed by logstash, I want to write only those logs onto my Elasticsearch index which have [ELK] in the message field and ignore all other logs.

I'm not able to do this through logstash filters, if anyone can guide me it will of great help.

2

You just need a conditional.

if "[ELK]" not in [message] {
    drop {}
}

This will drop every event that does not have the string "[ELK]" in the message field.

3

Do I need to add this in the filter section like this

filter {
json {
if "[ELK]" not in [message] {
    drop {}
}
}
4

It is in the filter section, but it is not inside the json filter, you need to put it after you parse the message.

filter {
    if "[ELK]" not in [message] {
        drop {}
    }
}
5

Thanks for the help. It's now working as I wanted.

6

Hi All,

Can somebody please help to check why the below error while using cipher_kms filter

ipeline error {:pipeline_id=>"main", :exception=>#<NoMethodError: undefined method blank' for nil:NilClass>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-cipher_kms-0.1.3/lib/logstash/filters/cipher_kms.rb:234:in init_cipher'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-cipher_kms-0.1.3/lib/logstash/filters/cipher_kms.rb:131:in register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:232:in block in register_plugins'", "org/jruby/RubyArray.java:1821:in each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:231:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:594:in maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:244:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:189:in run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:141:in `block in start'"], "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.