Filebeat drop_event

Elastic Stack Beats
1

Hi Folks,

I have an issue when using processors in filebeat for dropping event when certain condition matches.
I am trying to drop event from log which contains "log_time" in message field. But the drop filter is not working and all file started processing without dropping.

Below is my filebeat.yml configuration file->

############################# Filebeat ######################################
filebeat:
prospectors:
-
paths:
- /opt/logs/app_logs.txt
input_type: log
fields_under_root: true
fields:
type: app_log
server_group: localhost
processors:
- drop_event:
when:
regexp:
message: "(.|-)log_host(.|-)"

registry_file: /etc/filebeat/.filebeat
############################# Output ##########################################

output:
logstash:
hosts: ["127.0.0.1:5044"]
loadbalance: true

############################# Logging #########################################

logging:
to_syslog: false
to_files: true
files:
path: /opt/logs/filebeat
name: filebeat.log
rotateeverybytes: 10485760 # = 10MB
keepfiles: 7
selectors: ["*" ]
level: error

Filebeat Published log->

2017-06-07T16:21:31+05:30 DBG Publish: {
"@timestamp": "2017-06-07T10:51:31.750Z",
"beat": {
"hostname": "shakti",
"name": "shakti",
"version": "5.4.1"
},
"input_type": "log",
"message": "{"log_time":"9\/Jan\/2017:14:10:21 +05:30","level":"ERROR","log_message":"something","description":"{\"field1\":\"Exception Code\",\"field2\":000,\"field3\":\"Exception Message\",\"field4\":\"10.12.12.33: E11000 $id dup key: { : ObjectId('asdsadsad111') }\",\"error_name\":\"mongo_log_error\",\"data\":{\"order_id\":\"102000176\",\"order_info\":null,\"momoe_order_id\":\"292\",\"ref_id\":\"147738220035719142144\",\"order_tag_momoe_response\":null,\"_id\":{\"$id\":\"58734c728eeb965e5f4ebf8e\"}},\"additional_data\":{\"file\":\"\\\/home\\\/ssss\\\/vendor\\\/fusion\\\/model\\\/aaa\\\/asas.php\",\"line\":901,\"method\":\"clues\\\\model\\\\pgw\\\\Momoe::logInMongo\"},\"uUid\":\"1483951218_3Y8dDBytHiqIw2v\"}","url":"\/tools\/momoe_order_updates.php?batch_size=2\u0026batch_limit=1--time_before_current%201800%20--time_range%203600","module":"pgw","session":"","log_host":"acd111"}",
"offset": 1487718,
"server_group": "localhost",
"source": "/opt/logs/app_logs.txt",
"type": "app_log"
}

Pls suggest where I am going wrong.

2

With the config being quoted I cannot tell if the indentation is correct. It should look like this:

processors:
- drop_event:
    when:
      regexp:
        message: 'log_host'
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.