Want to exclude lines in file having "etluser" keyword

Elastic Stack Beats
1

hello team,

we want to exclude lines in file having "etluser" keyword. we added this in yml file . Is this correct way to achieve the goal or any other way ?

exclude_lines: ['etluser']

we have multiple filepath added in yml file but all files have same keyword

2

Any comments please

3

I tried above option but didnt worked so what we can do now to resolve this.

4

That should be the correct way to do it. Can u post an example line that you're trying to exclude? That config option takes regex strings so u may just need to tweak it trip make it match properly.

5

Sample line:

2021-09-23 12:52:44,592 | INFO | HiveServer2-Handler-Pool: Thread-68096693 | UserName=etluser UserIP=10.117.225.122 Time=2021/09/23 12:52:44 Opertaion=ExecuteStatement stmt={SET hive.exec.dynamic.partition.mode=nonstrict}Result= Detail= | org.apache.hive.service.cli.thrift.ThriftCLIService.logAuditEvent(ThriftCLIService.java:507).

We need to exclude all lines where "etluser" is coming

6

Try ^.*etluser.*

7

I tried above already but not working ..Still logs are showing in logstash with this keyword.

Can we try like this

exclude_lines: ['[e][t][l][u][s][e][r]']

8

We tried both one by one but but not working .

exclude_lines: ['[e][t][l][u][s][e][r]']
exclude_lines: [^.etluser.]

Sample log:

2021-09-27 16:07:47,976 | INFO | HiveServer2-Handler-Pool: Thread-71033208 | OperationId=6849f9d6-e005-4a35-a0f9-d619dfd55a27 UserName=etluser UserIP=10.117.225.126 Time=2021/09/27 16:07:47 Opertaion=ExecuteStatement: stmt={insert into iptv.tx_iptv_cms_channel_outage partition(yr) select * , date_format(current_timestamp , 'yyyy-MM-dd HH:mm:ss') as load_timestamp , date_format(current_timestamp , 'yyyy') as yr from iptv.iptv_cms_channel_outage_stg} Result=SUCCESS Detail= | org.apache.hive.service.cli.thrift.ThriftCLIService.logAuditEvent(ThriftCLIService.java:507)

9

Hello Team,

We try above suggestions but not working.

We are trying now below approach also to discard event from log file

processors:

  • drop_event:
    when:
    contains:
    status: "etluser"

here I want to know what is the field here we can take in my example instead of status ?

10

I tried many ways but nothing is working for me

processors:

  • drop_event:
    when:
    contains:
    message: "etluser"
11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.