Unable to exclude_lines in filebeat

balamelangi · January 22, 2019, 10:36am

@warkolm
Hi.
My log line is
2019-01-22 10:25:01,401 ERROR stderr org.jboss.stdio.AbstractLoggingWriter.write(AbstractLoggingWriter.java:71) - URI: urn:pronto.ver600
2019-01-22 10:25:01,401 ERROR stderr org.jboss.stdio.AbstractLoggingWriter.write(AbstractLoggingWriter.java:71) - DD.ProviderClass: null.
2019-01-22 10:34:11,426 ERROR stderr org.jboss.stdio.AbstractLoggingWriter.write(AbstractLoggingWriter.java:71) - In TemplateProvider.locate()

I tried to below conf in filebeat.yml

exclude_lines: ["org.jboss.stdio.AbstractLoggingWriter.write"]
exclude_lines: ["AbstractLoggingWriter.write"]
exclude_lines: [".AbstractLoggingWriter.write."].

But I'm getting same errors in notification. Please help me how to drop above type log only. Please some one help me.
In filebeat I nead include error and warn but exclude above AbstractLoggingWriter.write.

Thanks,
Bala Melangi

Tek_Chand · January 22, 2019, 10:52am

@balamelangi,

Please try the below config in yout filebeat.yml

exclude_lines: ['^.*AbstractLoggingWriter.write..*$']

Thanks.

Jorja073 · January 22, 2019, 11:44am

I'm having some issues getting filebeat to exclude lines. Here are the ways I've tried to use the regexp format with the exclude_lines: option in the apache2.yml file: ELK: start logstash fails prepaidgiftbalance illegalstateexception .

balamelangi · January 22, 2019, 12:26pm

Thanks for your replay @Tek_Chand.
But still I'm getting notifications. My new config in filebeat is below.
exclude_lines: ['^.AbstractLoggingWriter.write..$']

I'm getting
2019-01-22 12:18:29,146 ERROR stderr org.jboss.stdio.AbstractLoggingWriter.write(AbstractLoggingWriter.java:71) -
2019-01-22 12:18:30,312 ERROR stderr org.jboss.stdio.AbstractLoggingWriter.write(AbstractLoggingWriter.java:71) - URI: urn:pronto.ver600
2019-01-22 12:15:25,786 ERROR stderr org.jboss.stdio.AbstractLoggingWriter.write(AbstractLoggingWriter.java:71) - In TemplateProvider.invoke()

If you have any other suggestions please help me.

Thanks.

balamelangi · January 22, 2019, 12:31pm

my filebeat.yml file is :
filebeat.prospectors:

input_type: log
paths:
- /u04/jboss/standalone/log/server.log
- /var/log/soapradius.log
- /var/log/radius.log
  exclude_lines: ['^.AbstractLoggingWriter.write..$']

output.logstash:
hosts: ["x.x.x.x:5044"]
index: test

Please help me.

Thanks.

Tek_Chand · January 22, 2019, 12:31pm

@balamelangi,

Please try below config:

exclude_lines: ['^.*AbstractLoggingWriter.*$']

balamelangi · January 22, 2019, 12:51pm

Hi @Tek_Chand. But it's not working.
My new config in filebeat.yml is :
exclude_lines: ['^.AbstractLoggingWriter.$']

I'm getting below mail notification :

2019-01-22 12:48:54,060 ERROR stderr org.jboss.stdio.AbstractLoggingWriter.write(AbstractLoggingWriter.java:71) - URI: urn:pronto.ver600
2019-01-22 12:48:54,060 ERROR stderr org.jboss.stdio.AbstractLoggingWriter.write(AbstractLoggingWriter.java:71) - DD.ProviderClass: null 2019-01-22 12:48:54,514 ERROR stderr org.jboss.stdio.AbstractLoggingWriter.write(AbstractLoggingWriter.java:71) -
2019-01-22 12:48:54,514 ERROR stderr org.jboss.stdio.AbstractLoggingWriter.write(AbstractLoggingWriter.java:71) - DD.ProviderClass: null
2019-01-22 12:48:54,060 ERROR stderr org.jboss.stdio.AbstractLoggingWriter.write(AbstractLoggingWriter.java:71) - In TemplateProvider.invoke()

Thanks.

balamelangi · January 22, 2019, 12:54pm

@here Is there any other way to drop above log lines using filebeat?

Thanks.

Tek_Chand · January 23, 2019, 9:37am

@balamelangi,

Can you please let me know from where you are getting this mail notification? Means which server is generating this mail?

Thanks.

balamelangi · January 23, 2019, 9:48am

Hi @Tek_Chand
I'm using beat as filebeat.
My architecture is below :

Filebeat---->Logstash---->ElasticSearch<-------Kibana.

In kibana I'm set notification when message match : "error" , Using X-pack notification.
In filebeat.yml I'm set logfile /u04/jboss/standalone/log/server.log .
I'm getting below log to mail when error match.

2019-01-23 08:48:02,827 ERROR stderr org.jboss.stdio.AbstractLoggingWriter.write(AbstractLoggingWriter.java:71) - DD.ProviderClass: null
2019-01-23 08:48:02,827 ERROR stderr org.jboss.stdio.AbstractLoggingWriter.write(AbstractLoggingWriter.java:71) - DD.ServiceClass: org.apache.soap.providers.StatelessEJBProvider
2019-01-23 08:48:02,827 ERROR stderr org.jboss.stdio.AbstractLoggingWriter.write(AbstractLoggingWriter.java:71) - Call.MethodName: nems
2019-01-23 08:48:02,827 ERROR stderr org.jboss.stdio.AbstractLoggingWriter.write(AbstractLoggingWriter.java:71) - In TemplateProvider.locate()
2019-01-23 08:48:02,827 ERROR stderr org.jboss.stdio.AbstractLoggingWriter.write(AbstractLoggingWriter.java:71) - URI: urn:pronto.ver600
2019-01-23 08:48:02,828 ERROR stderr org.jboss.stdio.AbstractLoggingWriter.write(AbstractLoggingWriter.java:71) - In TemplateProvider.invoke() .

My filebeat.yml file is
exclude_lines: ['^.AbstractLoggingWriter.']

Tek_Chand · January 23, 2019, 10:06am

@balamelangi,
are you sure that your filebeat.yml have below pattern for exclude line?

exclude_lines: ['^.AbstractLoggingWriter.]

But the pattern i have provided you is little bit different then above, which is look like below:

exclude_lines: ['^.*AbstractLoggingWriter.*$']

balamelangi · January 23, 2019, 10:08am

@Tek_Chand
I tried both. But not working.

Thanks.

Tek_Chand · January 23, 2019, 10:10am

@balamelangi,

you have set the above configuration on all filebeat server or only single server? May be you are receiving these logs from any other server.

I have used the same pattern at my end and its working fine.

balamelangi · January 23, 2019, 10:20am

@Tek_Chand
my watcher is

{
"trigger": {
"schedule": {
"interval": "2m"
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"platform.wavespot.net-*"
],
"types": ,
"body": {
"query": {
"bool": {
"must": {
"match": {
"message": "error"
}
},
"filter": {
"range": {
"@timestamp": {
"from": "now-2m",
"to": "now"
}
}
}
}
}
}
}
}
},
"condition": {
"compare": {
"ctx.payload.hits.total": {
"gt": 0
}
}
},
"actions": {
"email_admin": {
"email": {
"profile": "standard",
"to": [
"Bala Melangi bala.melangi@xxxxxx.com"
],
"subject": "Platform: JBOSS Errors from Watcher",
"body": {
"text": " Total {{ctx.payload.hits.total}} errors. Below are the errors from {{ctx.payload.hits.hits.0._source.beat.hostname}} \n {{ctx.payload.hits.hits.0._source.message}} \n {{ctx.payload.hits.hits.1._source.message}} \n {{ctx.payload.hits.hits.2._source.message}} \n {{ctx.payload.hits.hits.3._source.message}} \n {{ctx.payload.hits.hits.4._source.message}} \n {{ctx.payload.hits.hits.5._source.message}} \n {{ctx.payload.hits.hits.6._source.message}} \n {{ctx.payload.hits.hits.7._source.message}} \n {{ctx.payload.hits.hits.8._source.message}} \n"
}
}
}
}
}

balamelangi · January 23, 2019, 10:24am

@Tek_Chand I added the exclude_line to all my productions. Platform is one of the prod-server, And the server index is platform.wavespot.net-* please check.
I got error notification is,

Total 11 errors. Below are the errors from platform.wavespot.net
2019-01-23 10:23:37,627 ERROR stderr org.jboss.stdio.AbstractLoggingWriter.write(AbstractLoggingWriter.java:71) - In TemplateProvider.locate()
2019-01-23 10:23:37,628 ERROR stderr org.jboss.stdio.AbstractLoggingWriter.write(AbstractLoggingWriter.java:71) - =============================================
2019-01-23 10:23:37,627 ERROR stderr org.jboss.stdio.AbstractLoggingWriter.write(AbstractLoggingWriter.java:71) - Call.MethodName: nems
2019-01-23 10:23:37,627 ERROR stderr org.jboss.stdio.AbstractLoggingWriter.write(AbstractLoggingWriter.java:71) - =============================================
2019-01-23 10:23:37,627 ERROR stderr org.jboss.stdio.AbstractLoggingWriter.write(AbstractLoggingWriter.java:71) - URI: urn:pronto.ver600
2019-01-23 10:23:37,627 ERROR stderr org.jboss.stdio.AbstractLoggingWriter.write(AbstractLoggingWriter.java:71) - DD.ServiceClass: org.apache.soap.providers.StatelessEJBProvider
2019-01-23 10:23:37,628 ERROR stderr org.jboss.stdio.AbstractLoggingWriter.write(AbstractLoggingWriter.java:71) - In TemplateProvider.invoke()
2019-01-23 10:23:37,627 ERROR stderr org.jboss.stdio.AbstractLoggingWriter.write(AbstractLoggingWriter.java:71) - DD.ProviderClass: null
2019-01-23 10:22:48,233 ERROR stderr org.jboss.stdio.AbstractLoggingWriter.write(AbstractLoggingWriter.java:71) - Wed Jan 23 10:22:48 UTC 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.

Thanks.

Tek_Chand · January 23, 2019, 10:31am

@balamelangi,

Can you please share you filebeat.yml file again. Please share it in formatted way so we can read it easily.

You can use </> to format your file.

Thanks.

balamelangi · January 23, 2019, 10:42am

@Tek_Chand
Please see my filebeat.yml file

filebeat.prospectors:

- input_type: log
  paths:
    - /u04/jboss/standalone/log/server.log
    - /var/log/soapradius.log
    - /var/log/radius.log
    exclude_lines: ['^.*AbstractLoggingWriter.*$']


output.logstash:
  hosts: ["x.x.x.x:5044"]
  index: platform.wavespot.net

Is there any other string I need to replace in place of AbstractLoggingWriter ?

Thanks.

steffens · January 23, 2019, 12:14pm

Please format logs and config files using the </> button. Config files are sensitive to indentation (YAML file format). In your last example config the indentation of exclude_lines is off.

Tip: using ^.* or .*$ expresses a sub-string match. This is exactly what exclude_lines does. This config should work:

filebeat.prospectors:

- type: log
  paths:
    - /u04/jboss/standalone/log/server.log
    - /var/log/soapradius.log
    - /var/log/radius.log
  exclude_lines: ['AbstractLoggingWriter']


output.logstash:
  hosts: ["x.x.x.x:5044"]
  index: "platform.wavespot.net"

balamelangi · January 23, 2019, 12:26pm

@steffens,

Same not working.

2019-01-23 10:23:37,627 ERROR stderr org.jboss.stdio.AbstractLoggingWriter.write(AbstractLoggingWriter.java:71) - In TemplateProvider.locate()

is my log.
Even I tried below
exclude_lines: ['^.AbstractLoggingWriter.$']
exclude_lines: ['AbstractLoggingWriter']
exclude_lines: ['^ERROR']
exclude_lines: ['ERROR']
exclude_lines: ['org.jboss.stdio.AbstractLoggingWriter.write(AbstractLoggingWriter.java:71)']

exclude_lines: ['^.org.jboss.stdio.AbstractLoggingWriter.write(AbstractLoggingWriter.java:71).$']

Thanks.

Tek_Chand · January 23, 2019, 12:31pm

@balamelangi,

Did you fix the identation issue in your filebeat.yml issue which is suggested in above post by Steffens?

Topic		Replies	Views
Exclude_lines are not working in my filebeat Beats filebeat	3	649	February 20, 2019
Using the exclude_lines option Beats filebeat	6	2679	July 5, 2017
Filebeat Bug: exclude_lines Beats filebeat	3	399	February 26, 2021
How to "exclude_lines but make these exceptions"? Beats filebeat	9	2810	July 5, 2017
Filebeat : how to exclude lines Beats filebeat	7	7634	October 5, 2021

Unable to exclude_lines in filebeat

Related topics