Single line with Filebeat to logstash

Elastic Stack Beats
1

Hi, is there any option with filebeat to extract lines only matching keywords from my input file ? normally my log file size is 25MB /day and similar to that i have almost more than 100+ files where i want to index only required lines which comes with JAVA stacktrace to extract using multiline method.
My logfile looks like something

[5/10/17 9:31:22:624 AEST] 0000005f NGUtil$Server I   ASND0003I: Detected server teamfo1pe1_1 stopped on node aho0ap39
[5/10/17 9:33:59:737 AEST] 00000060 NGUtil$Server I   ASND0002I: Detected server teamfo1pe1_2 started on node aho0ap39
[5/10/17 9:33:59:801 AEST] 00000001 WsServerImpl  A   WSVR0001I: Server teamfo1pe1_2 open for e-business
[5/10/17 9:34:08:790 AEST] 000000a9 SystemOut     O INFO  - WGC1054YMH12.09076.01.MR XX.00134.ReAuthenticate.01.MR XX.00135 - 2017/05/09-23:34:08,789 UTC - ao0ap39.foa.ford.com - Global pool configuration: 
[5/10/17 9:34:08:790 AEST] 000000a9 SystemOut     O INFO  - WGC1054YMH12.09076.01.MR XX.00134.ReAuthenticate.01.MR XX.00135 - 2017/05/09-23:34:08,790 UTC - ao0ap39.foa.ford.com - Global pool config published.
[5/10/17 9:34:08:849 AEST] 000000a9 SystemOut     O WARN  - WGC1054YMH12.09076.01.MR XX.00134.ReAuthenticate.01.MR XX.00135 - 2017/05/09-23:34:08,848 UTC - ao0ap39.foa.ford.com - Global pool configuration INVOCATION_MAX_CONNECTIONS not found.  Using 1000
[5/10/17 9:34:11:621 AEST] 0000005f NGUtil$Server I   ASND0002I: Detected server dmgr started on node ao0ap39Manager
[5/10/17 9:34:12:071 AEST] 00000060 NGUtil$Server I   ASND0002I: Detected server nodeagent started on node ao0ap39
[5/10/17 9:34:12:383 AEST] 0000005f NGUtil$Server I   ASND0002I: Detected server teamfoa1pe1_1 started on node aho0ap39

Where my input file will look like something
TeamInput.log
teamfo1pe1_2 ;e-business;Global pool.

what i wish to see is only lines which contains these keywords will be indexed to Elasticsearch via logstash

Any help is greatly appreciated.

2

Please format logs, configs and terminal input/output using the </>-Button or markdown code fences. This forum uses Markdown to format posts. Without proper formatting, it can be very hard to read your posts.

You should configure multiline in filebeat, not rely on multiline support from Logstash.
For filtering you can use exclude_lines and/or include_lines setting. These filters are applied after multiline reconstructing the full event.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.