Problem multiline logs

Elastic Stack Beats
1

Hello everybody,

I am trying to send to logstack some lines from an event generate in my log file, I don't know how to do that.

Below is an exemple of the event which is generate in my log, what i am trying to send to logstack is only 3 of this line. (the 3 lines in bold).
My problem is that at the end of the event we have similar patterns which cause my problem (I cannot exclude or include because i will have 2 identical lines.)

Exemple of an event in my logs :

09/06/2016 16:31:00 [7] Request from 127.0.0.1
09/06/2016 16:31:00 [7] action=QUERY&outputencoding=UTF8&xmlmeta=true&querysummary=true&minscore=20&securityinfo=MjY0MHx&databasematch=uas%5Fintranetedf%2B&combine=simple%2BREFERENCE%5FFIELD2%2BREFERENCE%5FFIELD3%2BNODEREF&predict=false&sort=Relevance%2BDate&timeoutms=20000&languagetype=frenchUTF8&anylanguage=true&start=1&printfields=F1%2CF3%2CF2%2CI1%2CF7%2CURL%2CDREDATE%2CUSERID&maxresults=10&totalresults=true&summary=context&characters=260&highlight=summaryterms&starttag=%3Cstrong%3E&endtag=%3C%2Fstrong%3E&text=%28EDF%20OR%20%28ELECTRICITE%20DE%20FRANCE%29%29&actionid=c65dec82cbec4b721c5f03ea5936fd12c0bff8c3&fieldtext=BIASVAL%7Bfr%5FFR%2C1%7D%3ABIAS%5FFIELD1%2BAND%2BBIASVAL%7BGed%20Direction%20Groupe%2C%2D20%7D%3AF7 (127.0.0.1)
09/06/2016 16:31:00 [7] L 12071; A 11307; F 1331; S 1447; DL 1331; SL 0; DT 388
09/06/2016 16:31:00 [7] Returning 10 matches
09/06/2016 16:31:00 [7] Generating query summary
09/06/2016 16:31:00 [7] Query complete
09/06/2016 16:31:00 [7] Request completed in 119 ms.
09/06/2016 16:31:01 [6] Request from 127.0.0.1
09/06/2016 16:31:01 [6] action=GETQUERYTAGVALUES&outputencoding=UTF8&minscore=20&securityinfo=MjY0MH&databasematch=uas%5Fintranetedf%2B&combine=simple%2BREFERENCE%5FFIELD2%2BREFERENCE%5FFIELD3%2BNODEREF&sort=DocumentCount&timeoutms=20000&languagetype=frenchUTF8&anylanguage=true&start=1&documentcount=true&fieldname=F3%2CF2%2CF1%2CF6%2CF5%2CF4%2CF10%2CF7&ranges=FIXED%7B%2E%2C16595%2C16778%2C16869%2C16930%2C16954%2C16962%7D%3AF10&text=%28EDF%20OR%20%28ELECTRICITE%20DE%20FRANCE%29%29&actionid=d168b7bb2dc9c63711626344135110a74b29ccc2 (127.0.0.1)
09/06/2016 16:31:01 [6] L 12071; A 11307; F 1331; S 0; DL 1331; SL 0; DT 388
09/06/2016 16:31:01 [6] GetQueryTagValues complete
09/06/2016 16:31:01 [6] Request completed in 75 ms.

For exemple, if I include the lines I want and exclude the line iI don't want in the configuration file, I will have something like this.
What I want is to have only the 3 line in bold.

09/06/2016 16:31:00 [7] action=QUERY&outputencoding=UTF8&xmlmeta=true&querysummary=true&minscore=20&securityinfo=MjY0MHx&databasematch=uas%5Fintranetedf%2B&combine=simple%2BREFERENCE%5FFIELD2%2BREFERENCE%5FFIELD3%2BNODEREF&predict=false&sort=Relevance%2BDate&timeoutms=20000&languagetype=frenchUTF8&anylanguage=true&start=1&printfields=F1%2CF3%2CF2%2CI1%2CF7%2CURL%2CDREDATE%2CUSERID&maxresults=10&totalresults=true&summary=context&characters=260&highlight=summaryterms&starttag=%3Cstrong%3E&endtag=%3C%2Fstrong%3E&text=%28EDF%20OR%20%28ELECTRICITE%20DE%20FRANCE%29%29&actionid=c65dec82cbec4b721c5f03ea5936fd12c0bff8c3&fieldtext=BIASVAL%7Bfr%5FFR%2C1%7D%3ABIAS%5FFIELD1%2BAND%2BBIASVAL%7BGed%20Direction%20Groupe%2C%2D20%7D%3AF7 (127.0.0.1)
09/06/2016 16:31:00 [7] L 12071; A 11307; F 1331; S 1447; DL 1331; SL 0; DT 388
09/06/2016 16:31:00 [7] Request completed in 119 ms.
09/06/2016 16:31:01 [6] L 12071; A 11307; F 1331; S 0; DL 1331; SL 0; DT 388
09/06/2016 16:31:01 [6] Request completed in 75 ms.

Do you have any advice for me, how to do that ?

Thank you in advance for your help.

Yves

2

I don't understand. You using multiline support in filebeat? If so, filebeat is not parsing content, but only merging multiple lines into one event. More elaborate processing is the domain of logstash (e.g. grok filter).

3

Hi,
I use filebeat for retriving logs from a log file. i am trying to merge all the line in bold and exclude all the other. so that when i receive it in logstack i have only 1 message.
The problem i have is that in the event i have similar line which i don't want.

4

As I said, filebeat is not parsing content. multiline support is only about merging consecutive lines. It's not possible to include/exclude files from within in multiline processing (feature request reconsidering processing order - multiline vs. exclude-lines filtering order - is very welcome). I'd try using some grok filter "parsing" content in logstash for postprocessing the multiline event.

5

Hi,
Ok, thanks for your response, I will try to do that in logstack after excluding some logs lines in beatfile.
yves

6

Hi, i have successed what i wanted to do in filebeat.
I have understand that if my logs are not consecutive it will not work.
Here are the configuration I use to do the job if someone is interrested.

The logs I use is in my first post.

First i configure in the configuration file the multiline parameters like this

      multiline:
        pattern: '^[0-9]{2}/[0-9]{2}/[0-9]{4}[[:space:]][0-9]{2}:[0-9]{2}:[0-9]{2}[[:space:]]\[[0-9]\][[:space:]]Request[[:space:]]from'
        negate: true
        match: "after"

Doing that i expected to generate two events. All the information I needed was in my first event.

{"@timestamp":"2016-06-13T13:01:09.929Z","beat":{"hostname":"noeyyri5.hadam.hadroot.edf.fr","name":"noeyyri5.hadam.hadroot.edf.fr"},"input_type":"log","message":"13/06/2016 15:01:08 [7] Request from 127.0.0.1\n13/06/2016 15:01:08 [7] action=QUERY\u0026outputencoding=UTF8\u0026xmlmeta=true\u0026querysummary=true\u0026minscore=20\u0026securityinfo=Mzk2fN2rLjQuBFs2BilVL8aMJQ%2FaRhTA4cgR1KkHP7JzLVB0zNW63KVNi%2BPX7bclqDGbYIGSuhczWBZeFS1%2FmROnxZLAaGHrffjo%2BeEo1RpaYmz%2FVyh7Gy3pHOUenX0mfIoQeNfVn%2Bitd9TK0zY1A8KyNfdC6iXpZkuqPnNUyHWI9bg76j%2BtdOE69zY6Ka4xhe5T%2Fgr1DYo89Sh87oT3x%2Bgf7KNXAnjxNyIbSF50RRfrSvRw3xmwpKa8VzW8e6MAKJtDF2gnrGV1AeuBxrA%2FgwxmA0owPFNzkZrlh7JDHMBbGJkbXTNTF3j0H6ewwX1tC3700RQENI4ksAYz3bV8ip73ZNk1AWitNydR3anifScIIfeDYUKCM%2B956d%2FrmQLdgC9pMmxGDteS4CtNalanAeySiGnvnUV0Rb3YokPRp%2B5qxVtlsgixOpMWUE5kiUdlMrAAlV1UMhlO%2B8dYi7rjg8siik68bq7ovlD390ZgwWwM6w5LbcLmyZaSRGMkR0ylfgonF52TLRp7ZQuhbcSXAA%3D%3D\u0026databasematch=uas%5Fintranetedf%2B\u0026combine=simple%2BREFERENCE%5FFIELD2%2BREFERENCE%5FFIELD3%2BNODEREF\u0026predict=false\u0026sort=Relevance%2BDate\u0026timeoutms=20000\u0026languagetype=frenchUTF8\u0026anylanguage=true\u0026start=1\u0026printfields=F1%2CF3%2CF2%2CI1%2CF7%2CURL%2CDREDATE%2CUSERID\u0026maxresults=10\u0026totalresults=true\u0026summary=context\u0026characters=260\u0026highlight=summaryterms\u0026starttag=%3Cstrong%3E\u0026endtag=%3C%2Fstrong%3E\u0026text=%28EDF%20OR%20%28ELECTRICITE%20DE%20FRANCE%29%29\u0026actionid=b98fd11aa3a248bc212704d889f281c100d34d90\u0026fieldtext=BIASVAL%7Bfr%5FFR%2C1%7D%3ABIAS%5FFIELD1%2BAND%2BBIASVAL%7BGed%20Direction%20Groupe%2C%2D20%7D%3AF7 (127.0.0.1)\n13/06/2016 15:01:08 [7] L 12070; A 11306; F 1331; S 1127; DL 1331; SL 0; DT 388\n13/06/2016 15:01:08 [7] Returning 10 matches\n13/06/2016 15:01:08 [7] Generating query summary\n13/06/2016 15:01:08 [7] Query complete\n13/06/2016 15:01:08 [7] Request completed in 96 ms.","offset":144022,"source":"/logiciels/idol/prod/idol10800/IDOL/content_10810/logs/content_query.log","type":"log"}

{"@timestamp":"2016-06-13T13:01:09.930Z","beat":{"hostname":"noeyyri5.hadam.hadroot.edf.fr","name":"noeyyri5.hadam.hadroot.edf.fr"},"input_type":"log","message":"13/06/2016 15:01:08 [6] Request from 127.0.0.1\n13/06/2016 15:01:08 [6] action=GETQUERYTAGVALUES\u0026outputencoding=UTF8\u0026minscore=20\u0026securityinfo=Mzk2fN2rLjQuBFs2BilVL8aMJQ%2FaRhTA4cgR1KkHP7JzLVB0zNW63KVNi%2BPX7bclqDGbYIGSuhczWBZeFS1%2FmROnxZLAaGHrffjo%2BeEo1RpaYmz%2FVyh7Gy3pHOUenX0mfIoQeNfVn%2Bitd9TK0zY1A8KyNfdC6iXpZkuqPnNUyHWI9bg76j%2BtdOE69zY6Ka4xhe5T%2Fgr1DYo89Sh87oT3x%2Bgf7KNXAnjxNyIbSF50RRfrSvRw3xmwpKa8VzW8e6MAKJtDF2gnrGV1AeuBxrA%2FgwxmA0owPFNzkZrlh7JDHMBbGJkbXTNTF3j0H6ewwX1tC3700RQENI4ksAYz3bV8ip73ZNk1AWitNydR3anifScIIfeDYUKCM%2B956d%2FrmQLdgC9pMmxGDteS4CtNalanAeySiGnvnUV0Rb3YokPRp%2B5qxVtlsgixOpMWUE5kiUdlMrAAlV1UMhlO%2B8dYi7rjg8siik68bq7ovlD390ZgwWwM6w5LbcLmyZaSRGMkR0ylfgonF52TLRp7ZQuhbcSXAA%3D%3D\u0026databasematch=uas%5Fintranetedf%2B\u0026combine=simple%2BREFERENCE%5FFIELD2%2BREFERENCE%5FFIELD3%2BNODEREF\u0026sort=DocumentCount\u0026timeoutms=20000\u0026languagetype=frenchUTF8\u0026anylanguage=true\u0026start=1\u0026documentcount=true\u0026fieldname=F3%2CF2%2CF1%2CF6%2CF5%2CF4%2CF10%2CF7\u0026ranges=FIXED%7B%2E%2C16599%2C16782%2C16873%2C16934%2C16958%2C16966%7D%3AF10\u0026text=%28EDF%20OR%20%28ELECTRICITE%20DE%20FRANCE%29%29\u0026actionid=24ab0c3d641b5e1d77b4bcbc13d232e140983431 (127.0.0.1)\n13/06/2016 15:01:08 [6] L 12070; A 11306; F 1331; S 0; DL 1331; SL 0; DT 388\n13/06/2016 15:01:08 [6] GetQueryTagValues complete\n13/06/2016 15:01:08 [6] Request completed in 45 ms.","offset":145358,"source":"/logiciels/idol/prod/idol10800/IDOL/content_10810/logs/content_query.log","type":"log"}

After that i decide to exclude the event i don't want in the parameter file.

exclude_lines: ["action=GETQUERYTAGVALUES"]

Doing that, it's work, only one event with all the infos i need are been send to logstack.

With this test i understand that the multiline process is doing the job first and after that, the exclude_lines is process in the second time. Is it OK ?

yves

7

This topic was automatically closed after 21 days. New replies are no longer allowed.