Hard to stash a log file with different occurrence of order for a field using Logstash

Elastic Stack Logstash
1

I am trying to stash a log file to elasticsearch using Logstash. I am facing a problem while doing this.

If the log file has same kind of log lines like the below,

[12/Sep/2016:18:23:07] VendorID=5037 Code=C AcctID=5317605039838520 [12/Sep/2016:18:23:22] VendorID=9108 Code=A AcctID=2194850084423218 [12/Sep/2016:18:23:49] VendorID=1285 Code=F AcctID=8560077531775179 [12/Sep/2016:18:23:59] VendorID=1153 Code=D AcctID=4433276107716482

where the date, vendorId, code and acctID's order of occurrence of fields does not change or a new element is not added in to it, then the filter(given below) in the config files work well.

\[%{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME}\] VendorID=%{INT:VendorID} Code=%{WORD:Code} AcctID=%{INT:AcctID}

Suppose the order changes like the example given below or if a new element is added to one of the log lines, then the grokparsefailure occurs.

[12/Sep/2016:18:23:07] VendorID=5037 Code=C AcctID=5317605039838520
[12/Sep/2016:18:23:22] VendorID=9108 Code=A AcctID=2194850084423218 [12/Sep/2016:18:23:49] VendorID=1285 Code=F AcctID=8560077531775179 [12/Sep/2016:18:23:59] VendorID=1153 Code=D AcctID=4433276107716482 [12/Sep/2016:18:24:50] AcctID=3168124750473449 VendorID=1065 Code=L [12/Sep/2016:18:24:50] AcctID=3168124750473449 VendorID=1065 Code=L [12/Sep/2016:18:24:50] AcctID=3168124750473449 VendorID=1065 Code=L

Here in the example, the last three log lines are different from the first four log lines in order of occurrence of the fields. And because of this, the filter message with the grok pattern could not parse the below three lines as it is written for the first four lines.

How should I handle this scenario, when i come across this case? Please help me solve this problem. Also provide any link to any document for detailed explanation with examples.

Thank you very much in advance.

2

Use a kv filter instead of grok.

3

KV filter works when there is a case like key comes first and then value comes after...

My problem comes when there is a scenario where there is no key value pairs.. but it is for other fields like

date username some message ----------- first log line
date username domain program port some message ----------- second log line

we can see the difference of fields in a same log file. What should i do in this case?

4

Use a grok filter with multiple expressions. The first one tries to match the kind of event that contains domain, program, and port and assumes the rest is key/value pairs, and the second expression assumes everything after the timestamp is key/value pairs.

5

using multiple expressions is the only solution for this?

Because I am trying to handle lot of log lines with multiple structure of data and it is not just limited to one or two structures..

I can not go and read the entire log file to write the expression and even if i do, what if there are more than 10 to 20 different structure of log lines included in there?

Configuration for that particular case will be of lot of lines and do you think that is a feasible solution?

6

Are you interested in saving the fields that come before the key/value pairs? If yes then I don't see a way to write something generic, but if you don't care about those fields (or at least not all of them) it's of course possible to ignore tokens up until the first one with a key=value form.

7

Yes I need to save the fields. And as you said, if there is no generic way to do this, I will go for writing multiple expressions.

Thank you Magnus

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.