Avoid buffer flush when no multiline pattern matches

Elastic Stack Beats
1

I encountered a certain problem when trying to create the multiline features in filebeat. I dont have online access to the examples right now, so I'll just present an abstraction of the situation.

I have a certain log file which the filebeat tails. Assume it has only two patterns of lines, A and B (where A is simply not B).
The multiline configuration is the following:

pattern: ^B
negate: true
match: after

Everything else is default.

I matched the regex which both types of lines in the Go Playground, and got the desired result, hence assume the regex is valid.

The problem occurs when the file has no lines of type B at all. Instead of getting multiple events containing a single A each, I get batches of A. For example:

A\n
A\n
A\n ==> {AA} {AA}
A\n

I suspect the batch's size is just the number of lines processed by Filebeat every Timout (which default to 5s).
As far as I understand, disabling the timeout in the " Only 'A' " would just result in Filebeat aggregating the lines into the buffer, and probably throw them in a single batch again, but instead after 5 seconds, after some size limitation.

As a note, in a log file where B occurs every now and then, I get the expected result of B followed by some 'A's.

How can I overcome this? (I don't eliminate the possibility that perhaps I misunderstood the Timeout property.)

Thanks

2

This is somewhat super abstract. Multiline supports a flush_pattern, this can be used if there is some difference in structure between first line of A and subsequent lines of A. One can also use | in regular expression to create an or-pattern.

3

I actually have no flush pattern, since I expect the log file to be of the form:
B
A
. (lots of As)
A
B

And I want Filebeat to aggregate every B with its following A, but in case A is not preceded by B, I want it to be in an event of its own (hence, in the example, a log file which has only As, should be just read line by line, where each A gets an event of its own).
If the problem is not clear please tell me, and I'll try to completely formulate it.

4

Well, the problem is clear. But the way you describe it in this abstract form I don't have a solution. But seeing actual logs I might be able to come up the an idea.

The question about the A events is, do the lines in A differ in some way or the other? By contents/structure/'shape'/correlation-id.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.