I have some data from the IBM nmon Linux/AIX monitoring tool. It's an awkward format with several different delimited and headerless fields. I've managed to create all of its grok patterns and process it (mostly) using the multiline filter.
For example, here's an example of some raw data:
ZZZZ,T0002,00:05:02,26-OCT-2016
TOP,0005626,T0002,2.97,2.80,0.17,3528824,990492,4,3319888,7640,1,0,java
TOP,0015478,T0002,0.56,0.39,0.17,3312584,677780,4,3124764,7188,0,0,java
TOP,0012483,T0002,0.54,0.35,0.19,3352620,718024,4,3165004,7144,0,0,java
ZZZZ,T0003,00:10:02,26-OCT-2016
There may be n TOP lines like this before the next timestamp appears, we can't know how many.
The main issue here is that only the first line contains a timestamp, so the event is spread over multiple lines
With some multiline such as:
codec => multiline {
# Joins all lines between lines with ^ZZZZ into single line.
# Line feed will still be present in the new 'single' line.
patterns_dir => "./patterns"
pattern => "^%{NMON_TOP_LINE}"
negate => true
what => previous
}
}
}
filter {
mutate {
#remove \n in message and replaces with LF0A
gsub => ["message", "\n", "###"]
}
...it becomes a more manageable (supposedly!) single line:
TOP,0005626,T0002,2.97,2.80,0.17,3528824,990492,4,3319888,7640,1,0,java###TOP,0015478,T0002,0.56,0.39,0.17,3312584,677780,4,3124764,7188,0,0,java###TOP,0012483,T0002,0.54,0.35,0.19,3352620,718024,4,3165004,7144,0,0,java
I'm trying to use these patterns to grok the data:
NMON_TIMETAG T[0-9]*
EOLDATA .*?###
NMON_TOP_LINE TOP,%{NUMBER:PID},%{NMON_TIMETAG:TopTimeTag},%{NUMBER:CPUPercent},%{NUMBER:UsrPercent},%{NUMBER:SysPercent},%{NUMBER:Size},%{NUMBER:ResSet},%{NUMBER:ResText},%{NUMBER:ResData},%{NUMBER:ShdLib},%{NUMBER:MinorFault},%{NUMBER:MajorFault},%{EOLDATA:Command}
and this particular grok line for the final string:
%{NMON_TOP_LINE}
This works fine for one line, but I want to use the same pattern repeatedly, something like:
(%{NMON_TOP_LINE)*
But that only matches the first line. Same with:
(%{NMON_TOP_LINE){3}
Except this wouldn't work in practice anyway because I won't know how may of these lines will be in the input data.
This works:
%{NMON_TOP_LINE}%{NMON_TOP_LINE}%{NMON_TOP_LINE}
and it puts the repeated data into arrays but it really violates the DRY principle and looks ugly, plus like I said I don;t know how many of these I'll need.
How can I re-use the same pattern without overwriting values?