I have some data from the IBM nmon Linux/AIX monitoring tool. It's a strange hybrid format with several different delimited and headerless fields.
For example:
ZZZZ,T0002,00:05:06,04-AUG-2016
CPU001,T0002,0.6,0.2,0.0,99.2
CPU002,T0002,0.6,0.2,0.0,99.2
CPU003,T0002,0.5,0.2,0.0,99.3
CPU004,T0002,0.3,0.3,0.0,99.4
CPU005,T0002,0.6,0.3,0.0,99.1
CPU006,T0002,0.3,0.1,0.0,99.5
CPU007,T0002,0.7,0.5,0.0,98.8
CPU_ALL,T0002,0.5,0.2,0.0,99.2,,7
MEM,T0002,5851.9,-0.0,-0.0,2047.0,1580.1,-0.0,-0.0,1984.7,-0.0,1480.6,2415.4,-1.0,427.7,10.7,765.1
VM,T0002,7,0,0,3053,18450,-1,4,5892,0,0,53321,76,0,128829,0,0,0,0,0,0,0,0,0,38654,0,0,0,0,0,0,0,0,0,0,0,0,0
PROC,T0002,1,0,777.9,-1.0,-1.0,-1.0,0.6,-1.0,-1.0,-1.0
NET,T0002,0.1,1.6,0.0,0.1,2.2,0.0,
NETPACKET,T0002,1.4,6.6,0.1,1.4,5.1,0.0,
JFSFILE,T0002,69.3,74.1,90.9
DISKBUSY,T0002,0.0,0.0,0.0,0.1,0.1,0.1
DISKREAD,T0002,0.0,0.0,0.0,0.0,0.0,0.0
DISKWRITE,T0002,3.6,3.6,0.0,16.1,16.1,16.1
DISKXFER,T0002,0.5,0.5,0.0,1.2,1.2,4.0
DISKBSIZE,T0002,7.3,7.3,0.0,13.9,13.9,4.0
TOP,0004692,T0002,4.29,3.28,1.01,6686460,1276380,4,6508344,36024,10,0,java
TOP,0006659,T0002,0.43,0.20,0.23,74776,19084,18592,21276,3252,215,0,netprobe.linux_
TOP,0004690,T0002,0.19,0.05,0.14,118908,1736,368,11064,1052,0,0,wrapper-3.5.25
TOP,0029842,T0002,0.16,0.10,0.06,277244,10400,676,113532,1864,0,0,vmtoolsd
ZZZZ,T0003,00:10:06,04-AUG-2016
CPU001,T0003,0.7,0.3,0.0,99.1
CPU002,T0003,0.3,0.2,0.0,99.5
CPU003,T0003,0.3,0.2,0.0,99.5
CPU004,T0003,0.2,0.2,0.0,99.6
CPU005,T0003,0.7,0.6,0.0,98.7
CPU006,T0003,0.3,0.1,0.0,99.6
CPU007,T0003,0.5,0.2,0.0,99.2
CPU_ALL,T0003,0.4,0.3,0.0,99.3,,7
MEM,T0003,5851.9,-0.0,-0.0,2047.0,1601.8,-0.0,-0.0,1984.7,-0.0,1480.6,2393.8,-1.0,427.7,10.7,765.1
VM,T0003,12,0,0,3005,18191,-1,0,2252,0,0,41814,50,0,124354,0,0,0,0,0,0,0,0,0,35226,0,0,0,0,0,0,0,0,0,0,0,0,0
PROC,T0003,1,0,776.1,-1.0,-1.0,-1.0,0.6,-1.0,-1.0,-1.0
NET,T0003,0.1,1.6,0.0,0.1,2.1,0.0,
NETPACKET,T0003,1.4,6.5,0.3,1.4,4.7,0.0,
JFSFILE,T0003,69.3,74.1,91.0
DISKBUSY,T0003,0.0,0.0,0.0,0.1,0.1,0.1
DISKREAD,T0003,0.0,0.0,0.0,0.0,0.0,0.0
DISKWRITE,T0003,2.3,2.3,0.0,5.2,5.2,5.2
DISKXFER,T0003,0.3,0.3,0.0,0.9,0.9,1.3
DISKBSIZE,T0003,6.7,6.7,0.0,6.0,6.0,4.0
TOP,0004692,T0003,3.80,2.68,1.12,6686460,1255040,4,6508344,36024,1,0,java
TOP,0006659,T0003,0.46,0.22,0.24,74776,19084,18592,21276,3252,217,0,netprobe.linux_
TOP,0004690,T0003,0.20,0.05,0.15,118908,1736,368,11064,1052,0,0,wrapper-3.5.25
TOP,0029842,T0003,0.16,0.11,0.05,277244,10400,676,113532,1864,0,0,vmtoolsd
You can see that the pattern of data begins with ZZZZ and a timestamp, followed by lines of data for various system metrics. The second field in the ZZZZ field is a king of tag, that indicates the timestamp in each of the following lines. So there is a structure, I'm just finding it difficult to come up with the right approach in the conf, even though I have grok patterns for the lines I'm interested in.
Using a multiline codec might be the way to go, but the regex pattern will be huge and makes me think that there might be a simpler way.
I've tried a few variations on this:
input {
stdin { }
}
filter {
grok {
patterns_dir => "./patterns"
break_on_match => false
match => { "message" => "%{NMON_TIMESTAMP_LINE}" }
}
date {
match => [ "timestamp", "HH:mm:ss,dd-MMM-yyyy" ]
# nmon log timestamp formatted as 00:05:06,04-AUG-2016
}
}
filter {
grok {
match => { "message" => "%{NMON_CPU_LINE}" }
}
if [timetag] == [cpuTimeTag] {
mutate{
add_field => [ "@timestamp", "@timestamp" ]
}
}
}
output {
stdout { codec => rubydebug }
}
With this grok:
MONTHCAPS \b(?:JAN(?:UARY)?|FEB(?:RUARY)?|MAR(?:CH)?|APR(?:IL)?|MAY|JUN(?:E)?|JUL(?:Y)?|AUG(?:UST)?|SEP(?:TEMBER)?|OCT(?:OBER)?|NOV(?:EMBER)?|DEC(?:EMBER)?)\b
NMON_TIMETAG T[0-9]*
NMON_TIME %{HOUR}:%{MINUTE}:%{SECOND}
NMON_DATE %{MONTHDAY}-%{MONTHCAPS}-%{YEAR}
NMON_TIMESTAMP %{NMON_TIME},%{NMON_DATE}
NMON_TIMESTAMP_LINE [Z]{4},%{NMON_TIMETAG:timetag},%{NMON_TIMESTAMP:timestamp}
NMON_CPU_TAG CPU[0-9]*
NMON_CPU_LINE %{NMON_CPU_TAG:cpuTag},%{NMON_TIMETAG:cpuTimeTag},%{NUMBER:User},%{NUMBER:Sys},%{NUMBER:Wait},%{NUMBER:Idle}
...in the hope that I can refer to the ZZZZ line's timestamp and add it to the others, with limited success.
Any pointers in the right direction would be appreciated.
Mark.