Hi,
I'm starting using filebeat->logstash->ES to ship logs data.
Here is bellow a sample of the application logs:
2020-01-01 01:00:01.590556482 : TOTO [AAA]
GroupX1
RED : 2225714
BLUE : 9119409
GREEN : 23122940
GroupY
RED : 12356
BLUE : 46816
GREEN : 555887
Group.Z
RED : 54567864
BLUE : 56468768
GREEN : 953311
2020-01-01 01:00:01.789555487 : TATA [BBB]
GroupY
RED : 4565455
BLUE : 4564
GREEN : 45646
Group.Z
RED : 897
BLUE : 789
GREEN : 78
I would like an output like this:
{
"time" : "2020-01-02 01:00:01.590556482"
, "username" : "TOTO"
, "usertype" : "AAA"
, "groups" : [
{ "name" : "GroupX1"
, "RED" : 2225714
, "BLUE" : 9119409
, "BLUE" : 23122940
} ,
{ "name" : "GroupY"
, "RED" : 12356
, "BLUE" : 46816
, "BLUE" : 555887
} ,
{ "name" : "Group.Z"
, "RED" : 54567864
, "BLUE" : 56468768
, "BLUE" : 953311
}
} ,
{
"time" : "2020-01-02 01:00:01.789555487"
, "username" : "TATA"
, "usertype" : "BBB"
, "groups" : [
{ "name" : "GroupW"
, "RED" : 4565455
, "BLUE" : 4564
, "BLUE" : 45646
} ,
{ "name" : "Group.Z"
, "RED" : 897
, "BLUE" : 789
, "BLUE" : 78
}
}
I started with a grok filter like this:
%{TIMESTAMP_ISO8601:time} : %{GREEDYDATA:username} %{SYSLOG5424SD:usertype}
%{GREEDYDATA:[group_1][group_name]}
RED : %{NUMBER:[group_1][RED]}
BLUE : %{NUMBER:[group_1][BLUE]}
GREEN : %{NUMBER:[group_1][GREEN]}
%{GREEDYDATA:[group_2][group_name]}
RED : %{NUMBER:[group_2][RED]}
BLUE : %{NUMBER:[group_2][BLUE]}
GREEN : %{NUMBER:[group_2][GREEN]}
%{GREEDYDATA:[group_3][group_name]}
RED : %{NUMBER:[group_3][RED]}
BLUE : %{NUMBER:[group_3][BLUE]}
GREEN : %{NUMBER:[group_3][GREEN]}
What would be the best strategy to not have to explicitly list the groups (group_1, group_2, group_3)? The number of groups will vary.
Thanks!
nad