Nested repeating group

[nad]

Hi,

I'm starting using filebeat->logstash->ES to ship logs data.
Here is bellow a sample of the application logs:

2020-01-01 01:00:01.590556482 : TOTO [AAA]
GroupX1
  RED  : 2225714
  BLUE : 9119409
  GREEN   : 23122940
GroupY
  RED  : 12356
  BLUE : 46816
  GREEN   : 555887
Group.Z
  RED  : 54567864
  BLUE : 56468768
  GREEN   : 953311
2020-01-01 01:00:01.789555487 : TATA [BBB]
GroupY
  RED  : 4565455
  BLUE : 4564
  GREEN   : 45646
Group.Z
  RED  : 897
  BLUE : 789
  GREEN   : 78

I would like an output like this:

{
	"time" : "2020-01-02 01:00:01.590556482"
	, "username" : "TOTO"
	, "usertype" : "AAA"
	, "groups" : [
		{	"name" : "GroupX1"
			, "RED" : 2225714
			, "BLUE" : 9119409
			, "BLUE" : 23122940
		} ,
		{	"name" : "GroupY"
			, "RED" : 12356
			, "BLUE" : 46816
			, "BLUE" : 555887
		} ,
		{	"name" : "Group.Z"
			, "RED" : 54567864
			, "BLUE" : 56468768
			, "BLUE" : 953311
		}
} ,
{
	"time" : "2020-01-02 01:00:01.789555487"
	, "username" : "TATA"
	, "usertype" : "BBB"
	, "groups" : [
		{	"name" : "GroupW"
			, "RED" : 4565455
			, "BLUE" : 4564
			, "BLUE" : 45646
		} ,
		{	"name" : "Group.Z"
			, "RED" : 897
			, "BLUE" : 789
			, "BLUE" : 78
		}
}

I started with a grok filter like this:

%{TIMESTAMP_ISO8601:time} : %{GREEDYDATA:username} %{SYSLOG5424SD:usertype}
%{GREEDYDATA:[group_1][group_name]}
  RED  : %{NUMBER:[group_1][RED]}
  BLUE : %{NUMBER:[group_1][BLUE]}
  GREEN   : %{NUMBER:[group_1][GREEN]}
%{GREEDYDATA:[group_2][group_name]}
  RED  : %{NUMBER:[group_2][RED]}
  BLUE : %{NUMBER:[group_2][BLUE]}
  GREEN   : %{NUMBER:[group_2][GREEN]}
%{GREEDYDATA:[group_3][group_name]}
  RED  : %{NUMBER:[group_3][RED]}
  BLUE : %{NUMBER:[group_3][BLUE]}
  GREEN   : %{NUMBER:[group_3][GREEN]}

What would be the best strategy to not have to explicitly list the groups (group_1, group_2, group_3)? The number of groups will vary.

Thanks!
nad

[Badger]

I would use ruby

    grok { match => { "message" => "%{TIMESTAMP_ISO8601:time} : %{GREEDYDATA:username} %{SYSLOG5424SD:usertype}%{GREEDYDATA:[@metadata][restOfLine]}" } }
    mutate { gsub => [ "[@metadata][restOfLine]", "\r", "" ] }
    mutate { gsub => [ "usertype", "[\[\]]", "" ] }
    ruby {
        code => "
            data = event.get('[@metadata][restOfLine]')
            matches = data.scan(/\n(\S+)\n\s+RED\s+: ([0-9]+)\n\s+BLUE\s+: ([0-9]+)\n\s+GREEN\s+: ([0-9]+)/)
            groups = []
            matches.each_index { |x|
                group = { 'name' => matches[x][0], 'red' => matches[x][1], 'blue' => matches[x][2], 'green' => matches[x][3] }
                groups << group
            }
            event.set('groups', groups)
        "
    }

You may not need the first mutate. And I assume you want red/blue/green, not red/blue/blue, which you cannot have.

[nad]

@Badger

You may not need the first mutate. And I

Thanks @Badger, I will give it a shot and confirm!
nad