# Nested repeating group

**URL:** https://discuss.elastic.co/t/nested-repeating-group/213638
**Category:** Logstash
**Created:** [January 3, 2020, 12:26am UTC](https://discuss.elastic.co/t/nested-repeating-group/213638 "2020-01-03T00:26:02Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![nad](https://avatars.discourse-cdn.com/v4/letter/n/ed8c4c/32.png) [@nad](https://discuss.elastic.co/u/nad)
#### Post date: [January 3, 2020, 12:26am UTC](https://discuss.elastic.co/t/nested-repeating-group/213638/1 "2020-01-03T00:26:02Z")

</div>

Hi,

I'm starting using filebeat-\>logstash-\>ES to ship logs data.  
Here is bellow a sample of the application logs:

```
2020-01-01 01:00:01.590556482 : TOTO [AAA]
GroupX1
  RED : 2225714
  BLUE : 9119409
  GREEN : 23122940
GroupY
  RED : 12356
  BLUE : 46816
  GREEN : 555887
Group.Z
  RED : 54567864
  BLUE : 56468768
  GREEN : 953311
2020-01-01 01:00:01.789555487 : TATA [BBB]
GroupY
  RED : 4565455
  BLUE : 4564
  GREEN : 45646
Group.Z
  RED : 897
  BLUE : 789
  GREEN : 78

```

I would like an output like this:

```
{
	"time" : "2020-01-02 01:00:01.590556482"
	, "username" : "TOTO"
	, "usertype" : "AAA"
	, "groups" : [
		{	"name" : "GroupX1"
			, "RED" : 2225714
			, "BLUE" : 9119409
			, "BLUE" : 23122940
		} ,
		{	"name" : "GroupY"
			, "RED" : 12356
			, "BLUE" : 46816
			, "BLUE" : 555887
		} ,
		{	"name" : "Group.Z"
			, "RED" : 54567864
			, "BLUE" : 56468768
			, "BLUE" : 953311
		}
} ,
{
	"time" : "2020-01-02 01:00:01.789555487"
	, "username" : "TATA"
	, "usertype" : "BBB"
	, "groups" : [
		{	"name" : "GroupW"
			, "RED" : 4565455
			, "BLUE" : 4564
			, "BLUE" : 45646
		} ,
		{	"name" : "Group.Z"
			, "RED" : 897
			, "BLUE" : 789
			, "BLUE" : 78
		}
}

```

I started with a grok filter like this:

```
%{TIMESTAMP_ISO8601:time} : %{GREEDYDATA:username} %{SYSLOG5424SD:usertype}
%{GREEDYDATA:[group_1][group_name]}
  RED : %{NUMBER:[group_1][RED]}
  BLUE : %{NUMBER:[group_1][BLUE]}
  GREEN : %{NUMBER:[group_1][GREEN]}
%{GREEDYDATA:[group_2][group_name]}
  RED : %{NUMBER:[group_2][RED]}
  BLUE : %{NUMBER:[group_2][BLUE]}
  GREEN : %{NUMBER:[group_2][GREEN]}
%{GREEDYDATA:[group_3][group_name]}
  RED : %{NUMBER:[group_3][RED]}
  BLUE : %{NUMBER:[group_3][BLUE]}
  GREEN : %{NUMBER:[group_3][GREEN]}

```

What would be the best strategy to not have to explicitly list the groups (group\_1, group\_2, group\_3)? The number of groups will vary.

Thanks!  
nad

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [January 3, 2020, 4:52pm UTC](https://discuss.elastic.co/t/nested-repeating-group/213638/2 "2020-01-03T16:52:50Z")

</div>

I would use ruby

```
    grok { match => { "message" => "%{TIMESTAMP_ISO8601:time} : %{GREEDYDATA:username} %{SYSLOG5424SD:usertype}%{GREEDYDATA:[@metadata][restOfLine]}" } }
    mutate { gsub => ["[@metadata][restOfLine]", "\r", "" ] }
    mutate { gsub => ["usertype", "[\[\]]", "" ] }
    ruby {
        code => "
            data = event.get('[@metadata][restOfLine]')
            matches = data.scan(/\n(\S+)\n\s+RED\s+: ([0-9]+)\n\s+BLUE\s+: ([0-9]+)\n\s+GREEN\s+: ([0-9]+)/)
            groups = []
            matches.each_index { |x|
                group = { 'name' => matches[x][0], 'red' => matches[x][1], 'blue' => matches[x][2], 'green' => matches[x][3] }
                groups << group
            }
            event.set('groups', groups)
        "
    }

```

You may not need the first mutate. And I assume you want red/blue/green, not red/blue/blue, which you cannot have.

---

<div class="post-metadata">

### Author: ![nad](https://avatars.discourse-cdn.com/v4/letter/n/ed8c4c/32.png) [@nad](https://discuss.elastic.co/u/nad)
#### Post date: [January 3, 2020, 8:48pm UTC](https://discuss.elastic.co/t/nested-repeating-group/213638/3 "2020-01-03T20:48:12Z")

</div>

> [@Badger](#):
>
> ```auto
> 
> ```
> 
> You may not need the first mutate. And I

Thanks @Badger, I will give it a shot and confirm!  
nad

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 31, 2020, 8:48pm UTC](https://discuss.elastic.co/t/nested-repeating-group/213638/4 "2020-01-31T20:48:16Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.