# Nested field in a grok filter **URL:** \ **Category:** Logstash\ **Created:** [July 16, 2019, 2:37pm UTC](https://discuss.elastic.co/t/nested-field-in-a-grok-filter/190758 "2019-07-16T14:37:22Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![theo1991](https://avatars.discourse-cdn.com/v4/letter/t/4da419/32.png) [@theo1991](https://discuss.elastic.co/u/theo1991)\ **Post date:** [July 16, 2019, 2:37pm UTC](https://discuss.elastic.co/t/nested-field-in-a-grok-filter/190758/1 "2019-07-16T14:37:22Z")
Hi, I have a pattern for log which is like this : inactive: No, sources : [{id: 1, type: fr, name:custom}, {id:2, type:fr, name: random}], assigned: Yes I wrote this custom grok filter : inactive: %{WORD:inactive}, sources : [id: %{NUMBER:id}, type: %{WORD:fr}, name: {WORD:name}], assigned: %{WORD:assigned} But the problem is that I want a nested structure like source1 : id1: type1: name1: source2: etc I tried %{NUMBER:[source][id] for example but it doesn't work. Any help ? Thank you. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 16, 2019, 3:05pm UTC](https://discuss.elastic.co/t/nested-field-in-a-grok-filter/190758/2 "2019-07-16T15:05:13Z")
``` grok { match => { "message" => "^inactive: %{WORD:inactive}, sources : \[(?<[@metadata][sources]>[^\]]+)\], assigned: %{WORD:assigned}" } } ruby { code => ' matches = event.get("[@metadata][sources]").scan(/{id:\s*([0-9]+), type:\s*([a-zA-Z0-9]+), name:\s*([a-zA-Z0-9]+)}/) event.set("matches", matches) ' } ``` will get you to ``` "matches" => [ [0] [ [0] "1", [1] "fr", [2] "custom" ], [1] [ [0] "2", [1] "fr", [2] "random" ] ] ``` It is unclear what structure you want that data in, so you might be able to move stuff around using mutate, or you may need ruby. ---
**Author:** ![theo1991](https://avatars.discourse-cdn.com/v4/letter/t/4da419/32.png) [@theo1991](https://discuss.elastic.co/u/theo1991)\ **Post date:** [July 17, 2019, 10:13am UTC](https://discuss.elastic.co/t/nested-field-in-a-grok-filter/190758/3 "2019-07-17T10:13:43Z")
Hello, Thanks for your help. Unfortunately, this doesn't work. Logstash failed to restart because of invalid syntax in the ruby filter. That's it : ``` ruby { code => ' matches = event.get("[@metadata][log_sources]").scan(/{u'id': ([0-9]+), u'type_name': u'([a-zA-Z0-9\D\s]+)', u'name': u'([a-zA-Z0-9\D\s]+)', u'type_id': ([0-9]+)/}) event.set("matches", matches) ' } ``` The payload in the data is : `{u'id': 50, u'type_name': u'Audit', u'name': u'Audit @ 10.10.10.10', u'type_id': 38}, {u'id': 55, u'type_name': u'Audit', u'name': u'Audit @ 10.10.10.10', u'type_id': 41},` I put `\[(?<[@metadata][sources]>[^\]]+)\]` in the grok filter and it seems good if I don't write the ruby filter. ---
**Author:** ![theo1991](https://avatars.discourse-cdn.com/v4/letter/t/4da419/32.png) [@theo1991](https://discuss.elastic.co/u/theo1991)\ **Post date:** [July 17, 2019, 12:58pm UTC](https://discuss.elastic.co/t/nested-field-in-a-grok-filter/190758/4 "2019-07-17T12:58:58Z")
I also tried these : ``` ruby { code => ' matches = event.get("[@metadata][log_sources]").scan(\{u\'id\': ([0-9]+), u\'type_name\': u\'([a-zA-Z0-9\D\s]+)\', u\'name\': u\'([a-zA-Z0-9\D\s]+)\', u\'type_id\': ([0-9]+)\}) event.set("matches", matches) ' } ruby { code => ' matches = event.get("[@metadata][log_sources]").scan(/{u\'id\': ([0-9]+), u\'type_name\': u\'([a-zA-Z0-9\D\s]+)\', u\'name\': u\'([a-zA-Z0-9\D\s]+)\', u\'type_id\': ([0-9]+)/}) event.set("matches", matches) ' } ruby { code => " matches = event.get('[@metadata][log_sources]').scan(/{u'id': ([0-9]+), u'type_name': u'([a-zA-Z0-9\D\s]+)', u'name': u'([a-zA-Z0-9\D\s]+)', u'type_id': ([0-9]+)/}) event.set('matches', matches) " } ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 17, 2019, 2:10pm UTC](https://discuss.elastic.co/t/nested-field-in-a-grok-filter/190758/5 "2019-07-17T14:10:07Z")
> [@theo1991](#): > > ruby { code =\> " matches = event.get('[@metadata][log\_sources]').scan(/{u'id': ([0-9]+), u'type\_name': u'([a-zA-Z0-9\D\s]+)', u'name': u'([a-zA-Z0-9\D\s]+)', u'type\_id': ([0-9]+)/}) event.set('matches', matches) " } There are a couple of problems with this. scan takes a regexp, which is delimited by /. At the end of the call to scan you have ``` u'type_id': ([0-9]+)/}) ``` which should be ``` u'type_id': ([0-9]+)}/) ``` The other problem is that by adding \D\s in order to capture the IP address, you actually cause the regexp to get really grabby and capture almost the entire string. Provided that the name of the field in the grok is consistent with the name in the ruby code ([@metadata][log\_sources] vs. [@metadata][sources]) then this ``` ruby { code => " matches = event.get('[@metadata][log_sources]').scan(/{u'id': ([0-9]+), u'type_name': u'([^']+)', u'name': u'([^']+)', u'type_id': ([0-9]+)}/) event.set('matches', matches) " } ``` will get you ``` "matches" => [ [0] [ [0] "50", [1] "Audit", [2] "Audit @ 10.10.10.10", [3] "38" ], [1] [ [0] "55", [1] "Audit", [2] "Audit @ 10.10.10.10", [3] "41" ] ] ``` ---
**Author:** ![theo1991](https://avatars.discourse-cdn.com/v4/letter/t/4da419/32.png) [@theo1991](https://discuss.elastic.co/u/theo1991)\ **Post date:** [July 17, 2019, 2:17pm UTC](https://discuss.elastic.co/t/nested-field-in-a-grok-filter/190758/6 "2019-07-17T14:17:58Z")
Wonderful ! Thank you very much ! It sounds clear now ! So I try that, and logstash start but I have these error in logs, and I don't see the fields in kibana : Jul 17 16:13:53 localhost logstash: [2019-07-17T16:13:53,620][ERROR][logstash.filters.ruby] Ruby exception occurred: undefined method `scan' for nil:NilClass ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 17, 2019, 2:22pm UTC](https://discuss.elastic.co/t/nested-field-in-a-grok-filter/190758/7 "2019-07-17T14:22:34Z")
> [@theo1991](#): > > Ruby exception occurred: undefined method `scan' for nil:NilClass That means that event.get did not return anything. You could try ``` code => " log_sources = event.get('[@metadata][log_sources]') if log_sources matches = log_sources.scan(/{u'id': ([0-9]+), u'type_name': u'([^']+)', u'name': u'([^']+)', u'type_id': ([0-9]+)}/) event.set('matches', matches) end " ``` ---
**Author:** ![theo1991](https://avatars.discourse-cdn.com/v4/letter/t/4da419/32.png) [@theo1991](https://discuss.elastic.co/u/theo1991)\ **Post date:** [July 17, 2019, 3:10pm UTC](https://discuss.elastic.co/t/nested-field-in-a-grok-filter/190758/8 "2019-07-17T15:10:24Z")
Everything seems right ! Thank you very very much Badger 😃 A last little question , I have other part in my log file which have also nested patterns. Do I have to add the other ones after the code =\> " like this ? ``` code => " log_sources = event.get('[@metadata][log_sources]') if log_sources matches = log_sources.scan(/{u'id': ([0-9]+), u'type_name': u'([^']+)', u'name': u'([^']+)', u'type_id': ([0-9]+)}/) event.set('matches', matches) end rules = event.get ... " ``` Edit : Another last question, if for one line of log, there is just one id, type\_name, name, and type\_id in [log\_sources], is it recorded in kibana ? If not, how can we do this ? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 17, 2019, 3:22pm UTC](https://discuss.elastic.co/t/nested-field-in-a-grok-filter/190758/9 "2019-07-17T15:22:31Z")
Yes, you can do the same type of treatment for more than one type of log line. scan should return an array with a single member if there is only one match in log\_sources, so it should show up in Kibana. ---
**Author:** ![theo1991](https://avatars.discourse-cdn.com/v4/letter/t/4da419/32.png) [@theo1991](https://discuss.elastic.co/u/theo1991)\ **Post date:** [July 18, 2019, 7:21am UTC](https://discuss.elastic.co/t/nested-field-in-a-grok-filter/190758/10 "2019-07-18T07:21:29Z")
Thank you Badger 😃 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 15, 2019, 7:27am UTC](https://discuss.elastic.co/t/nested-field-in-a-grok-filter/190758/11 "2019-08-15T07:27:33Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.