# Parsing Logs based on a Regex sequence (Prune, Grok, Mutate) **URL:** \ **Category:** Logstash\ **Created:** [May 29, 2020, 2:24pm UTC](https://discuss.elastic.co/t/parsing-logs-based-on-a-regex-sequence-prune-grok-mutate/234933 "2020-05-29T14:24:03Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![clearedskies](https://avatars.discourse-cdn.com/v4/letter/c/858c86/32.png) [@clearedskies](https://discuss.elastic.co/u/clearedskies)\ **Post date:** [May 29, 2020, 2:24pm UTC](https://discuss.elastic.co/t/parsing-logs-based-on-a-regex-sequence-prune-grok-mutate/234933/1 "2020-05-29T14:24:03Z")
The set-up I currently have is Filebeat --\> Logstash --\> Elasticsearch --\> Kibana. Filebeat naturally comes with a series of fields (that can be turned off in filebeat.yml) and I am trying to parse them in Logstash so certain fields are removed. Instead of having to write out every field I may or may not want in the outputted logs, I was hoping to use Regex to only remove select fields. For example, Filebeat adds on a series of agent.\* fields such as agent.id or agent.hostname. The regex I was hoping to use was the following, as proven in regex101, in conjunction with a remove\_field function. ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/a/4a02d6863e59ae94c45674f26499f3c9cf95b09d.png) I have tried this setup with prune, grok and mutate but this regex string will not work, despite other (more basic) regex strings working such as agent.\* . (Note, I realise that Prune appears to be the only plugin that supports regex, very tired after not getting anywhere) (Note only one plugin is used at one time, this is for demonstration purpose) ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/3/931c577e799c7bceac2e06b9ab80efbe60fcb382.png) To note, as a result of the parsing, for example when only grok remove\_field is being used, the tag \_grokparsefaliure is added. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [May 29, 2020, 2:54pm UTC](https://discuss.elastic.co/t/parsing-logs-based-on-a-regex-sequence-prune-grok-mutate/234933/2 "2020-05-29T14:54:09Z")
I do not think filebeat adds a field called agent.hostname (a field name containing a period). I believe it adds an agent object that contains a hostname field (and a type field etc.). The prune filter only works on top level fields, so you could blacklist [agent], but not [agent][id]. ---
**Author:** ![clearedskies](https://avatars.discourse-cdn.com/v4/letter/c/858c86/32.png) [@clearedskies](https://discuss.elastic.co/u/clearedskies)\ **Post date:** [June 1, 2020, 8:16am UTC](https://discuss.elastic.co/t/parsing-logs-based-on-a-regex-sequence-prune-grok-mutate/234933/3 "2020-06-01T08:16:50Z")
Understood, that makes a lot of sense. Do you, or anyone watching the thread, know of a Logstash filter that could parse out an agent object? ---
**Author:** ![chitreshg](https://avatars.discourse-cdn.com/v4/letter/c/df705f/32.png) [@chitreshg](https://discuss.elastic.co/u/chitreshg)\ **Post date:** [June 1, 2020, 11:17am UTC](https://discuss.elastic.co/t/parsing-logs-based-on-a-regex-sequence-prune-grok-mutate/234933/4 "2020-06-01T11:17:12Z")
> [@clearedskies](#): > > o you, or anyone watching the thread, know of a Logstash filter that could parse out an agent object? better to use ruby-code filter, there you can write ruby program to parse the agent object. ---
**Author:** ![clearedskies](https://avatars.discourse-cdn.com/v4/letter/c/858c86/32.png) [@clearedskies](https://discuss.elastic.co/u/clearedskies)\ **Post date:** [June 1, 2020, 11:32am UTC](https://discuss.elastic.co/t/parsing-logs-based-on-a-regex-sequence-prune-grok-mutate/234933/5 "2020-06-01T11:32:09Z")
I am not familiar with Ruby, is there any documentation I can be pointed towards. Thanks ---
**Author:** ![chitreshg](https://avatars.discourse-cdn.com/v4/letter/c/df705f/32.png) [@chitreshg](https://discuss.elastic.co/u/chitreshg)\ **Post date:** [June 1, 2020, 12:18pm UTC](https://discuss.elastic.co/t/parsing-logs-based-on-a-regex-sequence-prune-grok-mutate/234933/6 "2020-06-01T12:18:48Z")
once go through the following logstash guidelines to know the working of ruby-code filter and using event APIs, [https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html#plugins-filters-ruby-code](https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html#plugins-filters-ruby-code) [https://www.elastic.co/guide/en/logstash/current/event-api.html](https://www.elastic.co/guide/en/logstash/current/event-api.html) and for ruby programming you can refer some basic tutorials with JSON parsing. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 29, 2020, 12:19pm UTC](https://discuss.elastic.co/t/parsing-logs-based-on-a-regex-sequence-prune-grok-mutate/234933/7 "2020-06-29T12:19:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.