Parsing Logs based on a Regex sequence (Prune, Grok, Mutate)

Elastic Stack Logstash
1

The set-up I currently have is Filebeat --> Logstash --> Elasticsearch --> Kibana.

Filebeat naturally comes with a series of fields (that can be turned off in filebeat.yml) and I am trying to parse them in Logstash so certain fields are removed.

Instead of having to write out every field I may or may not want in the outputted logs, I was hoping to use Regex to only remove select fields.

For example, Filebeat adds on a series of agent.* fields such as agent.id or agent.hostname. The regex I was hoping to use was the following, as proven in regex101, in conjunction with a remove_field function.
image

I have tried this setup with prune, grok and mutate but this regex string will not work, despite other (more basic) regex strings working such as agent.* . (Note, I realise that Prune appears to be the only plugin that supports regex, very tired after not getting anywhere)

(Note only one plugin is used at one time, this is for demonstration purpose)
image

To note, as a result of the parsing, for example when only grok remove_field is being used, the tag _grokparsefaliure is added.

2

I do not think filebeat adds a field called agent.hostname (a field name containing a period). I believe it adds an agent object that contains a hostname field (and a type field etc.). The prune filter only works on top level fields, so you could blacklist [agent], but not [agent][id].

3

Understood, that makes a lot of sense. Do you, or anyone watching the thread, know of a Logstash filter that could parse out an agent object?

4

better to use ruby-code filter, there you can write ruby program to parse the agent object.

5

I am not familiar with Ruby, is there any documentation I can be pointed towards.

Thanks

6

once go through the following logstash guidelines to know the working of ruby-code filter and using event APIs,
https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html#plugins-filters-ruby-code
https://www.elastic.co/guide/en/logstash/current/event-api.html

and for ruby programming you can refer some basic tutorials with JSON parsing.