The set-up I currently have is Filebeat --> Logstash --> Elasticsearch --> Kibana.
Filebeat naturally comes with a series of fields (that can be turned off in filebeat.yml) and I am trying to parse them in Logstash so certain fields are removed.
Instead of having to write out every field I may or may not want in the outputted logs, I was hoping to use Regex to only remove select fields.
For example, Filebeat adds on a series of agent.* fields such as agent.id or agent.hostname. The regex I was hoping to use was the following, as proven in regex101, in conjunction with a remove_field function.
I have tried this setup with prune, grok and mutate but this regex string will not work, despite other (more basic) regex strings working such as agent.* . (Note, I realise that Prune appears to be the only plugin that supports regex, very tired after not getting anywhere)
(Note only one plugin is used at one time, this is for demonstration purpose)
To note, as a result of the parsing, for example when only grok remove_field is being used, the tag _grokparsefaliure is added.