The set-up I currently have is Filebeat --> Logstash --> Elasticsearch --> Kibana.
Filebeat naturally comes with a series of fields (that can be turned off in filebeat.yml) and I am trying to parse them in Logstash so certain fields are removed.
Instead of having to write out every field I may or may not want in the outputted logs, I was hoping to use Regex to only remove select fields.
For example, Filebeat adds on a series of agent.* fields such as agent.id or agent.hostname. The regex I was hoping to use was the following, as proven in regex101, in conjunction with a remove_field function.
I have tried this setup with prune, grok and mutate but this regex string will not work, despite other (more basic) regex strings working such as agent.* . (Note, I realise that Prune appears to be the only plugin that supports regex, very tired after not getting anywhere)
(Note only one plugin is used at one time, this is for demonstration purpose)
To note, as a result of the parsing, for example when only grok remove_field is being used, the tag _grokparsefaliure is added.
I do not think filebeat adds a field called agent.hostname (a field name containing a period). I believe it adds an agent object that contains a hostname field (and a type field etc.). The prune filter only works on top level fields, so you could blacklist [agent], but not [agent][id].
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
