I have setup filbeat on Kubernetes (ECK) with sample and guide from docs:
- Role Based Access Control for Beats | Elastic Cloud on Kubernetes [2.5]
- Run Filebeat on Kubernetes | Filebeat Reference [8.5]
Version:
ECK: 2.5.0
filebeat: 8.5.3
The filebeat deployment and configuration already running.
Here's some manifest snipet that used for deploy the filebeat:
$ cat filebeat-eck.autodiscover.yaml
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
name: elastic
spec:
type: filebeat
version: 8.5.3
elasticsearchRef:
name: elastic
config:
filebeat:
autodiscover:
providers:
- node: ${NODE_NAME}
type: kubernetes
hints:
enabled: true
default_config:
type: container
paths:
- /var/log/containers/*${data.kubernetes.container.id}.log
daemonSet:
podTemplate:
spec:
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
securityContext:
runAsUser: 0
...
Kubernetes logs has been successfully parsed. However, In the image example, I have specified Apps/Pod that running NGINX and I want to process the "message" field with NGINX filebeat module. Since the "message" field is exactly match the filebeat module for NGINX.
I've read some possible feature realated to "processor" in the filebeat.
The document said that:
The libbeat library provides processors for:
- reducing the number of exported fields
- enhancing events with additional metadata-
- performing additional processing and decoding
So it can be used for performing additional processing and decoding. it's amazing feature.
event -> processor 1 -> event1 -> processor 2 -> event2 ...
However, from the docs I can assume that it can process event with pre-defined/supported "processor":
I see a there's "dissect" processor that can be used to add custom enrich for specific field (Like we can use with logstash). However, the custom filter/grok actually is not what's expected here, since the filebeat itself has many of built-in module (that include pipeline/filter), i.e: nginx.
Is that possible to do something like this?
[module] -> [ event ] -> [ processor ] ( to process specific field with antother filebeat module) -> [ event ] -> ...
I have considered for post-processing with Logstash. However, if that's possible to be done in filebeat using pre-define modulle/processor it would be better.
I've take a look some possible feature:
- Hints based autodiscover | Filebeat Reference [8.5] | Elastic
- Example: Parse logs in the Common Log Format | Elasticsearch Guide [8.5] | Elastic
If there's more best practice to do this, or any help and suggestions would be greatly appreciated!
Thanks!