I have Filebeat to collecting logs from my kubernetes cluster. In Filebeat configuration I using autodiscover with type kubernetes which collecting logs from stdout container.
This is my filebeat.yml:
filebeat.autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
hints.enabled: true
templates:
- condition:
equals:
kubernetes.namespace: default
config:
- type: container
paths:
- /var/log/containers/*${data.kubernetes.container.id}.log
# exclude_lines: ['INFO', '200']
# hints.default_config:
# type: container
# paths:
# - /var/log/containers/*${data.kubernetes.container.id}.log
# processors:
# - drop_event:
# when.contains:
# message: 'GET'
output.logstash:
hosts: ['logstash:5044']
And this is the example logs that comes out from stdout:
130.211.2.198 - - [03/Nov/2021:04:38:10 +0000] "GET / HTTP/1.1" 200 2278 "-" "GoogleHC/1.0"
130.211.3.216 - - [03/Nov/2021:04:38:11 +0000] "GET / HTTP/1.1" 200 2278 "-" "GoogleHC/1.0"
130.211.3.180 - - [03/Nov/2021:04:38:20 +0000] "GET / HTTP/1.1" 200 2278 "-" "GoogleHC/1.0"
[2021-10-29 07:16:45] production.ERROR: strtolower(): Argument #1 ($string) must be of type string, array given {"userId":8,"exception":"[object] (TypeError(code: 0): strtolower(): Argument #1 ($string) must be of type string, array given at /app/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php:835)
[stacktrace]
etc
I also have running application such as Laravel running on my container and I set the log config of Laravel to stderr. Since the container logs and Laravel logs been together now, I want to Filebeat just collecting the Laravel instead of both with container. I've been trying to use processors with drop_event to ignore the logs except Laravel log, but still not worked. How to collecting the specific logs with Filebeat?
Many Thanks
#UPDATES
I finally did it with regexp and added '(^\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}).*$' to match the access logs container. But, now my filebeat just like collecting themselves because the message's value was the part of the log just like this
"message" => " \"id\" => \"ddbb97ce-3d76-4846-bbb6-89442ce5cef3\",",
where the id is part of the log that send to logstash. how to fix this? thanks