How to collecting specfic logs with filebeat

I have Filebeat to collecting logs from my kubernetes cluster. In Filebeat configuration I using autodiscover with type kubernetes which collecting logs from stdout container.

This is my filebeat.yml:

filebeat.autodiscover:
      providers:
       - type: kubernetes
         node: ${NODE_NAME}
         hints.enabled: true
         templates:
            - condition:
                equals:
                  kubernetes.namespace: default
              config:
                - type: container
                  paths:
                    - /var/log/containers/*${data.kubernetes.container.id}.log
                  # exclude_lines: ['INFO', '200']

         # hints.default_config:
         #   type: container
         #   paths:
         #     - /var/log/containers/*${data.kubernetes.container.id}.log
    # processors:
    #   - drop_event:
    #       when.contains:
    #         message: 'GET'

    output.logstash:
      hosts: ['logstash:5044']

And this is the example logs that comes out from stdout:

130.211.2.198 - - [03/Nov/2021:04:38:10 +0000] "GET / HTTP/1.1" 200 2278 "-" "GoogleHC/1.0"
130.211.3.216 - - [03/Nov/2021:04:38:11 +0000] "GET / HTTP/1.1" 200 2278 "-" "GoogleHC/1.0"
130.211.3.180 - - [03/Nov/2021:04:38:20 +0000] "GET / HTTP/1.1" 200 2278 "-" "GoogleHC/1.0"
[2021-10-29 07:16:45] production.ERROR: strtolower(): Argument #1 ($string) must be of type string, array given {"userId":8,"exception":"[object] (TypeError(code: 0): strtolower(): Argument #1 ($string) must be of type string, array given at /app/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php:835)
[stacktrace]
etc

I also have running application such as Laravel running on my container and I set the log config of Laravel to stderr. Since the container logs and Laravel logs been together now, I want to Filebeat just collecting the Laravel instead of both with container. I've been trying to use processors with drop_event to ignore the logs except Laravel log, but still not worked. How to collecting the specific logs with Filebeat?

Many Thanks

#UPDATES

I finally did it with regexp and added '(^\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}).*$' to match the access logs container. But, now my filebeat just like collecting themselves because the message's value was the part of the log just like this

       "message" => "                  \"id\" => \"ddbb97ce-3d76-4846-bbb6-89442ce5cef3\",",

where the id is part of the log that send to logstash. how to fix this? thanks

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.