Filebeats in kubernetes - how to export specific log file in a pod to logstash?

[sivakr (Siva Kumar)]

I have few pods with label app1

I want to send some log files of those specific pods with custom tags to logstash.

Ex: /tmp/*.log of all pods with label: app1 should be sent to logstash with tag envoy-access-log

I have filebeat config like this

  filebeat.autodiscover:
    providers:
      - type: kubernetes
        templates:
          - condition.contains:
              kubernetes.labels.app: app1
            config:
              prospectors:
                paths:
                   - /tmp/*.log
                tags: ["envoy-access-log"]

  logging.metrics.enabled: false
  output.logstash:
    hosts: ["logstash.monit:5044"]

But Its not working. How can I achieve this ?

[jsoriano (Jaime Soriano)]

I think there is a missing hyphen before prospectors. The config option has to contain a list of configurations, in your case it'd be something like:

  filebeat.autodiscover:
    providers:
      - type: kubernetes
        templates:
          - condition.contains:
              kubernetes.labels.app: app1
            config:
              - prospectors:
                  paths:
                     - /tmp/*.log
                  tags: ["envoy-access-log"]

  logging.metrics.enabled: false
  output.logstash:
    hosts: ["logstash.monit:5044"]

You can also use the docker prospector to collect container logs, take a look to the examples in the documentation: https://www.elastic.co/guide/en/beats/filebeat/6.2/configuration-autodiscover.html