Log entries gathered by filebeat in EKS are occasionally joined

Elastic Stack Beats
3

The solution here appears to be to use filebeat autodiscover. Individual configs can then be written, per normal filebeat config rules.

It's almost as if someone else has encountered this issue before. :smile:

My filebeat.yml looks like:

---
filebeat:
  autodiscover:
    providers:
      path: '${path.config}/providers.d/*.yml'
      reload:
        enabled: false

logging:
  # Minimum log level. One of debug, info, warning, or error. The
  # default log level is info.
  level: 'info'
output:
  logstash:
    hosts:
      - 'logstash'
    index: 'filebeat'
    ssl:
      enabled: true
      verification_mode: 'none'

and my provider.yml is:

---
- type: 'kubernetes'
  templates:
    - condition:
        and:
          - equals:
              kubernetes.namespace: 'mynamespace'
          - equals:
              kubernetes.container.image: 'myimage'
      config:
        - type: 'container'
          paths: '/var/lib/docker/containers/${data.docker.container.id}/*.log'
          multiline:
            pattern: '^[\{\[]'
            negate: true
            match: 'after'
      processors:
        - add_locale:
            format: 'offset'