How to define multiline in filebeat.inputs base on image?

Elastic Stack Beats
1

Hey, in our cluster some apps are sending logs as multiline, and the problem is that the log structure is different from app to app.

How can we set up an 'if' condition that will include the

        multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
        multiline.negate: true
        multiline.match: after

In it?

Our code:

  filebeatConfig:
    filebeat.yml: |
      filebeat.inputs:
      - type: container
        paths:
          - /var/log/containers/*.log
        processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"
        - drop_event:
            when:
              contains:
                container.image.name: "kibana"
              

      output.logstash:
        hosts: ["logstash-listener:5044"]
2

Multiline will only apply if it doesn't detect the start of your pattern. That means you can define the pattern and it'll skip individual entries. The docs mention this;

Filebeat takes all the lines that do not start with [ and combines them with the previous line that does.

However if you have multiple multiline patterns you need to define, then it gets a tonne harder.

3

Hey, the problem is that I also have apps which send logs as single line that does not start in the same pattern as the apps that send multiline.

For example app A will send regular one line log:

action started, log level 5, connected

and app B will send multiline log:

2016-10-14 20:31:07,447 INFO  [ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' ...
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.