Mulitline pattern in docker autodiscover

Elastic Stack Beats
1

I have the following logs coming from a docker container:

    2021-03-17T13:43:43+0000 DEBUG    [__main__.<module>] Debug
    2021-03-17T13:43:43+0000 INFO     [__main__.<module>] Info
    2021-03-17T13:43:43+0000 WARNING  [__main__.<module>] Warn
    2021-03-17T13:43:43+0000 ERROR    [__main__.<module>] Error
    2021-03-17T13:43:43+0000 ERROR    [__main__.<module>] Exception
    Traceback (most recent call last):
      File "test.py", line 45, in <module>
        raise Exception("Test")
    Exception: Test
    2021-03-17T13:43:43+0000 CRITICAL [__main__.<module>] Critical
    2021-03-17T13:43:58+0000 DEBUG    [__main__.<module>] Debug
    2021-03-17T13:43:58+0000 INFO     [__main__.<module>] Info
    2021-03-17T13:43:58+0000 WARNING  [__main__.<module>] Warn
    2021-03-17T13:43:58+0000 ERROR    [__main__.<module>] Error
    2021-03-17T13:43:58+0000 ERROR    [__main__.<module>] Exception

I can parse the sections (time, log level, logger and message) just fine with logstash, the problem is the multiline exception. From what I have gathered, this should not occur in logstash, but rather filebeat.
I am using autodiscover with type hints to handle several different applications:

    filebeat:
      config:
        modules:
          path: ${path.config}/modules.d/*.yml
          reload.enabled: false

      autodiscover:
        providers:
          - type: docker
            hints.enabled: true

I have tried using templates here, but to no avail:

    filebeat:
      config:
        modules:
          path: ${path.config}/modules.d/*.yml
          reload.enabled: false

      autodiscover:
        providers:
          - type: docker
            hints.enabled: true
            templates:
              - condition:
                  contains:
                    docker.container.name: "test"
                config:
                  - input_type: container
                    combine_partial: true
                    multiline.pattern: '^\d\d\d\d-\d\d-\d\d'
                    multiline.negate: true
                    multiline.match: after

With this config for example it stops sending logs to logstash all together

2

I think you are on the right track with using templates. But input_type: container looks incorrect to me. I think it should be just type: container.

Shaunak

3

Thank you, but sadly no change with

filebeat:
  config:
    modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: false

  autodiscover:
    providers:
      - type: docker
        hints.enabled: true
        templates:
          - condition:
              contains:
                docker.container.name: "test"
            config:
              - type: container
                combine_partial: true
                multiline.pattern: '^\d\d\d\d-\d\d-\d\d'
                multiline.negate: true
                multiline.match: after

As previously it stops sending logs from this container to logstash (I have a stdout{} there). Meanwhile filebeat does not log any problems or anything for that matter..

Can I try something else? Or get you some specific information from filebeat?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.