Filebeat kafka input using multiline parser gives no output

Elastic Stack Beats
,
1

I have posted the issue on stackoverflow as well
Filebeat kafka input using multiline parser gives no output

Filebeat is configured to use input from kafka and output to file

When the multiline setting is turned off, the output is published to a file. But when kafka input is configured with mutiline , no output in the file(file is not even created)

Here is relevant filebeat configuration
Filebeat is configured to use input from kafka and output to file

When the multiline setting is turned off, the output is published to a file. But when kafka input is configured with mutiline , no output in the file(file is not even created)

Here is relevant filebeat configuration

Input configuration

filebeat.inputs:
  - type: kafka
    hosts:
      - <ip>:9092
    topics:
      - "my-multiline-log"
    group_id: "kafka-consumer-filebeat"
    parsers:
      - multiline:
          #          type: pattern
          pattern: '^'
          negate: true
          match: after

Output Configuration:

output.file:
  path: "/tmp/filebeat"
  filename: filebeat
#  codec.format:
#    string: '%{[message]}'

Filebeat relevant logs

2021-12-16T11:02:34.551Z    INFO    [input.kafka]   compat/compat.go:111    Input kafka starting    {"id": "19A7FFEEC9EDFC04"}
2021-12-16T11:02:34.551Z    INFO    [input.kafka.kafka input]   kafka/input.go:129  Starting Kafka input    {"id": "19A7FFEEC9EDFC04", "hosts": ["<ip>:9092"]}
2021-12-16T11:02:38.158Z    DEBUG   [reader_multiline]  multiline/pattern.go:142    Multiline event flushed because timeout reached.
2021-12-16T11:02:44.767Z    DEBUG   [reader_multiline]  multiline/pattern.go:142    Multiline event flushed because timeout reached.
2021-12-16T11:02:51.481Z    DEBUG   [reader_multiline]  multiline/pattern.go:142    Multiline event flushed because timeout reached.
2021-12-16T11:02:58.225Z    DEBUG   [reader_multiline]  multiline/pattern.go:142    Multiline event flushed because timeout reached.
2021-12-16T11:03:04.555Z    DEBUG   cgroup/util.go:276  PID 1 contains a cgroups V2 path (0::/) but no V2 mountpoint was found.
This may be because metricbeat is running inside a container on a hybrid system.
To monitor cgroups V2 processess in this way, mount the unified (V2) hierarchy inside
the container as /sys/fs/cgroup/unified and start metricbeat with --system.hostfs.

Am I configuring something incorrectly?

P.S. disabling the multiline configuration and setting output codec to %message, I get the output as is as input. This ensures filebeat is able to read from kafka

2

The support for multi-line parser has been added to filebeat in version 7.16

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.