I have posted the issue on stackoverflow as well
Filebeat kafka input using multiline parser gives no output
Filebeat is configured to use input from kafka and output to file
When the multiline setting is turned off, the output is published to a file. But when kafka input is configured with mutiline , no output in the file(file is not even created)
Here is relevant filebeat configuration
Filebeat is configured to use input from kafka and output to file
When the multiline setting is turned off, the output is published to a file. But when kafka input is configured with mutiline , no output in the file(file is not even created)
Here is relevant filebeat configuration
Input configuration
filebeat.inputs:
- type: kafka
hosts:
- <ip>:9092
topics:
- "my-multiline-log"
group_id: "kafka-consumer-filebeat"
parsers:
- multiline:
# type: pattern
pattern: '^'
negate: true
match: after
Output Configuration:
output.file:
path: "/tmp/filebeat"
filename: filebeat
# codec.format:
# string: '%{[message]}'
Filebeat relevant logs
2021-12-16T11:02:34.551Z INFO [input.kafka] compat/compat.go:111 Input kafka starting {"id": "19A7FFEEC9EDFC04"}
2021-12-16T11:02:34.551Z INFO [input.kafka.kafka input] kafka/input.go:129 Starting Kafka input {"id": "19A7FFEEC9EDFC04", "hosts": ["<ip>:9092"]}
2021-12-16T11:02:38.158Z DEBUG [reader_multiline] multiline/pattern.go:142 Multiline event flushed because timeout reached.
2021-12-16T11:02:44.767Z DEBUG [reader_multiline] multiline/pattern.go:142 Multiline event flushed because timeout reached.
2021-12-16T11:02:51.481Z DEBUG [reader_multiline] multiline/pattern.go:142 Multiline event flushed because timeout reached.
2021-12-16T11:02:58.225Z DEBUG [reader_multiline] multiline/pattern.go:142 Multiline event flushed because timeout reached.
2021-12-16T11:03:04.555Z DEBUG cgroup/util.go:276 PID 1 contains a cgroups V2 path (0::/) but no V2 mountpoint was found.
This may be because metricbeat is running inside a container on a hybrid system.
To monitor cgroups V2 processess in this way, mount the unified (V2) hierarchy inside
the container as /sys/fs/cgroup/unified and start metricbeat with --system.hostfs.
Am I configuring something incorrectly?
P.S. disabling the multiline configuration and setting output codec to %message, I get the output as is as input. This ensures filebeat is able to read from kafka