Kafka output dropping single message every time connection to kafka is lost

[tporeba (Tomasz Poręba)]

I tested the behaviour of filebeat kafka output when connection to kafka is lost and recovered. Events are correctly redelivered even if the connection is down for a few minutes, but I observe a single event (probably first one) to be lost for every connection loss. I'm not sure if I this is some misconfiguration from my side or a bug that should be reported to beats github.

For details (config and log) see my msg on slack Slack

[tporeba (Tomasz Poręba)]

I have a setup where I am watching the same log file with two independent log crawler instances each writing to separate kafka topic. For one of them I interrupted the connection by severing the ssh tunnel it is using, the other one was using a working connection all the time. Then I compared outputs of those and yes I could see that this single missing message was sucessfully passed via kafka to elasticsearch by the other crawler.

[tporeba (Tomasz Poręba)]

I simplified my config a little to produce simpler output on kafka by adding

codec.format:
string: '%{[@timestamp]} %{[message]}'

now on the input I used a simple script:

for ((i=0; i<=10000; i++)); do echo $i >> X.log; sleep 1; done

This is my input:

0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

and this is my output on kafka

2021-07-15T13:54:49.427Z 0
2021-07-15T13:54:50.428Z 1
2021-07-15T13:54:51.429Z 2
2021-07-15T13:54:52.429Z 3
2021-07-15T13:54:53.430Z 4
2021-07-15T13:54:54.430Z 5
2021-07-15T13:54:55.431Z 6
2021-07-15T13:54:56.431Z 7
2021-07-15T13:54:57.431Z 8
2021-07-15T13:54:58.432Z 9
2021-07-15T13:54:59.432Z 10
2021-07-15T13:55:00.433Z 11
2021-07-15T13:55:05.435Z 16
2021-07-15T13:55:02.434Z 13
2021-07-15T13:55:06.436Z 17
2021-07-15T13:55:07.436Z 18
2021-07-15T13:55:08.437Z 19
2021-07-15T13:55:09.437Z 20
2021-07-15T13:55:10.444Z 21
2021-07-15T13:55:11.456Z 22
2021-07-15T13:55:12.467Z 23
2021-07-15T13:55:13.478Z 24
2021-07-15T13:55:14.489Z 25
2021-07-15T13:55:15.500Z 26
2021-07-15T13:55:16.512Z 27
2021-07-15T13:55:17.523Z 28
2021-07-15T13:55:18.534Z 29
2021-07-15T13:55:19.545Z 30
2021-07-15T13:55:20.556Z 31
2021-07-15T13:55:21.567Z 32
2021-07-15T13:55:22.578Z 33
2021-07-15T13:55:23.590Z 34
2021-07-15T13:55:24.601Z 35
2021-07-15T13:55:25.612Z 36
2021-07-15T13:55:26.623Z 37
2021-07-15T13:55:27.635Z 38
2021-07-15T13:55:28.646Z 39
2021-07-15T13:55:29.657Z 40
2021-07-15T13:55:30.668Z 41
2021-07-15T13:55:31.679Z 42
2021-07-15T13:55:32.690Z 43
2021-07-15T13:55:33.701Z 44
2021-07-15T13:55:34.714Z 45
2021-07-15T13:55:35.724Z 46
2021-07-15T13:55:36.735Z 47
^CProcessed a total of 45 messages

3 messages are missing: 12, 14 and 15

the connection was interrupted ~13:55:00 and was back up ~13:55:09