Hi @TiagoQueiroz,
unfortunately I was not able to reproduce with debug enabled in a test environment up to now.
This morning I see this in our production where debug is not enabled:
Filebeat is running since 01 June:
-bash-4.2$ ls -alh log
-rw------- 1 graylog graylog 11M Jun 6 05:25 filebeat-20230601-1.ndjson
-rw------- 1 graylog graylog 5.0K Jun 1 09:56 filebeat-20230601.ndjson
-rw------- 1 graylog graylog 389K Jun 6 09:45 filebeat-20230606.ndjson
-bash-4.2$ ps -ef | grep filebeat
graylog 81071 2279 1 Jun01 ? 02:06:16 /usr/share/filebeat/bin/filebeat -c /var/lib/graylog-sidecar/generated/filebeat.conf --path.home /usr/share/filebeat
Filebeat collects it's own logfiles:
-bash-4.2$ cat filebeat-20230606.ndjson | jq -r '. | "\(."@timestamp") \(."log.level") \(.message)"'
2023-06-06T08:15:23.530+0200 info File was truncated. Reading file from offset 0. Path=/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson
2023-06-06T08:15:38.066+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:15:39.093+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:19:03.913+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:20:06.006+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:20:06.006+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:20:08.012+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:20:08.012+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:20:08.012+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:20:08.012+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:20:08.012+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:20:08.012+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:20:08.012+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:25:19.589+0200 info Reader was closed. Closing.
2023-06-06T08:25:23.530+0200 info File was truncated. Reading file from offset 0. Path=/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson
2023-06-06T08:25:35.055+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:25:38.044+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:25:38.051+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:25:41.075+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:25:48.953+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:25:51.075+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:25:58.045+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:26:05.068+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:26:09.041+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:28:06.758+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:30:04.505+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:30:06.969+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:30:06.969+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:30:06.969+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:30:06.969+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:30:06.969+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:30:06.969+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:30:08.972+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:30:08.972+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:30:08.972+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:35:13.005+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:35:13.628+0200 info Reader was closed. Closing.
2023-06-06T08:35:13.806+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:35:13.990+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:35:13.991+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:35:15.010+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:35:15.010+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:35:15.010+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:35:16.804+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:35:23.530+0200 info File was truncated. Reading file from offset 0. Path=/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson
2023-06-06T08:35:38.005+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:35:38.999+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:36:07.545+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:37:04.144+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:38:59.009+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:39:01.813+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:40:05.931+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:40:05.931+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:40:05.931+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:40:05.932+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:40:05.932+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:40:07.935+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:40:07.935+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:40:07.935+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:40:07.935+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:40:07.935+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:45:12.856+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:45:12.981+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:45:13.609+0200 info Reader was closed. Closing.
2023-06-06T08:45:13.983+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:45:15.973+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:45:15.973+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:45:17.610+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:45:18.976+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:45:19.988+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:45:23.530+0200 info File was truncated. Reading file from offset 0. Path=/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson
2023-06-06T08:45:38.004+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:45:45.974+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:45:58.544+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:48:04.758+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:48:59.016+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:49:14.994+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:50:06.947+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:50:06.947+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:50:08.962+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:50:08.962+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:50:08.962+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:50:08.962+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:50:08.962+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:50:08.962+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:50:08.962+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:50:08.962+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:55:13.622+0200 info Reader was closed. Closing.
2023-06-06T08:55:13.992+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:55:14.827+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:55:14.994+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:55:17.006+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:55:17.068+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:55:17.993+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:55:17.993+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:55:18.610+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:55:19.985+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:55:23.531+0200 info File was truncated. Reading file from offset 0. Path=/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson
2023-06-06T08:55:41.010+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:55:41.010+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:56:43.596+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:58:03.733+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T08:59:02.023+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T08:59:32.025+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:00:07.949+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:00:07.962+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:00:07.962+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:00:07.962+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:00:07.962+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:00:07.962+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:00:07.962+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:00:07.969+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:00:09.966+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:00:09.966+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:00:09.966+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:05:13.994+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:05:14.773+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:05:15.000+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:05:15.000+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:05:15.000+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:05:15.000+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:05:15.000+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:05:16.012+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:05:16.012+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:05:21.628+0200 info Reader was closed. Closing.
2023-06-06T09:05:23.530+0200 info File was truncated. Reading file from offset 0. Path=/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson
2023-06-06T09:05:30.522+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:05:40.020+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:07:24.689+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:07:53.659+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:08:04.753+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:10:06.972+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:10:06.972+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:10:06.972+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:10:06.973+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:10:06.973+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:10:06.973+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:10:06.973+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:10:06.974+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:15:13.017+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:15:13.029+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:15:14.732+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:15:14.793+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:15:15.613+0200 info Reader was closed. Closing.
2023-06-06T09:15:16.022+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:15:16.022+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:15:19.010+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:15:23.530+0200 info File was truncated. Reading file from offset 0. Path=/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson
2023-06-06T09:15:35.056+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:15:38.013+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:16:07.010+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:18:05.751+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:18:27.787+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:18:59.099+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:19:05.594+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:20:07.959+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:20:07.959+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:20:07.959+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:20:07.960+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:20:07.960+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:20:07.960+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:20:07.960+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:20:07.960+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:20:07.960+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:20:07.961+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:25:13.009+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:25:13.988+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:25:13.988+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:25:15.613+0200 info Reader was closed. Closing.
2023-06-06T09:25:15.782+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:25:16.995+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:25:17.833+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:25:23.531+0200 info File was truncated. Reading file from offset 0. Path=/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson
2023-06-06T09:25:37.998+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:25:39.009+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:26:00.562+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:26:08.010+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:26:09.009+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:28:04.759+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:28:58.040+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:29:09.041+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:30:08.954+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:30:08.954+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:30:08.954+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:30:08.954+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:30:08.954+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:30:08.954+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:30:08.954+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:30:08.954+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:30:08.954+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:30:08.954+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:35:14.990+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:35:14.993+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:35:14.993+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:35:15.792+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:35:16.983+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:35:16.987+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:35:16.987+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:35:16.987+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:35:20.615+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:35:39.470+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:35:48.980+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:35:53.496+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:36:05.519+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:38:03.708+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:38:50.007+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:39:01.818+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:39:03.822+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:40:07.925+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:40:07.925+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:40:07.926+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:40:07.926+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:40:07.926+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:45:12.953+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:45:12.953+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:45:13.668+0200 info Reader was closed. Closing.
2023-06-06T09:45:13.748+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:45:13.855+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:45:15.948+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:45:18.554+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:45:19.515+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:45:21.872+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:45:23.532+0200 info File was truncated. Reading file from offset 0. Path=/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson
2023-06-06T09:45:37.959+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:45:40.084+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:48:04.694+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:48:47.973+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:48:50.605+0200 info File is inactive. Closing because close_inactive of 5m0s reached.
2023-06-06T09:50:06.883+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:50:06.883+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:50:06.883+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:50:06.884+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:50:06.884+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:50:06.884+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:50:06.884+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:50:06.884+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:50:06.884+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:50:08.887+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
2023-06-06T09:50:08.888+0200 info Harvester started for paths: [/db2/*/sqllib/db2dump/db2diag.log]
Here are the "truncated"-messages filtered from the log:
-bash-4.2$ cat filebeat-20230606.ndjson | jq -r '. | "\(."@timestamp") \(."log.level") \(.message)"' | grep trunc
2023-06-06T05:25:53.530+0200 info File was truncated. Reading file from offset 0. Path=/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson
2023-06-06T05:45:23.530+0200 info File was truncated. Reading file from offset 0. Path=/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson
2023-06-06T06:15:23.530+0200 info File was truncated. Reading file from offset 0. Path=/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson
2023-06-06T06:25:23.530+0200 info File was truncated. Reading file from offset 0. Path=/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson
2023-06-06T06:35:23.530+0200 info File was truncated. Reading file from offset 0. Path=/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson
2023-06-06T06:45:23.531+0200 info File was truncated. Reading file from offset 0. Path=/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson
2023-06-06T06:55:23.530+0200 info File was truncated. Reading file from offset 0. Path=/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson
2023-06-06T07:05:23.530+0200 info File was truncated. Reading file from offset 0. Path=/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson
2023-06-06T07:15:23.530+0200 info File was truncated. Reading file from offset 0. Path=/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson
2023-06-06T07:25:23.530+0200 info File was truncated. Reading file from offset 0. Path=/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson
2023-06-06T07:55:23.530+0200 info File was truncated. Reading file from offset 0. Path=/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson
2023-06-06T08:05:23.530+0200 info File was truncated. Reading file from offset 0. Path=/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson
2023-06-06T08:15:23.530+0200 info File was truncated. Reading file from offset 0. Path=/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson
2023-06-06T08:25:23.530+0200 info File was truncated. Reading file from offset 0. Path=/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson
2023-06-06T08:35:23.530+0200 info File was truncated. Reading file from offset 0. Path=/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson
2023-06-06T08:45:23.530+0200 info File was truncated. Reading file from offset 0. Path=/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson
2023-06-06T08:55:23.531+0200 info File was truncated. Reading file from offset 0. Path=/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson
2023-06-06T09:05:23.530+0200 info File was truncated. Reading file from offset 0. Path=/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson
2023-06-06T09:15:23.530+0200 info File was truncated. Reading file from offset 0. Path=/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson
2023-06-06T09:25:23.531+0200 info File was truncated. Reading file from offset 0. Path=/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson
2023-06-06T09:45:23.532+0200 info File was truncated. Reading file from offset 0. Path=/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson
Remarkable is the pattern of mostley 10 minutes between each entry.
Here are the attributes of the logfile:
-bash-4.2$ stat /var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson
File: ‘/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson’
Size: 406492 Blocks: 896 IO Block: 4096 regular file
Device: fd28h/64808d Inode: 71 Links: 1
Access: (0600/-rw-------) Uid: ( 389/ graylog) Gid: ( 385/ graylog)
Access: 2023-06-06 09:50:09.595512897 +0200
Modify: 2023-06-06 09:50:08.886518646 +0200
Change: 2023-06-06 09:50:08.886518646 +0200
Birth: -
And this is the registry:
-bash-4.2$ ls -alh ../data/registry/filebeat/
total 8.3M
drwxr-x--- 2 graylog graylog 158 Jun 6 09:50 .
drwxr-x--- 3 graylog graylog 22 Apr 26 17:26 ..
-rw------- 1 graylog graylog 172K Jun 6 09:50 37999763.json
-rw------- 1 graylog graylog 163K Apr 26 17:26 572.json-1682522782670094302.bak
-rw------- 1 graylog graylog 81 Jun 6 09:50 active.dat
-rw------- 1 graylog graylog 7.3M Jun 6 09:56 log.json
-rw------- 1 graylog graylog 0 Apr 26 17:26 log.json-1682522782670294442.bak
-rw------- 1 graylog graylog 16 Apr 26 17:26 meta.json
The content of the registry:
-bash-4.2$ grep -r /var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson .
./log.json:{"k":"filestream::filebeat::native::71-64808","v":{"meta":{"source":"/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson","identifier_name":"native"},"ttl":1800000000000,"updated":[515926818243,1686038123],"cursor":{"offset":5585756}}}
./log.json:{"k":"filestream::filebeat::native::71-64808","v":{"ttl":1800000000000,"updated":[515947789710,1686038123],"cursor":{"offset":5995709},"meta":{"identifier_name":"native","source":"/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson"}}}
./log.json:{"k":"filestream::filebeat::native::71-64808","v":{"cursor":{"offset":5996334},"meta":{"source":"/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson","identifier_name":"native"},"ttl":1800000000000,"updated":[515950719029,1686038137]}}
./log.json:{"k":"filestream::filebeat::native::71-64808","v":{"cursor":{"offset":5996962},"meta":{"source":"/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson","identifier_name":"native"},"ttl":1800000000000,"updated":[515951807862,1686038139]}}
./log.json:{"k":"filestream::filebeat::native::71-64808","v":{"cursor":{"offset":5997589},"meta":{"source":"/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson","identifier_name":"native"},"ttl":1800000000000,"updated":[515953167899,1686038145]}}
./log.json:{"k":"filestream::filebeat::native::71-64808","v":{"ttl":1800000000000,"updated":[515955793284,1686038169],"cursor":{"offset":5998214},"meta":{"source":"/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson","identifier_name":"native"}}}
./log.json:{"k":"filestream::filebeat::native::71-64808","v":{"ttl":1800000000000,"updated":[515966328886,1686038293],"cursor":{"offset":5998839},"meta":{"source":"/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson","identifier_name":"native"}}}
./37999763.json:{"_key":"filestream::filebeat::native::71-64808","ttl":1800000000000,"updated":[515993614898,1686037809],"cursor":{"offset":5583874},"meta":{"identifier_name":"native","source":"/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat-20230606.ndjson"}},
It seems suspiciuous for me that value of the cursor is much greater than the size of the file.
My observations:
- The issue occures more often but sporadic on some of the 1.800 nodes.
- The affected logfiles on this nodes changes.
- If the filebeat logs a collected logfile as truncated, he will do this cyclic in a 10 minute intervall.
I'll keep trying to reproduce in a test environment whith debugging enabled.