We are having issues while running filebeat as a deamon set.
File beat version : 5.6.12
We are running file beat as a deamonset across our different kubernates clusters.
We are running deamonset in its own name sapce (beat namespace)
What we started seeing lately is when ever log files gets rotated under path (we rotated logs every 10MB).
Logs files under path (/var/lib/docker/containers/container_id/container_id.log) rotated when the file reaches 10mb. filebeat detecting the rotation, picking up the new file, but fails to update the offset in the registry file. new file must have offset 0, which filebeat sets in the registry but it doesn’t read file properly (not sure why not). here’s part of registry file regarding onbe pod logs
I don't see anything in the logs. Can you also share your filebeat configs?
If a app writes too much of a log, can we determine if we can keep up with existing filebeat.
Not sure, you are using a quite old filebeat version. In 6.5 we're adding a metric for reporting the number of open file handles. This is often an indicator of filebeat being 'behind', as filebeat keeps rotated files open. In the future we will try to add metrics of actual file size vs. read offsets, so one can tell how many bytes filebeat is behind.
Any configs to improve the the filebeat throughput.
Throughput does not only depend on filebeat, but even more so on the data sink and intermediate services processing events from filebeat. From experience, filebeat is rarely the bottleneck. But still, it must be tuned in order to 'satisfy' the event consumers.
Some recent blog post on Beats -> Elasticsearch tuning, which makes sense to tune after tuning Elasticsearch.
2018/10/24 23:02:53.743802 prospector_log.go:324: INFO File is falling under ignore_older before harvesting is finished. Adjust your close_* settings: /var/log/containers/poc-****-2_default_****-****-71e86e5b4bd6915cb16ee846c369737b99e6c94c20ff343280dff5cde998f26c.log
2018/10/24 23:02:55.744999 prospector_log.go:324: INFO File is falling under ignore_older before harvesting is finished. Adjust your close_* settings: /var/log/containers/poc-****-2_default_****-****-71e86e5b4bd6915cb16ee846c369737b99e6c94c20ff343280dff5cde998f26c.log
2018/10/24 23:02:57.746367 prospector_log.go:324: INFO File is falling under ignore_older before harvesting is finished. Adjust your close_* settings: /var/log/containers/poc-****-2_default_****-****-71e86e5b4bd6915cb16ee846c369737b99e6c94c20ff343280dff5cde998f26c.log
2018/10/24 23:02:59.747627 prospector_log.go:324: INFO File is falling under ignore_older before harvesting is finished. Adjust your close_* settings: /var/log/containers/poc-****-2_default_****-****-71e86e5b4bd6915cb16ee846c369737b99e6c94c20ff343280dff5cde998f26c.log
2018/10/24 23:03:01.749090 prospector_log.go:324: INFO File is falling under ignore_older before harvesting is finished. Adjust your close_* settings: /var/log/containers/poc-****-2_default_****-****-71e86e5b4bd6915cb16ee846c369737b99e6c94c20ff343280dff5cde998f26c.log
2018/10/24 23:03:02.589197 metrics.go:39: INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=3 libbeat.logstash.publish.read_bytes=315 libbeat.logstash.publish.write_bytes=49082 libbeat.logstash.published_and_acked_events=384 libbeat.publisher.published_events=384 publish.events=384 registrar.states.update=384 registrar.writes=3
Does that mean we are falling behind and kubernates keeps rotating logs
OK, this is a clear sign filebeat lagging behind. The offsets increase in your logs, because the old files are still open. The old logs are treated by filebeat as separate files, that's what the inode is for.
Switching to a more recent filebeat (event serialization + async publishing 6.x releases helps a little) + properly tuning the queue size and output can also help (your output settings are not as effective as you might think, due to the spool settings in 5.6 limiting the total number of events being forwared to the outputs.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.