We use Azure Kubernetes Service, and our HTTP logs are written by the loadbalancer (ingress) to shared Azure storage. I'd like to read these logs and import them to Elasticsearch.
I realise reading from shared storage isn't supported, but I can't change the architecture of what we already have running and would like to get this to work if possible. The log files are structured in directories as "year/month/day/service" are never moved, renamed or truncated, but are deleted once their retention period has passed.
So from reading the filebeat docs, I should be able to set "file_identity" to "path" as follows:
However I'm still getting the error File was truncated. Begin reading file from offset 0
This is with Filebeat 7.13.4: Build info {"system_info": {"build": {"commit": "1907c246c8b0d23ae4027699c44bf3fbef57f4a4", "libbeat": "7.13.4", "time": "2021-07-14T18:42:41.000Z", "version": "7.13.4"}}}
Any suggestions? Should this work like I'm hoping?!
Hi no, we don't move, rotate or rename any log files. They are in a directory structure of "/year/month/date/service" e.g. "/2021/07/29/foo.log" and then each date directory is just deleted after 60days.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.