Stopped Filebeat for 8 hours and old logs did not get pushed out

[ramzey1981 (rame)]

hello team,

so i have filebeat 6.0 and our application is logging to a single file. Last night i put filebeat in debug mode to troubleshoot an issue and then I killed the debug session and did not resume the filebeat process. This morning when i resumed filebeat i was hoping that all the old logs from our application would get pumped into logstash and I would see them in kibana but filebeat seems to have lost its pointer in the file and is now is only pushing new log lines into kibana. I tried replication the issue by stopping filebeat and generating new logs and then starting filebeats to see if I can reproduce but filebeat is behaving the way it shoudld be. Is this an issue with the length of time filebeat was down and then started hence the reason it generated a new pointer? Please let me know if this is a setting i need to adjust?

thanks

[ruflin]

As you are writing all logs to 1 file, I wonder if you ever truncate the file because it's getting too big? Or is it ever growing? What do you have set for your clean_* and close_* options? Please share your config.

[ramzey1981 (rame)]

hey Ruflin,

my filebeat config is as follows which reads yml files from conf.d folder,

filebeat:
config_dir: /etc/filebeat/conf.d
config.modules:
enabled: true
path: /etc/filebeat/modules.d/*.yml
output.redis:

Array of hosts to connect to.

hosts: ["logging.service.consul:6379"]
ssl.enabled: false

they yml file in question in conf.d is ,

https://pastebin.com/iiMpUdQg

in particular the firs log path

• /var/log/iam-service/rest-api/*

which has the following log files in its directory and they never get rotated since they grow in time. When the issue comes up there are only about 70 log lines in there or less
-rw-r--r--. 1 iam iam 0 Dec 10 23:37 iam_critical
-rw-r--r--. 1 iam iam 24311 Dec 14 07:13 iam_error
-rw-r--r--. 1 iam iam 1980911 Dec 14 22:17 iam_info

to solve this issue of the logs not making it once i deleted the registry file and restarted filebeat and all the logs came in. The second time the issue occurred i did the same thing a few times but the logs did not make it to kibana so i manually added a log entry and then all the logs came in at once and many duplicates since i deleted the registry fiel a few times to debug. In terms of your questions for clean_* and close_* I have all the defaults that are set with filebeat 6.0 . I also checked to see if elasticsearch was dropping the logs or logstash and I dont see any errors. I also believe that they are not dropping the logs becasue the logs are eventually making it to kibana.

thanks

[ruflin]

Can you share some details on how the log files are written?

Please also share the filebeat log file with debug enabled. There should be quite a bit of additional info inside which should help us solve this issue.

Note: filebeat.config_dir is deprecated and filbeat.config.prospector should be used: https://github.com/elastic/beats/pull/5321