I've made quite a few other posts, and have successfully ingested more than 1 billion messages in the last couple weeks, but we've come to the conclusion that Logstash just isn't keeping up with the backlog of messages. The reason I say this is that Logstash will seemingly lose its place in the ingestion, and start over, randomly.
What we have is NetApp CIFS audit logs turned on, and writing to a CIFS share on itself (not in the audit logs). There are 10,001 files in this directory - a single current file, and 10,000 previous files; each 100MB in size; or approximately 77,000 lines. I had the input configured to
/mnt/cifs/*.xml, and this was working, for the most part like I said above - hundreds of millions of messages have been ingested so far. But since it wasn't catching up, I opted to change the input to the single current file. I've tried different configurations of
start_position => "beginning",
sincedb_path => "/dev/null",
stat_interval => 1,
mode => read, and others, and it seems that Logstash will read the file when it starts, and only that one time, and never again, no matter what, until I restart Logstash.
I've read this page, but I'm not seeing where a specific problem is. Can someone break this down for me? Thank you in advance.