Hi,
thanks for the answer. Checking timestamps of last change of the files in S3, I see there are new logs popping up sometimes every few seconds, sometimes it might be up to a minutes.
So, I think not that many that filebeat should be overwhelmed.
As I understand, it gets a message from the SQS queue, that a new file is in S3, then fetches the file, and then ingests it. Therefore, I think the message receive and fetch works, it's just that the parsing of the log by the module is having a problem?
Therefore I don't believe if further bumping these timeouts will help.
Actually looking at todays logs, I see a number of these
Half of the set visibilityTimeout passed, visibility timeout needs to be updated
Message visibility timeout updated to 500 seconds
but only one of these
readStringAndTrimDelimiter
in the logs. So don't know if this may even be a red herring?
I did some other test, downloading a log file, and ingested it manually into a test cluster instance, just using file input. That worked fine, number of lines in the file was the same as the number of records that ended up in elasticsearch, etc. all fields parsed properly etc.
Also what I observed while looking at it, it doesn't seem to affect all files, and I found a file, where there was 1 record more in Elasticsearch than lines in the file. As it seems from timestamps, there was a last entry with an empty message. No idea where this comes from.
Sebastian