The log file I'm parsing are constantly writing new logs. My logstash works perfectly for other logs, except that everyday morning at around 7am, it will have about 5 consequent messages that only contain header of the log files
They look like this.
{
"_index": "logstash-2016.08.31",
"_type": "ValidationErrors",
"_id": "AVbfcfTL6JZG4fsBrC2f",
"_score": null,
"_source": {
"@timestamp": "2016-08-31T07:13:21.388Z",
"message": "########################################\r",
"@version": "1",
"path": "//path/ValidationErrors.log",
"host": "SF-machine",
"type": "ValidationErrors",
"tags": [
"_grokparsefailure"
]
"_index": "logstash-2016.08.31",
"_type": "ValidationErrors",
"_id": "AVbfcZEj6JZG4fsBrC1c",
"_score": null,
"_source": {
"@timestamp": "2016-08-31T07:12:56.147Z",
"message": "########################################\r",
"@version": "1",
"path": "//path/ValidationErrors.log",
"host": "SF-machine",
"type": "ValidationErrors",
"tags": [
"_grokparsefailure"
]
Among the five failure messages, the time interval is about 20s-2mins.
My code looks like this:
file {
path => "//path/ValidationErrors.log"
start_position => beginning
ignore_older => 0
type => "ValidationErrors"
codec => multiline {
pattern => "^#########"
negate => true
what => "previous"
}
close_older => 300
}
When I apply the code to other log files, it works fine. I have also checked this target log file, but couldn't find any singe line of header and the program shouldn't delete anything.
My guess is that when they system writes logs to the file, it writes the header first, and before they can write the message body, logstash detects the file change and upload the changes. Is there anything we can do to make logstash wait for sometime everytime it detectects new changes?