Gz log files deleted by logstash when it crashed

Elastic Stack Logstash
1

logstash version: 6.4.1

I was sending gz logs to elasticsearch by logstash. The elasticsearch's disk was full but i leave it, so that the logstash reported many 403 errors.
But some day after that, the logstash crashed, and some of my gz logs was diappeared. The crash did not generate any hprof files or any other dumps.
I configed the gz logs path to the soft links of its parent folders.
I think, no matter how logstash crashed, it should not delete my raw logs.

Logstash last line log:
[logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>1000}

2

Finally, i found that file input plugin starting from v4.1.0 add a new config "file_completed_action" which default value is delete.
https://www.elastic.co/guide/en/logstash-versioned-plugins/current/v4.1.0-plugins-inputs-file.html#v4.1.0-plugins-inputs-file-file_completed_action
I think this MURDER my raw gz log data.
The dev team, do you know the raw gz log data is always priceless?
Do you know the delete operation always should be carefull?

3

There is an open issue for that.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.