Logstash file input not reparsing file

Elastic Stack Logstash
1

I have the following problem, I need logstash to reparse already parsed files:

Scenario that doesn't work but should:

  • upload file to watched folder

  • logstash processes it, saves to elastic, removes it (file_completed_action => "log_and_delete"), great

  • I upload the same file again, same name, same content.

  • logstash doesnt do anything, I want it to process it again

Here is my file input config:

file {
          mode => "read"
          exclude => "*.tif"
          path => ["/home/xmls/*.xml"]
          file_completed_action => "log_and_delete"
          file_completed_log_path => "/var/log/logstash/completed.log"
          sincedb_path => "/dev/null"
          start_position => "beginning"
          codec => multiline {
                  pattern => ".*"
                  what => "previous"
                  max_lines => 100000
                  max_bytes => "200 MiB"
          }
          type => "my-custom-type-1"
}

sincedb_path is set to /dev/null, it should not remember processed files, also tried setting ignore_older to 0, didn't help.

Also tried messing with queue settings in logstash.yml, changed it to persistent, didn't work ...

I'm using logstash version 7.5, logstash-input-file (4.1.11), running in linux machine.

When I restart logstash, then the unprocessed files get processed and cleaned up.
I need it to work without restarting.

2

No, setting sincedb_path to /dev/null prevents the file input persisting the sincedb across restarts. There is no way to prevent a file input remembering that it has processed a file.

Setting ignore_older to zero tells a file input to ignore any files more than zero seconds old. It is completely different to the effect it has in filebeat.

3

Thanks for the response,

so what I need is not possible with the file input plugin.
I'll try to dodge the problem somehow or try with some other input.

4

You can also try pipeline-to-pipeline operation. Or do all the parsing in the logstash filter. I am sure you can parse message as many times as you like

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.