File input plugin does not read files

[Nathan_Sowatskey]

I have a file input plugin configured as below. The first time I ran Logstash with this configuration, I think it read the files in question and created an index. Subsequently, it definitely has not.

A similar configuration without the file_completed_action definitely did work, but it also deletes the files after reading them, by default. That is not what I wanted, so hence the need to add an explicit file_completed_action as log, and so also the file_completed_log_path.

I suspect that the problem lies in the logic of not reading files that have already been read. Note that I have file_completed_log_path configured below. That file is empty though.

The behaviour here is as though the plugin "knows" that it has read the files in question. I can't, though, see which configuration/log file, where, has the list of files that this plugin thinks it has read already. It does not appear to be the file configured as file_completed_log_path, as that file is empty.

I have tried touching the files in question, to see if a date change would trigger a read, but that did not seem to make any difference either.

I know that the files are not being read as there is no index corresponding to those files.

Any ideas please?

Many thanks

Nathan

input {
   file {
      path => ["/.../docker_inspect_k8s*.log"]
      type => "rca_docker_inspect"
      codec => "line"
      mode => "read"
      file_completed_action => "log"
      file_completed_log_path => "logged_file.list"
   }

...```

[Badger]

@Nathan_Sowatskey

The behaviour here is as though the plugin "knows" that it has read the files in question.

Yes, it does. It saves state in the "sincedb". This is maintained in-memory and peristed to sincedb_path. If you want the state reset every time you restart logstash then set

sincedb_path => "/dev/null"

[Nathan_Sowatskey]

@Badger

sincedb_path

First of all, thank you, and apologies for the delay in following up. I think that this suggestion did make a difference, but I am still seeing unexpected behaviour.

I now have:

input {
   file {
   	id => "rca_docker_inspect"
        path => ["/.../docker_inspect_k8s*.log"]
      	type => "rca_docker_inspect"
      	codec => "line"
      	mode => "read"
  	file_completed_action => "log"
	file_completed_log_path => "logged_file.list"
	sincedb_path => "/dev/null"
   }

...

output {
     elasticsearch {
            hosts => [ "localhost:9200" ]
            index => "rca_docker_inspect-%{+YYYY.MM.dd}"
        }
...

BUT, the index only appears in ES after I stop Logstash with a Ctrl-C. When Logstash is running, I don't see the index at: http://localhost:9200/_aliases?pretty=true

So, I now have something that is repeatable, but weird.

Regards

Nathan

[Badger]

That would suggest that the codec is never seeing a newline to tell it to flush an event, so it only flushes on shutdown.

[Nathan_Sowatskey]

Yes, the mode is 'read' though, so it should just read the files and get on with it. What I am seeing does not make sense. I am still missing something I suspect.

I am also a bit baffled that it is so tricky to just read some files. This seems like a core use case for Logstash/ES, but it is proving to be much harder than I expected.

Regards

Nathan

[Nathan_Sowatskey]

OK, though I can see that many files are being processed, within which there are many loge entries, so there must be a new line in somewhere ...