File input plugin does not read files

Elastic Stack Logstash
1

I have a file input plugin configured as below. The first time I ran Logstash with this configuration, I think it read the files in question and created an index. Subsequently, it definitely has not.

A similar configuration without the file_completed_action definitely did work, but it also deletes the files after reading them, by default. That is not what I wanted, so hence the need to add an explicit file_completed_action as log, and so also the file_completed_log_path.

I suspect that the problem lies in the logic of not reading files that have already been read. Note that I have file_completed_log_path configured below. That file is empty though.

The behaviour here is as though the plugin "knows" that it has read the files in question. I can't, though, see which configuration/log file, where, has the list of files that this plugin thinks it has read already. It does not appear to be the file configured as file_completed_log_path, as that file is empty.

I have tried touching the files in question, to see if a date change would trigger a read, but that did not seem to make any difference either.

I know that the files are not being read as there is no index corresponding to those files.

Any ideas please?

Many thanks

Nathan

input {
   file {
      path => ["/.../docker_inspect_k8s*.log"]
      type => "rca_docker_inspect"
      codec => "line"
      mode => "read"
      file_completed_action => "log"
      file_completed_log_path => "logged_file.list"
   }

...```
2

Yes, it does. It saves state in the "sincedb". This is maintained in-memory and peristed to sincedb_path. If you want the state reset every time you restart logstash then set

sincedb_path => "/dev/null"
3

First of all, thank you, and apologies for the delay in following up. I think that this suggestion did make a difference, but I am still seeing unexpected behaviour.

I now have:

input {
   file {
   	id => "rca_docker_inspect"
        path => ["/.../docker_inspect_k8s*.log"]
      	type => "rca_docker_inspect"
      	codec => "line"
      	mode => "read"
  	file_completed_action => "log"
	file_completed_log_path => "logged_file.list"
	sincedb_path => "/dev/null"
   }

...

output {
     elasticsearch {
            hosts => [ "localhost:9200" ]
            index => "rca_docker_inspect-%{+YYYY.MM.dd}"
        }
...

BUT, the index only appears in ES after I stop Logstash with a Ctrl-C. When Logstash is running, I don't see the index at: http://localhost:9200/_aliases?pretty=true

So, I now have something that is repeatable, but weird.

Regards

Nathan

4

That would suggest that the codec is never seeing a newline to tell it to flush an event, so it only flushes on shutdown.

5

Yes, the mode is 'read' though, so it should just read the files and get on with it. What I am seeing does not make sense. I am still missing something I suspect.

I am also a bit baffled that it is so tricky to just read some files. This seems like a core use case for Logstash/ES, but it is proving to be much harder than I expected.

Regards

Nathan

6

OK, though I can see that many files are being processed, within which there are many loge entries, so there must be a new line in somewhere ...

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.