As the author of the file input changes, I am interested in getting to the bottom of why read mode looks broken after some time.
While developing the code, I did not run an instance for many days.
Please make sure you are using version 4.1.6 of the file input.
Filebeat does not yet have a read mode - we are waiting to iron out any kinks first in LS.
This may be hard to recreate.
At the time it "hangs", are the existing files fully read? I'm guessing you detect this situation when a file is dumped in the folder and it does not appear as a dashboard.
When this happens, you can turn on trace logging via the rest API.
First do curl -XGET 'localhost:9600/_node/logging?pretty'
You see something like this:
{
"host" : "Elastics-MacBook-Pro.local",
"version" : "6.4.0",
"http_address" : "127.0.0.1:9600",
"id" : "8789409b-7126-4034-9347-de47e6ce12a9",
"name" : "Elastics-MacBook-Pro.local",
"loggers" : {
"filewatch.discoverer" : "DEBUG",
"filewatch.observingtail" : "DEBUG",
"filewatch.sincedbcollection" : "DEBUG",
"filewatch.tailmode.handlers.createinitial" : "DEBUG",
"filewatch.tailmode.processor" : "DEBUG",
"logstash.agent" : "DEBUG",
"logstash.api.service" : "DEBUG",
"logstash.codecs.json" : "DEBUG",
...
"logstash.filters.date" : "DEBUG",
"logstash.inputs.file" : "DEBUG",
...
"logstash.outputs.stdout" : "DEBUG",
"logstash.pipeline" : "DEBUG",
...
"slowlog.logstash.codecs.json" : "TRACE",
"slowlog.logstash.codecs.rubydebug" : "TRACE",
"slowlog.logstash.filters.date" : "TRACE",
"slowlog.logstash.inputs.file" : "TRACE",
"slowlog.logstash.outputs.stdout" : "TRACE"
}
}
The ones you need to change are prefixed with filewatch
curl -XPUT 'localhost:9600/_node/logging?pretty' -H 'Content-Type: application/json' -d'
{
"logger.filewatch.discoverer" : "TRACE"
}
'
Now you will get more information on what the discoverer is doing.
e.g.
[...][TRACE][filewatch.discoverer ] discover_files handling: {"new discovery"=>false, "watched_file details"=>"<FileWatch::WatchedFile: @filename='testing-b.txt', @state='active', @recent_states='[:watched, :watched]', @bytes_read='11', @bytes_unread='0', current_size='11', last_stat_size='11', file_open?='true', @initial=false, @sincedb_key='10211069 1 4'>"}
[...][TRACE][filewatch.discoverer ] discover_files handling: {"new discovery"=>false, "watched_file details"=>"<FileWatch::WatchedFile: @filename='testing-a.txt', @state='active', @recent_states='[:watched, :watched]', @bytes_read='22', @bytes_unread='0', current_size='22', last_stat_size='22', file_open?='true', @initial=false, @sincedb_key='10211020 1 4'>"}
(The above is tail mode) The above tells me that it "knows" about two files, they are not new, they are actively being monitored for new content, they are fully read, the file handle is open etc.
A fresh new dropped file should appear as new discovery"=>true, @state='active' tells me that the file is in the Active state.
You can set the logging back to INFO for discoverer. Now move on to filewatch.readmode.processor, set its log level to TRACE.
You should see the processor moving though the Watched and Active files. Files that are "done" are neither Watched nor Active. A file is moved from the Watched state to the Active state when there is a slot in the max_open_files window - for total file count of < 4095 a new file should go to Active pretty quickly and then the filewatch.readmode.handlers.read_file takes over.
Do you, by any chance, see 4095 files in your completed log when the "hang" occurs?