Hi,
I upgraded from Logstash 2.2.2 to Logstash 5.0 and have this ERROR message outputted every 3-10 seconds. I did not encounter such errors with LS 2.2.2. I'm running on windows and my configuration has file input configurations to 472 distinct log files.
I ran with the --log.level debug flag too but I don't see anything that provides additional information about the error.
When I cut down the number of file inputs to 50 or 75, then this message goes away.
Any idea what this error signifies? What is the PeriodicPoller polling? I was under the impression that the files were being monitored by an OS level file watcher.
[2016-10-28T06:36:31,259][ERROR][logstash.instrument.periodicpoller.jvm] PeriodicPoller: exception {:poller=>#<LogStash::Instrument::PeriodicPoller::JVM:0x52c272c2 @task=#<Concurrent::TimerTask:0x16b80ef1 @observers=#<Concurrent::Collection::CopyOnNotifyObserverSet:0x43cdad59 @observers={#<LogStash::Instrument::PeriodicPoller::JVM:0x52c272c2 ...>=>:update}>, @timeout_interval=60.0, @running=#<Concurrent::AtomicBoolean:0x1550a7cd>, @StoppedEvent=#<Concurrent::Event:0x4349fc10 @set=false, @iteration=0>, @execution_interval=1.0, @do_nothing_on_deref=true, @run_now=nil, @freeze_on_deref=nil, @executor=#<Concurrent::SafeTaskExecutor:0x6dc03645 @task=#<Proc:0xd0adc3f@c:/java8/elk5/logstash-5.0.0/logstash-core/lib/logstash/instrument/periodic_poller/base.rb:52>, @exception_class=StandardError>, @StopEvent=#<Concurrent::Event:0x1d747c9c @set=false, @iteration=0>, @value=nil, @copy_on_deref=nil, @dup_on_deref=nil>, @peak_threads=274, @peak_open_fds=-1, @metric=#<LogStash::Instrument::Metric:0x11d1aa5e @collector=#<LogStash::Instrument::Collector:0x2254be97 @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x7dcfc43f @store=#<Concurrent::Map:0x267f049d @default_proc=nil>, @structured_lookup_mutex=#<Mutex:0x18510c52>, @fast_lookup=#<Concurrent::Map:0x44ec65c @default_proc=nil>>, @observer_state=true, @snapshot_task=#<Concurrent::TimerTask:0x7a314d6 @observers=#<Concurrent::Collection::CopyOnNotifyObserverSet:0x5926a353 @observers={#<LogStash::Instrument::Collector:0x2254be97 ...>=>:update}>, @timeout_interval=600.0, @running=#<Concurrent::AtomicBoolean:0x3b0b54e3>, @StoppedEvent=#<Concurrent::Event:0x2411e2f0 @set=false, @iteration=0>, @execution_interval=1.0, @do_nothing_on_deref=true, @run_now=nil, @freeze_on_deref=nil, @executor=#<Concurrent::SafeTaskExecutor:0x48c1f11a @task=#<Proc:0x29989f64@c:/java8/elk5/logstash-5.0.0/logstash-core/lib/logstash/instrument/collector.rb:87>, @exception_class=StandardError>, @StopEvent=#<Concurrent::Event:0x48d56af8 @set=false, @iteration=0>, @value=false, @copy_on_deref=nil, @dup_on_deref=nil>>>, @options={:polling_interval=>1, :polling_timeout=>60}>, :result=>nil, :exception=>#<Concurrent::TimeoutError: Concurrent::TimeoutError>, :executed_at=>2016-10-28 06:36:31 -0400}
The file input looks like
file {
path => "X:/development/myserver/user/server.log"
start_position => beginning
sincedb_path => "NUL"
ignore_older => 0
}
The log file is on a share mapped as as network drive but I recall seeing the same behavior when all the log files were on the local machine too.
One other notable difference I see between LS 2.2.2 and 5.0 is that changes to the log file was almost immediately picked up by logstash, but with LS 5.0 I've seen a change made to the log file picked up as late as 40 seconds after. Appears to be looping through the list of all the file inputs ad picks up the change only when it gets to it.
Thanks,
Sanjiv