Windows 10 File input hangs [SOLVED]

Elastic Stack Logstash
1

Running in virtual test environment on Windows 10. Looked at all sorts of topics on this only to find all solutions (often just how to type the file path) are coming up unsuccessful. Behavior is Logstash hangs on "Logstash startup complete" and fails to read files. Stdin test does just fine. Tested with wildcards and without, and made sure Logstash ran on even admin permissions to read/write files/directories.

Current config:

input {
	file {
		path => ["C:/ELK/LogSamples/00605828/clientA.log"]
		sincedb_path => "C:/ELK/logstash-1.5.4/sincedb"
		codec => multiline {
			pattern => "^\s"
			what => "previous"
		}
	}
}

#filter {
#	grok {
#		match => { "message" => "%{LOGLEVEL:loglevel}: %{MONTHNUM2:month}/%{MONTHNUM2:day} %{TIME:timestamp}(?:\s\d\d|\s\s\d)\s%{DATA:event}" }
#		}
#	}

output {
	elasticsearch {
		}
	file {
		path => "c:/elk/logstashtest.txt"
		}
}

Output is as follows:

C:\ELK\logstash-1.5.4\bin>logstash -f C:\ELK\logstash-1.5.4\logstash.conf
io/console not supported; tty will not be manipulated
Oct 01, 2015 2:48:17 AM org.elasticsearch.node.internal.InternalNode <init>
INFO: [logstash-DESKTOP-2QF2QGE-2804-13462] version[1.7.0], pid[2804], build[929b973/2015-07-16T14:31:07Z]
Oct 01, 2015 2:48:17 AM org.elasticsearch.node.internal.InternalNode <init>
INFO: [logstash-DESKTOP-2QF2QGE-2804-13462] initializing ...
Oct 01, 2015 2:48:17 AM org.elasticsearch.plugins.PluginsService <init>
INFO: [logstash-DESKTOP-2QF2QGE-2804-13462] loaded [], sites []
Oct 01, 2015 2:48:19 AM org.elasticsearch.bootstrap.Natives <clinit>
WARNING: JNA not found. native methods will be disabled.
Oct 01, 2015 2:48:20 AM org.elasticsearch.node.internal.InternalNode <init>
INFO: [logstash-DESKTOP-2QF2QGE-2804-13462] initialized
Oct 01, 2015 2:48:20 AM org.elasticsearch.node.internal.InternalNode start
INFO: [logstash-DESKTOP-2QF2QGE-2804-13462] starting ...
Oct 01, 2015 2:48:20 AM org.elasticsearch.transport.TransportService doStart
INFO: [logstash-DESKTOP-2QF2QGE-2804-13462] bound_address {inet[/0:0:0:0:0:0:0:0:9301]}, publish_address {inet[/10.0.2.15:9301]}
Oct 01, 2015 2:48:20 AM org.elasticsearch.discovery.DiscoveryService doStart
INFO: [logstash-DESKTOP-2QF2QGE-2804-13462] elasticsearch/X6Vjq2FcSNKIhmiPNvFU5w
Oct 01, 2015 2:48:23 AM org.elasticsearch.cluster.service.InternalClusterService$UpdateTask run
INFO: [logstash-DESKTOP-2QF2QGE-2804-13462] detected_master [Yukon Jack][FNrfZ1S4TvOFTV7d83L2fA][DESKTOP-2QF2QGE][inet[/10.0.2.15:9300]], added {[Yukon Jack][FNrfZ1S4TvOFTV7d83L2fA][DESKTOP-2QF2QGE][inet[/10.0.2.15:9300]],}, reason: zen-disco-receive(from master [[Yukon Jack][FNrfZ1S4TvOFTV7d83L2fA][DESKTOP-2QF2QGE][inet[/10.0.2.15:9300]]])
Oct 01, 2015 2:48:23 AM org.elasticsearch.node.internal.InternalNode start
INFO: [logstash-DESKTOP-2QF2QGE-2804-13462] started
Logstash startup completed

Preemptive thanks on my part!

2

Has new data been appended to C:/ELK/LogSamples/00605828/clientA.log since you started Logstash?

3

Hi Magnus, thank you for the help.

It's seen no changes, but I have no sourcedb or any related files present that I know of that would show Logstash keeping a record of looking at that file. The one I assigned in the sourcedb_path is not being created.

I have also changed the path to direct to other files that Logstash has not seen in order to test this, to no success.

4

By default Logstash's file input tails files and does not read them from the beginning. If you want something else you'll have to look into the start_position parameter.

5

That is my intended design, but no matter what I push to Logstash, even brand new files, the file input plugin does not want to touch them.

I did test with start_position => beginning but no change in behavior.

Through my research on this I discovered there are some issues with how the file plugin works that does not like Windows, but I'm not sure what alternatives to use to read such text files. Even so, some people do seem to have no problems on Windows with the file plugin.

6

Cranking up Logstash's log level with --verbose or --debug may reveal interesting information.

7

As I thought, it is stuck trying to discover the file:

e[36m_discover_file_glob: C:/ELK/LogSamples/00605828/clientA.log: glob is: ["C:/ELK/LogSamples/00605828/clientA.log"] {:level=>:debug, :file=>"/ELK/logstash-1.5.4/vendor/bundle/jruby/1.9/gems/filewatch-0.6.5/lib/filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}e[0m

It keeps repeating that event.

8

Well, that's what it's supposed to be doing every ten seconds. The question is, why isn't it proceeding and trying to open and read the file? Is the periodic discover_file_glob all you get even with --debug?

9

Found the problem, debug output showed me that it was in fact reading a sincedb file. Then it dawned on me: silly me, I forgot to turn on show hidden files in the test environment! Deleting it did trigger it to read properly. Thank you for your patience in helping me with this!