Logstash File Input Plugin Read Mode Not Loading Data

[Kenneth_M_Kolano]

I'm having trouble getting Logstash to load data to Elasticsearch when using read mode.

input {
	file {
		mode => "read"
		path => ["/var/log/PASS/analyses/*"]
		sincedb_path => ["/var/lib/logstash/file-analysis_sincedb"]
		codec => json
		type => "File-Analysis"
	}
}
output {
	if [type] == "file-analysis" {
		elasticsearch {
			hosts => localhost
			index => "file-analysis"
			template => "/etc/logstash/file-analysis_template.json"
			manage_template => true
			template_overwrite => true
			template_name => "file-analysis"
		}
	}
}

I don't see any errors in the logstash logs...

Jan 26 17:00:59 pass logstash[24026]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
Jan 26 17:01:00 pass logstash[24026]: [2019-01-26T17:01:00,053][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.5.4"}
Jan 26 17:01:05 pass logstash[24026]: [2019-01-26T17:01:05,582][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
Jan 26 17:01:06 pass logstash[24026]: [2019-01-26T17:01:06,873][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
Jan 26 17:01:07 pass logstash[24026]: [2019-01-26T17:01:07,362][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
Jan 26 17:01:07 pass logstash[24026]: [2019-01-26T17:01:07,488][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
Jan 26 17:01:07 pass logstash[24026]: [2019-01-26T17:01:07,494][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
Jan 26 17:01:07 pass logstash[24026]: [2019-01-26T17:01:07,598][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>"/etc/logstash/file-analysis_template.json"}
Jan 26 17:01:07 pass logstash[24026]: [2019-01-26T17:01:07,615][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost"]}
Jan 26 17:01:07 pass logstash[24026]: [2019-01-26T17:01:07,658][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"index_patterns"=>["file-analysis*"], "mappings"=>{...mapping here...}}}
Jan 26 17:01:07 pass logstash[24026]: [2019-01-26T17:01:07,831][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/file-analysis
Jan 26 17:01:08 pass logstash[24026]: [2019-01-26T17:01:08,328][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x5389b3e0 run>"}
Jan 26 17:01:08 pass logstash[24026]: [2019-01-26T17:01:08,489][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
Jan 26 17:01:08 pass logstash[24026]: [2019-01-26T17:01:08,521][INFO ][filewatch.observingread  ] START, creating Discoverer, Watch with file and sincedb collections
Jan 26 17:01:09 pass logstash[24026]: [2019-01-26T17:01:09,192][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

Nor are there errors from ElasticSearch, and I can see from the entries in the sincedb file that it's seeing the files...

3412186 0 64512 2054 1548536329.043684 /var/log/PASS/analyses/1559663f-cb51-4aaf-ae85-580f2267ec4c
3412194 0 64512 1204 1548536329.037922 /var/log/PASS/analyses/4591de5a-8643-430f-a23a-3240f25751fb
3412199 0 64512 0 1548536329.0432 /var/log/PASS/analyses/2b1a946c-6a9c-4875-8caa-d9a834f2fdf1
3412224 0 64512 1204 1548536328.9789329 /var/log/PASS/analyses/6682d7cd-32bd-4663-967a-0bb4d8007a07
3412226 0 64512 2055 1548536329.0414908 /var/log/PASS/analyses/b57499e9-3fa5-408b-acbf-31d8e497ba1f
3412227 0 64512 1204 1548536329.042288 /var/log/PASS/analyses/ca2652f0-886a-4293-99e7-8379e162f589
3412228 0 64512 2069 1548536329.030073 /var/log/PASS/analyses/5b3ec37f-3288-4875-860a-3e12dae01105
3412229 0 64512 1204 1548536329.040015 /var/log/PASS/analyses/e36c9a42-34a5-49bb-8cab-c4c3b6e389ef

• No data appears in Elasticsearch
• The input files aren't deleted
• The input files are newly generated (i.e. a few minutes to hours old), so this should be unrelated to the ignore_older setting.
• A config using the tail mode loads data without issue to the logstash index, so flow between Logstash and ES is working.

What am I missing?

[Badger]

Enable debug level logging and see what filewatch has to say.

[Kenneth_M_Kolano]

There does not appear to be any relevant logging from filewatch

Jan 26 21:57:16 pass logstash[8397]: [2019-01-26T21:57:16,687][INFO ][filewatch.observingread  ] START, creating Discoverer, Watch with file and sincedb collections
...
Jan 26 21:57:17 pass logstash[8397]: [2019-01-26T21:57:17,812][DEBUG][filewatch.sincedbcollection] writing sincedb (delta since last write = 1548557837)
...
Jan 26 21:57:32 pass logstash[8397]: [2019-01-26T21:57:32,077][DEBUG][filewatch.sincedbcollection] writing sincedb (delta since last write = 15)

I purged the sincedb before that run to ensure files should be reprocessed. I can see the following...

• "Received line" logs related to each file being read (capitalization is a bit off with those logs)
• "filter received" logs related to each file being filtered
• "output received" logs for each file
• "Pushing flush onto pipeline" repeatedly once all files are processed

There are no errors, but the "file-analysis" index remains unpopulated.

[Kenneth_M_Kolano]

Hrm, I think this may be a dumb capitalization issue between my input and output config (i.e. "File-Analysis" vs "file-analysis").