# Logstash File Input Plugin Read Mode Not Loading Data **URL:** \ **Category:** Logstash\ **Created:** [January 26, 2019, 10:13pm UTC](https://discuss.elastic.co/t/logstash-file-input-plugin-read-mode-not-loading-data/165869 "2019-01-26T22:13:31Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Kenneth\_M\_Kolano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kenneth_m_kolano/32/38752_2.png) [@Kenneth\_M\_Kolano](https://discuss.elastic.co/u/Kenneth_M_Kolano)\ **Post date:** [January 26, 2019, 10:13pm UTC](https://discuss.elastic.co/t/logstash-file-input-plugin-read-mode-not-loading-data/165869/1 "2019-01-26T22:13:32Z")
I'm having trouble getting Logstash to load data to Elasticsearch when using read mode. ``` input { file { mode => "read" path => ["/var/log/PASS/analyses/*"] sincedb_path => ["/var/lib/logstash/file-analysis_sincedb"] codec => json type => "File-Analysis" } } output { if [type] == "file-analysis" { elasticsearch { hosts => localhost index => "file-analysis" template => "/etc/logstash/file-analysis_template.json" manage_template => true template_overwrite => true template_name => "file-analysis" } } } ``` I don't see any errors in the logstash logs... ``` Jan 26 17:00:59 pass logstash[24026]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties Jan 26 17:01:00 pass logstash[24026]: [2019-01-26T17:01:00,053][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"6.5.4"} Jan 26 17:01:05 pass logstash[24026]: [2019-01-26T17:01:05,582][INFO][logstash.pipeline] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50} Jan 26 17:01:06 pass logstash[24026]: [2019-01-26T17:01:06,873][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}} Jan 26 17:01:07 pass logstash[24026]: [2019-01-26T17:01:07,362][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"} Jan 26 17:01:07 pass logstash[24026]: [2019-01-26T17:01:07,488][INFO][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6} Jan 26 17:01:07 pass logstash[24026]: [2019-01-26T17:01:07,494][WARN][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6} Jan 26 17:01:07 pass logstash[24026]: [2019-01-26T17:01:07,598][INFO][logstash.outputs.elasticsearch] Using mapping template from {:path=>"/etc/logstash/file-analysis_template.json"} Jan 26 17:01:07 pass logstash[24026]: [2019-01-26T17:01:07,615][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost"]} Jan 26 17:01:07 pass logstash[24026]: [2019-01-26T17:01:07,658][INFO][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"index_patterns"=>["file-analysis*"], "mappings"=>{...mapping here...}}} Jan 26 17:01:07 pass logstash[24026]: [2019-01-26T17:01:07,831][INFO][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/file-analysis Jan 26 17:01:08 pass logstash[24026]: [2019-01-26T17:01:08,328][INFO][logstash.pipeline] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#"} Jan 26 17:01:08 pass logstash[24026]: [2019-01-26T17:01:08,489][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} Jan 26 17:01:08 pass logstash[24026]: [2019-01-26T17:01:08,521][INFO][filewatch.observingread] START, creating Discoverer, Watch with file and sincedb collections Jan 26 17:01:09 pass logstash[24026]: [2019-01-26T17:01:09,192][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600} ``` Nor are there errors from ElasticSearch, and I can see from the entries in the sincedb file that it's seeing the files... ``` 3412186 0 64512 2054 1548536329.043684 /var/log/PASS/analyses/1559663f-cb51-4aaf-ae85-580f2267ec4c 3412194 0 64512 1204 1548536329.037922 /var/log/PASS/analyses/4591de5a-8643-430f-a23a-3240f25751fb 3412199 0 64512 0 1548536329.0432 /var/log/PASS/analyses/2b1a946c-6a9c-4875-8caa-d9a834f2fdf1 3412224 0 64512 1204 1548536328.9789329 /var/log/PASS/analyses/6682d7cd-32bd-4663-967a-0bb4d8007a07 3412226 0 64512 2055 1548536329.0414908 /var/log/PASS/analyses/b57499e9-3fa5-408b-acbf-31d8e497ba1f 3412227 0 64512 1204 1548536329.042288 /var/log/PASS/analyses/ca2652f0-886a-4293-99e7-8379e162f589 3412228 0 64512 2069 1548536329.030073 /var/log/PASS/analyses/5b3ec37f-3288-4875-860a-3e12dae01105 3412229 0 64512 1204 1548536329.040015 /var/log/PASS/analyses/e36c9a42-34a5-49bb-8cab-c4c3b6e389ef ``` - No data appears in Elasticsearch - The input files aren't deleted - The input files are newly generated (i.e. a few minutes to hours old), so this should be unrelated to the `ignore_older` setting. - A config using the tail mode loads data without issue to the logstash index, so flow between Logstash and ES is working. What am I missing? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [January 26, 2019, 10:44pm UTC](https://discuss.elastic.co/t/logstash-file-input-plugin-read-mode-not-loading-data/165869/2 "2019-01-26T22:44:26Z")
Enable debug level logging and see what filewatch has to say. ---
**Author:** ![Kenneth\_M\_Kolano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kenneth_m_kolano/32/38752_2.png) [@Kenneth\_M\_Kolano](https://discuss.elastic.co/u/Kenneth_M_Kolano)\ **Post date:** [January 27, 2019, 3:12am UTC](https://discuss.elastic.co/t/logstash-file-input-plugin-read-mode-not-loading-data/165869/3 "2019-01-27T03:12:54Z")
There does not appear to be any relevant logging from filewatch ``` Jan 26 21:57:16 pass logstash[8397]: [2019-01-26T21:57:16,687][INFO][filewatch.observingread] START, creating Discoverer, Watch with file and sincedb collections ... Jan 26 21:57:17 pass logstash[8397]: [2019-01-26T21:57:17,812][DEBUG][filewatch.sincedbcollection] writing sincedb (delta since last write = 1548557837) ... Jan 26 21:57:32 pass logstash[8397]: [2019-01-26T21:57:32,077][DEBUG][filewatch.sincedbcollection] writing sincedb (delta since last write = 15) ``` I purged the sincedb before that run to ensure files should be reprocessed. I can see the following... - "Received line" logs related to each file being read (capitalization is a bit off with those logs) - "filter received" logs related to each file being filtered - "output received" logs for each file - "Pushing flush onto pipeline" repeatedly once all files are processed There are no errors, but the "file-analysis" index remains unpopulated. ---
**Author:** ![Kenneth\_M\_Kolano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kenneth_m_kolano/32/38752_2.png) [@Kenneth\_M\_Kolano](https://discuss.elastic.co/u/Kenneth_M_Kolano)\ **Post date:** [January 27, 2019, 3:14am UTC](https://discuss.elastic.co/t/logstash-file-input-plugin-read-mode-not-loading-data/165869/4 "2019-01-27T03:14:10Z")
Hrm, I think this may be a dumb capitalization issue between my input and output config (i.e. "File-Analysis" vs "file-analysis"). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 24, 2019, 3:28am UTC](https://discuss.elastic.co/t/logstash-file-input-plugin-read-mode-not-loading-data/165869/5 "2019-02-24T03:28:46Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.