# Logstash file input works on one path but not others **URL:** \ **Category:** Logstash\ **Created:** [October 11, 2018, 8:18pm UTC](https://discuss.elastic.co/t/logstash-file-input-works-on-one-path-but-not-others/152126 "2018-10-11T20:18:32Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Scott\_Newby](https://avatars.discourse-cdn.com/v4/letter/s/eb8c5e/32.png) [@Scott\_Newby](https://discuss.elastic.co/u/Scott_Newby)\ **Post date:** [October 11, 2018, 8:18pm UTC](https://discuss.elastic.co/t/logstash-file-input-works-on-one-path-but-not-others/152126/1 "2018-10-11T20:18:32Z")
I was able to load the json logs we're generating, flattening with a ruby script, and then loading to elastic. We're on a windows server 2012 environment. Logstash 6.3.0. This works on one path, but not another > input{ > file { > #file monitor - glob to monitor multiple directories > # this works - path =\> ["/Jenkins/jobs/Testing Pipeline/jobs/Pipeline/builds/star-star/cucumber-html-reports/.cache/star.json"] > # this doesn't work - path =\> ["/Jenkins/jobs/Testing Pipeline\_UnderDevelopment/jobs/projecg-name\_herewithunderscore/builds/star-star/cucumber-html-reports/.cache/reports/star.json"] > # this also doesn't work - path =\> ["/ELK/Test/reports/star.json"] > start\_position =\> "beginning" > #use the multiline codec to treat the message as one big message > codec =\> multiline { pattern =\> "something" negate =\> true what =\> "previous" auto\_flush\_interval =\> 2 } > } > } star is asterisk for wildcard matching. This is running in my pipeline.yml, and has been tested against each of the path cases above. In the first 2 cases - .cache is not a hidden directory, and that wouldn't explain why the 3rd path is not working. I was wondering if the underscore was causing an issue. I've deleted the data directory in between executions to remove any tracking, besides the files are not duplicates. I've checked permissions between all three and they are all the same. Last log file below: > [2018-10-11T16:00:57,106][INFO][logstash.setting.writabledirectory] Creating directory {:setting=\>"path.queue", :path=\>"c:/ELK/logstash/data/queue"} > [2018-10-11T16:00:57,122][INFO][logstash.setting.writabledirectory] Creating directory {:setting=\>"path.dead\_letter\_queue", :path=\>"c:/ELK/logstash/data/dead\_letter\_queue"} > [2018-10-11T16:00:57,525][INFO][logstash.agent] No persistent UUID file found. Generating new UUID {:uuid=\>"eeb40e51-c9d7-4663-b2dd-3df0db29cc92", :path=\>"c:/ELK/logstash/data/uuid"} > [2018-10-11T16:00:58,520][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"6.3.0"} > [2018-10-11T16:01:04,906][INFO][logstash.filters.ruby.script] Test run complete {:script\_path=\>"/ELK/logstash/config/Code/cucumber-agg-json.rb", :results=\>{:passed=\>0, :failed=\>0, :errored=\>0}} > [2018-10-11T16:01:05,159][INFO][logstash.pipeline] Starting pipeline {:pipeline\_id=\>"CucumberAggregationAlt", "pipeline.workers"=\>4, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50} > [2018-10-11T16:01:05,779][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=\>{:removed=\>, :added=\>[[http://localhost:9200/](http://localhost:9200/)]}} > [2018-10-11T16:01:05,795][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[http://localhost:9200/](http://localhost:9200/), :path=\>"/"} > [2018-10-11T16:01:06,089][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>"[http://localhost:9200/](http://localhost:9200/)"} > [2018-10-11T16:01:06,303][INFO][logstash.outputs.elasticsearch] ES Output version determined {:es\_version=\>6} > [2018-10-11T16:01:06,314][WARN][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document \_type {:es\_version=\>6} > [2018-10-11T16:01:06,348][INFO][logstash.outputs.elasticsearch] Using mapping template from {:path=\>nil} > [2018-10-11T16:01:06,379][INFO][logstash.outputs.elasticsearch] Attempting to install template {:manage\_template=\>{"template"=\>"logstash-_", "version"=\>60001, "settings"=\>{"index.refresh\_interval"=\>"5s"}, "mappings"=\>{"default"=\>{"dynamic\_templates"=\>[{"message\_field"=\>{"path\_match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false}}}, {"string\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false, "fields"=\>{"keyword"=\>{"type"=\>"keyword", "ignore\_above"=\>256}}}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date"}, "@version"=\>{"type"=\>"keyword"}, "geoip"=\>{"dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip"}, "location"=\>{"type"=\>"geo\_point"}, "latitude"=\>{"type"=\>"half\_float"}, "longitude"=\>{"type"=\>"half\_float"}}}}}}}} > [2018-10-11T16:01:06,449][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=\>"LogStash::Outputs::Elasticsearch", :hosts=\>["[//localhost](https://localhost)"]} > [2018-10-11T16:01:07,754][INFO][logstash.pipeline] Pipeline started successfully {:pipeline\_id=\>"CucumberAggregationAlt", :thread=\>"#\"} > [2018-10-11T16:01:07,881][INFO][logstash.agent] Pipelines running {:count=\>1, :running\_pipelines=\>[:CucumberAggregationAlt], :non\_running\_pipelines=\>} > [2018-10-11T16:01:08,546][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600} > [2018-10-11T16:15:59,122][WARN][logstash.runner] SIGINT received. Shutting down. > [2018-10-11T16:16:00,298][INFO][logstash.pipeline] Pipeline has terminated {:pipeline\_id=\>"CucumberAggregationAlt", :thread=\>"#\"} I'm at a loss here - Scott ---
**Author:** ![Scott\_Newby](https://avatars.discourse-cdn.com/v4/letter/s/eb8c5e/32.png) [@Scott\_Newby](https://discuss.elastic.co/u/Scott_Newby)\ **Post date:** [October 12, 2018, 12:07pm UTC](https://discuss.elastic.co/t/logstash-file-input-works-on-one-path-but-not-others/152126/2 "2018-10-12T12:07:44Z")
This appears to be a data ingestion issue with the json we're trying to ingest, not a path issue. I moved known working files to the path(s) in question and logstash was able to access, and vice-versa to validate. ---
**Author:** ![davemoore](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davemoore/32/134233_2.png) [@davemoore](https://discuss.elastic.co/u/davemoore)\ **Post date:** [October 12, 2018, 2:41pm UTC](https://discuss.elastic.co/t/logstash-file-input-works-on-one-path-but-not-others/152126/3 "2018-10-12T14:41:34Z")
@Scott_Newby Can you execute the following pipeline in your terminal and share your observations? Ideally you will see Logstash printing parsed log messages to your terminal. ```ruby input { file { path => ["/path/to/file"] # Replace me start_position => "beginning" sincedb_path => "NUL" codec => multiline { pattern => "something" # Replace me negate => true what => "previous" auto_flush_interval => 2 } } } output { stdout { codec => rubydebug } } ``` Please take note: - Replace the values of `path` and `pattern`. - I added `sincedb_path => "NUL"` to ensure that the file is read from the beginning whenever you run this pipeline. ([Details](https://github.com/logstash-plugins/logstash-input-file/issues/26#issuecomment-173202887)) - The `stdout` output will print each message to your terminal in a structured format, allowing you to see if/how the messages are being parsed. ---
**Author:** ![Scott\_Newby](https://avatars.discourse-cdn.com/v4/letter/s/eb8c5e/32.png) [@Scott\_Newby](https://discuss.elastic.co/u/Scott_Newby)\ **Post date:** [October 12, 2018, 6:08pm UTC](https://discuss.elastic.co/t/logstash-file-input-works-on-one-path-but-not-others/152126/4 "2018-10-12T18:08:00Z")
The output of the logs we were ingesting changed from multi-line JSON to single line JSON, which might make things easier. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 9, 2018, 6:09pm UTC](https://discuss.elastic.co/t/logstash-file-input-works-on-one-path-but-not-others/152126/5 "2018-11-09T18:09:04Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.