# Recursive depth of Logstash - not finding my files? **URL:** \ **Category:** Logstash\ **Created:** [May 17, 2019, 1:37pm UTC](https://discuss.elastic.co/t/recursive-depth-of-logstash-not-finding-my-files/181602 "2019-05-17T13:37:58Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![sgreszcz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sgreszcz/32/46322_2.png) [@sgreszcz](https://discuss.elastic.co/u/sgreszcz)\ **Post date:** [May 17, 2019, 1:37pm UTC](https://discuss.elastic.co/t/recursive-depth-of-logstash-not-finding-my-files/181602/1 "2019-05-17T13:37:58Z")
I'm really having problems finding the csv files I need to parse with Logstash. I'm wondering if it is my configuration or there is a limit to the recursion depth. test@host-001:/data/tracelogger/tracelogger\_data# find . -name "PerfMon\*.csv" ./processed/AP4/10.66.6.21/2019-05-17\_06-00-03/cm/log/ris/csv/PerfMon\_05\_17\_2019\_04\_25.csv ./processed/AP4/10.66.6.21/2019-05-17\_01-45-23/cm/log/ris/csv/PerfMon\_05\_17\_2019\_00\_55.csv ./processed/AP4/10.66.6.21/2019-05-17\_06-59-20/cm/log/ris/csv/PerfMon\_05\_17\_2019\_06\_10.csv ./processed/AP4/10.66.6.21/2019-05-17\_00-00-07/cm/log/ris/csv/PerfMon\_05\_16\_2019\_23\_10.csv` However the logstash config doesn’t find them based on this config: input { file { id =\> "perfmon-in" path =\> "/data/tracelogger/tracelogger\_data/\*\*/PerfMon\*" tags =\> ["perfmon"] } } filter { csv { id =\> "perfmon-filter" autodetect\_column\_names =\> "true" convert =\> { "column1" =\> "date\_time" } } } output { if "perfmon" in [tags]{ ``` elasticsearch { hosts => "${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}" index => "perfmon-%{+YYYY.MM.dd}" id => "perfmon-out" } stdout { codec => rubydebug } } ``` } And I can see logstash configuring the pipeline OK in the logs: 2019-05-17 12:06:37.762 [2019-05-17T11:06:37,762][INFO][logstash.inputs.file] No sincedb\_path set, generating one based on the "path" setting {:sincedb\_path=\>"/usr/share/logstash/data/plugins/inputs/file/.sincedb\_d0460663a664248d37e0e08a24eedf1f", :path=\>["/data/tracelogger/tracelogger\_data/\*\*/PerfMon\*"]} 2019-05-17 12:06:37.817 [2019-05-17T11:06:37,816][INFO][logstash.javapipeline] Pipeline started {"pipeline.id"=\>"perfmon"} 2019-05-17 12:06:37.826 [2019-05-17T11:06:37,826][DEBUG][logstash.javapipeline] Pipeline started successfully {:pipeline\_id=\>"perfmon", :thread=\>"#\"} I have debug logs on but I never see logstash finding any files. ---
**Author:** ![pastechecker](https://avatars.discourse-cdn.com/v4/letter/p/0ea827/32.png) [@pastechecker](https://discuss.elastic.co/u/pastechecker)\ **Post date:** [May 17, 2019, 1:51pm UTC](https://discuss.elastic.co/t/recursive-depth-of-logstash-not-finding-my-files/181602/2 "2019-05-17T13:51:35Z")
> [@sgreszcz](#): > > e "PerfMon\*.csv Could you please try with the absolute path? path =\> "/data/tracelogger/tracelogger\_data/\*\*/PerfMon\*.csv" Also can you verify that the Logstash has the permissions to read the files from the directory? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [May 17, 2019, 2:23pm UTC](https://discuss.elastic.co/t/recursive-depth-of-logstash-not-finding-my-files/181602/3 "2019-05-17T14:23:38Z")
> [@sgreszcz](#): > > I have debug logs on but I never see logstash finding any files. All the filewatch file handling is logged at trace level. ---
**Author:** ![sgreszcz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sgreszcz/32/46322_2.png) [@sgreszcz](https://discuss.elastic.co/u/sgreszcz)\ **Post date:** [May 17, 2019, 2:37pm UTC](https://discuss.elastic.co/t/recursive-depth-of-logstash-not-finding-my-files/181602/4 "2019-05-17T14:37:01Z")
OK, thanks I'll set the logging to that level. ---
**Author:** ![sgreszcz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sgreszcz/32/46322_2.png) [@sgreszcz](https://discuss.elastic.co/u/sgreszcz)\ **Post date:** [May 17, 2019, 2:41pm UTC](https://discuss.elastic.co/t/recursive-depth-of-logstash-not-finding-my-files/181602/5 "2019-05-17T14:41:40Z")
I put the explicit path now (no \*\* glob expansion) and I still don't see the file being picked up: path =\> "/data/tracelogger/tracelogger\_data/processed/AP4/10.66.132.36/2019-05-17\_21-29-18/cm/log/ris/csv/PerfMon\*" I also know for sure the file can be found. I'm using docker but the /data volume is shared between host and container. I entered into the container and ran a bash 'find . -name "PerfMon\*" ' command from the /data/tracelogger/tracelogger\_data/processed/ path and could find the .csv files. I'm going to set up trace logging level to see what I find. Super frustrating, yikes. ---
**Author:** ![sgreszcz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sgreszcz/32/46322_2.png) [@sgreszcz](https://discuss.elastic.co/u/sgreszcz)\ **Post date:** [May 17, 2019, 2:51pm UTC](https://discuss.elastic.co/t/recursive-depth-of-logstash-not-finding-my-files/181602/6 "2019-05-17T14:51:55Z")
OK, put the specific path in and I can see Logstash "watching" the file that it found. However I guess it thinks that it already parsed the csv as it says "no change" based in he "sincedb\_key". Thanks for your help so far, I'm going to try and wipe the "sincedb" and try again, also to see if I can see the proper \*\* glob expansion now that I'm using the "trace" debugs. [2019-05-17T14:46:34,446][TRACE][filewatch.tailmode.processor] Active - no change {"watched\_file"=\>"\"} ---
**Author:** ![pastechecker](https://avatars.discourse-cdn.com/v4/letter/p/0ea827/32.png) [@pastechecker](https://discuss.elastic.co/u/pastechecker)\ **Post date:** [May 17, 2019, 2:54pm UTC](https://discuss.elastic.co/t/recursive-depth-of-logstash-not-finding-my-files/181602/7 "2019-05-17T14:54:44Z")
Unless you do not want to monitor that files as it is rotating with time, you can go to the dev null, but after the potential logstash crash you will ingest the data again. ---
**Author:** ![sgreszcz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sgreszcz/32/46322_2.png) [@sgreszcz](https://discuss.elastic.co/u/sgreszcz)\ **Post date:** [May 17, 2019, 3:38pm UTC](https://discuss.elastic.co/t/recursive-depth-of-logstash-not-finding-my-files/181602/8 "2019-05-17T15:38:55Z")
@Badger OK, something is definitely not working well with the wildcard patterns. I wonder if it is due to the depth of path? I can't find any documentation about maximum file path depth (although in filebeat docs it mentions something about 8). Anyway, I can "find" my files if I use this pattern: `path => "/data/tracelogger/tracelogger_data/processed/AP4/10.66.132.23/2019-05-17_20-21-30/cm/log/ris/csv/PerfMon*"` But if I use this pattern it doesn't work (variable expansion wildcard): `path => "/data/tracelogger/tracelogger_data/**/PerfMon*"` Nor this one (explicit wildcards for the fixed path depth): `path => "/data/tracelogger/tracelogger_data/*/*/*/*/*/*/*/*/PerfMon*"` I guess I can try filebeat and see if I can stream the files into logstash to process. ---
**Author:** ![mailtoarasu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mailtoarasu/32/46429_2.png) [@mailtoarasu](https://discuss.elastic.co/u/mailtoarasu)\ **Post date:** [May 18, 2019, 7:47pm UTC](https://discuss.elastic.co/t/recursive-depth-of-logstash-not-finding-my-files/181602/9 "2019-05-18T19:47:01Z")
facing same issue with wildcard patterns. I am trying to process some of log files that are 2 weeks older. I am running logstash - 7.0.1 as a docker in "read" mode since log files are complete. This is example directory structure /backup\_may2019/received\_date=2019-05-05/received\_time=02-00-00/ /backup\_may2019/received\_date=2019-05-05/received\_time=03-00-00/ It is working fine if i give absolute path up to the last level folder `path => "/backup_may2019/received_date=2019-05-05/received_time=03-00-00/*"` it doesn't read files if i use wildcards in the folder path: `path => "/backup_may2019/received_date=2019-05-05/*/*"` `path => "/backup_may2019/received_date=2019-05-05/**/*"` ---
**Author:** ![sgreszcz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sgreszcz/32/46322_2.png) [@sgreszcz](https://discuss.elastic.co/u/sgreszcz)\ **Post date:** [May 18, 2019, 7:59pm UTC](https://discuss.elastic.co/t/recursive-depth-of-logstash-not-finding-my-files/181602/10 "2019-05-18T19:59:55Z")
Thanks for much for verifying the problem that I'm also experiencing. I'm going to try filebeat next week, but I'm afraid I'll have the same problem. I'm not sure why there is a directory depth limit. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 15, 2019, 7:59pm UTC](https://discuss.elastic.co/t/recursive-depth-of-logstash-not-finding-my-files/181602/11 "2019-06-15T19:59:59Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.