# Logstash file as input plugin isn't working **URL:** \ **Category:** Logstash\ **Created:** [April 4, 2018, 6:55pm UTC](https://discuss.elastic.co/t/logstash-file-as-input-plugin-isnt-working/126806 "2018-04-04T18:55:11Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![aysala](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@aysala](https://discuss.elastic.co/u/aysala)\ **Post date:** [April 4, 2018, 6:55pm UTC](https://discuss.elastic.co/t/logstash-file-as-input-plugin-isnt-working/126806/1 "2018-04-04T18:55:11Z")
Hi, I am currently running ELK stack on my dcos cluster. I wanted to add few more log paths to Logstash configuration using file input plugin as follows. ``` input { file { path => ['/var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/tasks/*/stderr*', '/var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/tasks/*/stdout*'] start_position => 'beginning' sincedb_path => '/dev/null' ignore_older => 0 } beats { port=> 5044 } } ``` I even 'vim'ed the stderr and stdout files and I don't see the logs yet on Kibana. Is it my path definition? But the logstash seems to read the path though as per the logs. ``` {:timestamp=>"2018-04-04T19:00:53.105000+0000", :message=>"starting agent", :level=>:info} {:timestamp=>"2018-04-04T19:00:53.306000+0000", :message=>"starting pipeline", :id=>"main", :level=>:info} {:timestamp=>"2018-04-04T19:01:41.703000+0000", :message=>"Registering file input", :path=>["/var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/tasks/*/stderr*", "/var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/tasks/*/stdout*"], :level=>:info} {:timestamp=>"2018-04-04T19:01:46.004000+0000", :message=>"Beats inputs: Starting input listener", :address=>"0.0.0.0:5044", :level=>:info} ``` ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [April 4, 2018, 7:32pm UTC](https://discuss.elastic.co/t/logstash-file-as-input-plugin-isnt-working/126806/2 "2018-04-04T19:32:01Z")
Remove ignore\_older, its default is to **not** ignore anything. To test whether the paths point to some files try this for each string: ```auto bin/logstash -i irb irb(main):001:0> Dir.glob('/var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/tasks/*/stderr*') ``` type `exit` to return to the prompt. ---
**Author:** ![aysala](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@aysala](https://discuss.elastic.co/u/aysala)\ **Post date:** [April 5, 2018, 4:28am UTC](https://discuss.elastic.co/t/logstash-file-as-input-plugin-isnt-working/126806/3 "2018-04-05T04:28:40Z")
Hi @guyboertje. I have tested the path and this is what I see. ``` irb(main):001:0> Dir.glob('/var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/tasks/*/stderr*') => [] irb(main):002:0> Dir.glob('/var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/tasks/*/stdout*') => [] ``` I have also tested: ``` irb(main):001:0> File.exists?('/var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/tasks/*/stderr*') => false irb(main):002:0> File.exists?('/var/lib/mesos/slave/slaves') => false irb(main):003:0> File.exists?('/var/lib') => true irb(main):004:0> File.exists?('/var/lib/mesos') => false irb(main):005:0> ``` It could read the path `/var/lib` but not `/var/lib/mesos`. Both with root as the owner. Also I'm using Logstash 2.3.4, Elasticsearch and Kibana of version 5.5.1. ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [April 5, 2018, 5:10pm UTC](https://discuss.elastic.co/t/logstash-file-as-input-plugin-isnt-working/126806/4 "2018-04-05T17:10:01Z")
Any reason why you are using such an old version of Logstash? LS 5.6.X is OK, no? Are you running Logstash as a service? Which user account are you running Logstash as? Ruby essentially does a OS system call under the hood, which is sort of like `ls -l /var/lib/mesos*` ---
**Author:** ![aysala](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@aysala](https://discuss.elastic.co/u/aysala)\ **Post date:** [April 5, 2018, 5:22pm UTC](https://discuss.elastic.co/t/logstash-file-as-input-plugin-isnt-working/126806/5 "2018-04-05T17:22:55Z")
@guyboertje As of now only that version of Logstash package is available on DCOS and yes Logstash is running as a service. By default logstash package is running with the root user. So what is the work around for this issue? ---
**Author:** ![Suman\_Reddy1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/suman_reddy1/32/27303_2.png) [@Suman\_Reddy1](https://discuss.elastic.co/u/Suman_Reddy1)\ **Post date:** [April 5, 2018, 5:27pm UTC](https://discuss.elastic.co/t/logstash-file-as-input-plugin-isnt-working/126806/6 "2018-04-05T17:27:47Z")
As far as I know, we cant use wild card characters in path. but for file name we can use. Try removing \* and give proper path. ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [April 5, 2018, 5:53pm UTC](https://discuss.elastic.co/t/logstash-file-as-input-plugin-isnt-working/126806/7 "2018-04-05T17:53:42Z")
There is not really a work around for this. If the Logstash IRB shell `Dir.glob('path')` does not return an array of files then the file input will not discover any files - because this `Dir.glob` is exactly what the file input does under the hood. I think this is a DCOS permissions issue. How do you know that files exists at that deep level of nesting? This [link](https://docs.mesosphere.com/1.11/monitoring/logging/access-task-logs/#verifying-access) has some output mentioning a path that resembles what you are trying to glob. @Suman_Reddy1 - wildcard characters **are** supported in the `path` setting of the file input. ---
**Author:** ![aysala](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@aysala](https://discuss.elastic.co/u/aysala)\ **Post date:** [April 5, 2018, 6:43pm UTC](https://discuss.elastic.co/t/logstash-file-as-input-plugin-isnt-working/126806/8 "2018-04-05T18:43:51Z")
> [@guyboertje](#): > > I think this is a DCOS permissions issue. How do you know that files exists at that deep level of nesting? You mean the Cluster permissions or the service's? ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [April 5, 2018, 6:53pm UTC](https://discuss.elastic.co/t/logstash-file-as-input-plugin-isnt-working/126806/9 "2018-04-05T18:53:51Z")
I don't know DCOS. Is there a community forum that you can ask? If `ls -l /var/lib/mesos` gives permission denied then you should investigate what you need to do to get read access to the files you want Logstash to read. ---
**Author:** ![aysala](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@aysala](https://discuss.elastic.co/u/aysala)\ **Post date:** [April 5, 2018, 9:06pm UTC](https://discuss.elastic.co/t/logstash-file-as-input-plugin-isnt-working/126806/10 "2018-04-05T21:06:04Z")
Thanks @guyboertje Any idea when the latest version of logstash will be available on DCOS? ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [April 5, 2018, 10:06pm UTC](https://discuss.elastic.co/t/logstash-file-as-input-plugin-isnt-working/126806/11 "2018-04-05T22:06:43Z")
We do not control it. This may help [https://github.com/mesos/logstash](https://github.com/mesos/logstash). You may have to build a more up to date LS version yourself. After you have sorted out the permissions thing though. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 3, 2018, 10:07pm UTC](https://discuss.elastic.co/t/logstash-file-as-input-plugin-isnt-working/126806/12 "2018-05-03T22:07:01Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.