Logstash file as input plugin isn't working

[aysala]

Hi,

I am currently running ELK stack on my dcos cluster. I wanted to add few more log paths to Logstash configuration using file input plugin as follows.

input { 
   file { 
      path => ['/var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/tasks/*/stderr*', '/var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/tasks/*/stdout*'] 
      start_position => 'beginning' 
      sincedb_path => '/dev/null' 
      ignore_older => 0
  }
  beats { 
     port=> 5044
  }
}

I even 'vim'ed the stderr and stdout files and I don't see the logs yet on Kibana. Is it my path definition?
But the logstash seems to read the path though as per the logs.

{:timestamp=>"2018-04-04T19:00:53.105000+0000", :message=>"starting agent", :level=>:info}
{:timestamp=>"2018-04-04T19:00:53.306000+0000", :message=>"starting pipeline", :id=>"main", :level=>:info}
{:timestamp=>"2018-04-04T19:01:41.703000+0000", :message=>"Registering file input", :path=>["/var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/tasks/*/stderr*", "/var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/tasks/*/stdout*"], :level=>:info}
{:timestamp=>"2018-04-04T19:01:46.004000+0000", :message=>"Beats inputs: Starting input listener", :address=>"0.0.0.0:5044", :level=>:info}

[guyboertje]

Remove ignore_older, its default is to not ignore anything.

To test whether the paths point to some files try this for each string:

bin/logstash -i irb
irb(main):001:0> Dir.glob('/var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/tasks/*/stderr*')

type exit to return to the prompt.

[aysala]

Hi @guyboertje. I have tested the path and this is what I see.

irb(main):001:0> Dir.glob('/var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/tasks/*/stderr*')
=> []
irb(main):002:0> Dir.glob('/var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/tasks/*/stdout*')
=> [] 

I have also tested:

irb(main):001:0> File.exists?('/var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/tasks/*/stderr*')
=> false
irb(main):002:0> File.exists?('/var/lib/mesos/slave/slaves')                                                                                              
=> false
irb(main):003:0> File.exists?('/var/lib')
=> true
irb(main):004:0> File.exists?('/var/lib/mesos')
=> false
irb(main):005:0>

It could read the path /var/lib but not /var/lib/mesos. Both with root as the owner. Also I'm using Logstash 2.3.4, Elasticsearch and Kibana of version 5.5.1.

[guyboertje]

Any reason why you are using such an old version of Logstash? LS 5.6.X is OK, no?

Are you running Logstash as a service?

Which user account are you running Logstash as?

Ruby essentially does a OS system call under the hood, which is sort of like ls -l /var/lib/mesos*

[aysala]

@guyboertje As of now only that version of Logstash package is available on DCOS and yes Logstash is running as a service. By default logstash package is running with the root user.

So what is the work around for this issue?

[Suman_Reddy1 (Suman Reddy)]

As far as I know, we cant use wild card characters in path. but for file name we can use. Try removing * and give proper path.

[guyboertje]

There is not really a work around for this. If the Logstash IRB shell Dir.glob('path') does not return an array of files then the file input will not discover any files - because this Dir.glob is exactly what the file input does under the hood.

I think this is a DCOS permissions issue. How do you know that files exists at that deep level of nesting?

This link has some output mentioning a path that resembles what you are trying to glob.

@Suman_Reddy1 - wildcard characters are supported in the path setting of the file input.

[aysala]

@guyboertje

I think this is a DCOS permissions issue. How do you know that files exists at that deep level of nesting?

You mean the Cluster permissions or the service's?

[guyboertje]

I don't know DCOS. Is there a community forum that you can ask?
If ls -l /var/lib/mesos gives permission denied then you should investigate what you need to do to get read access to the files you want Logstash to read.

[aysala]

Thanks @guyboertje Any idea when the latest version of logstash will be available on DCOS?

[guyboertje]

We do not control it. This may help https://github.com/mesos/logstash. You may have to build a more up to date LS version yourself. After you have sorted out the permissions thing though.