Logstash file as input plugin isn't working

Elastic Stack Logstash
1

Hi,

I am currently running ELK stack on my dcos cluster. I wanted to add few more log paths to Logstash configuration using file input plugin as follows.

input { 
   file { 
      path => ['/var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/tasks/*/stderr*', '/var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/tasks/*/stdout*'] 
      start_position => 'beginning' 
      sincedb_path => '/dev/null' 
      ignore_older => 0
  }
  beats { 
     port=> 5044
  }
}

I even 'vim'ed the stderr and stdout files and I don't see the logs yet on Kibana. Is it my path definition?
But the logstash seems to read the path though as per the logs.

{:timestamp=>"2018-04-04T19:00:53.105000+0000", :message=>"starting agent", :level=>:info}
{:timestamp=>"2018-04-04T19:00:53.306000+0000", :message=>"starting pipeline", :id=>"main", :level=>:info}
{:timestamp=>"2018-04-04T19:01:41.703000+0000", :message=>"Registering file input", :path=>["/var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/tasks/*/stderr*", "/var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/tasks/*/stdout*"], :level=>:info}
{:timestamp=>"2018-04-04T19:01:46.004000+0000", :message=>"Beats inputs: Starting input listener", :address=>"0.0.0.0:5044", :level=>:info}
2

Remove ignore_older, its default is to not ignore anything.

To test whether the paths point to some files try this for each string:

bin/logstash -i irb
irb(main):001:0> Dir.glob('/var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/tasks/*/stderr*')

type exit to return to the prompt.

3

Hi @guyboertje. I have tested the path and this is what I see.

irb(main):001:0> Dir.glob('/var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/tasks/*/stderr*')
=> []
irb(main):002:0> Dir.glob('/var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/tasks/*/stdout*')
=> []

I have also tested:

irb(main):001:0> File.exists?('/var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/tasks/*/stderr*')
=> false
irb(main):002:0> File.exists?('/var/lib/mesos/slave/slaves')                                                                                              
=> false
irb(main):003:0> File.exists?('/var/lib')
=> true
irb(main):004:0> File.exists?('/var/lib/mesos')
=> false
irb(main):005:0>

It could read the path /var/lib but not /var/lib/mesos. Both with root as the owner. Also I'm using Logstash 2.3.4, Elasticsearch and Kibana of version 5.5.1.

4

Any reason why you are using such an old version of Logstash? LS 5.6.X is OK, no?

Are you running Logstash as a service?

Which user account are you running Logstash as?

Ruby essentially does a OS system call under the hood, which is sort of like ls -l /var/lib/mesos*

5

@guyboertje As of now only that version of Logstash package is available on DCOS and yes Logstash is running as a service. By default logstash package is running with the root user.

So what is the work around for this issue?

6

As far as I know, we cant use wild card characters in path. but for file name we can use. Try removing * and give proper path.

7

There is not really a work around for this. If the Logstash IRB shell Dir.glob('path') does not return an array of files then the file input will not discover any files - because this Dir.glob is exactly what the file input does under the hood.

I think this is a DCOS permissions issue. How do you know that files exists at that deep level of nesting?

This link has some output mentioning a path that resembles what you are trying to glob.

@Suman_Reddy1 - wildcard characters are supported in the path setting of the file input.

8

You mean the Cluster permissions or the service's?

9

I don't know DCOS. Is there a community forum that you can ask?
If ls -l /var/lib/mesos gives permission denied then you should investigate what you need to do to get read access to the files you want Logstash to read.

10

Thanks @guyboertje Any idea when the latest version of logstash will be available on DCOS?

11

We do not control it. This may help https://github.com/mesos/logstash. You may have to build a more up to date LS version yourself. After you have sorted out the permissions thing though.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.