Logstash.inputs.file failed to open

[niraj_kumar]

Hi,

I am facing an issue where logstash is not able to read my files. I tried granted all permission to the files ( 777 ) and changing the owner from root to logstash and even to my local user which i use to login. I am attaching the details below.

LS Version:- 5.5.0-1

Config:

input {
file {
        path => "/data/**/*.gz"
        codec => "gzip_lines"
        start_position => "beginning"
        sincedb_path => "/var/lib/logstash/.sincedb"
        type => "cloudtrail"
        max_open_files => "65535"
  }
}

filter {
   grok {
       match => {"[path]" => "/data/(?<tstmp>\S+)/.*"}
   }
}

output {
  stdout { codec => json }

  elasticsearch {
        hosts => ["xx.xx.xx.xx:9200"]
        index => "%{[tstmp]}-%{+YYYY-MM}"
   }
}

ls -lrt output

niraj@niraj-z820:/var/log/logstash$ ll /data/bigdata/AWSLogs/xxxxxxxxxxxxx/CloudTrail/ca-central-1/2017/06/25/xxxxxxxxxxxxx_CloudTrail_ca-central-1_20170625T1045Z_y7hwapLjNSMOYnLh.json.gz
-rwxrwxrwx 1 logstash logstash 1022 Aug  3 00:54 /data/bigdata/AWSLogs/xxxxxxxxxxxxx/CloudTrail/ca-central-1/2017/06/25/xxxxxxxxxxxxx_CloudTrail_ca-central-1_20170625T1045Z_y7hwapLjNSMOYnLh.json.gz*

Error in Logstash

[2017-08-03T01:14:45,067][WARN ][logstash.inputs.file ] failed to open /data/bigdata/AWSLogs/xxxxxxxxxxxxx/CloudTrail/ca-central-1/2017/06/25/xxxxxxxxxxxxx_CloudTrail_ca-central-1_xxxxxxxxxxxxZ_y7hwapLjNSMOYnLh.json.gz: Permission denied - /data/bigdata/AWSLogs/xxxxxxxxxxxxx/CloudTrail/ca-central-1/2017/06/25/xxxxxxxxxxxxx_CloudTrail_ca-central-1_xxxxxxxxxxxxxxZ_y7hwapLjNSMOYnLh.json.gz

--
Niraj

[magnusbaeck (Magnus Bäck)]

Having read access to the file itself is necessary but not sufficient. Does the user that Logstash runs as have read and execute permissions to all of directories leading up to the file you're trying to get it to read?

[niraj_kumar]

@magnusbaeck I understand that and i have recursively given permission on the parent directory itself.

Like a chmod -R 777 /data.

Let me know if you want any output to validate that.

niraj@niraj-z820:/data$ ls -lrt
total 4
drwxrwxrwx 3 logstash logstash 4096 Aug  3 00:48 bigdata

[niraj_kumar]

@magnusbaeck I figured out what the problem was. It was the max open file thing. When i reduced it to 1024 it started working again. I am not sure why it was throwing a permission denied but a debug run helped identify it.