Logstash with CSV filter not creating index when LS service is started as service

[kottapar (R a V)]

Hello,

We're trying to index server details using the csv filter. The config file is as below. The weird problem we're facing is the index is created if I run logstash in foreground for testing.

COMMAND
/usr/share/logstash/vendor/jruby/bin/jruby /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb --path.settings /etc/logstash

But if I just start the server using systemctl start logstash, then the index is not created.

CONFIG

input {
  file {
    path => "/infra/elk/unixtoelk_inv/aix_inv.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    type => unixinv
  }
}

filter {
  if [type] == "unixinv" {
    csv {
      separator => ","
      columns => ["Date","Location","HostName","IP_Address","OS","OS_Version","Model","CPU_Type","Serial","Firmware","Kernel","Logical CPUs","Entitled CPUs","Vir_Min_CPU","Vir_Max_CPU","Phy_Min_CPU","Phy_Max_CPU","Capacity_Weight","Mode","Memory","Min_Mem","Max_Mem","Swap"]
    }
    date {
      match => ["Date", "ddMMYYYYHHmmss"]
    }
    mutate {
      remove_field => ["Date", "type", "path", "host", "message"]
    }
  }
}

output {
  if [type] == "unixinv" {
    elasticsearch {
      index => "inv-unix-%{+dd-MM-YYYY}"
      hosts => [ "10.1.1.1:9200" ]     
    }
  }
}

[kottapar (R a V)]

It's actually a mistake with the permissions The debug showed the below.

[2017-07-05T11:35:30,176][DEBUG][logstash.inputs.file     ] _globbed_files: /infra/elk/unixtoelk_inv/*.csv: glob is: ["/infra/elk/unixtoelk_inv/05072017_aix_inv.csv"]
[2017-07-05T11:47:29,710][DEBUG][logstash.inputs.file     ] _globbed_files: /infra/elk/unixtoelk_inv/*.csv: glob is: []
[2017-07-05T11:47:43,742][DEBUG][logstash.inputs.file     ] _globbed_files: /infra/elk/unixtoelk_inv/*.csv: glob is: [] 

the first line is when we ran it in the foreground. Then suddenly we realized that Logstash runs as "logstash" user when started using systemctl.
I changed the ownership of /infra/elk/unixtoelk_inv/aix_inv.csv to logstash and it worked.