Logstash with CSV filter not creating index when LS service is started as service


We're trying to index server details using the csv filter. The config file is as below. The weird problem we're facing is the index is created if I run logstash in foreground for testing.

/usr/share/logstash/vendor/jruby/bin/jruby /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb --path.settings /etc/logstash

But if I just start the server using systemctl start logstash, then the index is not created.


input {
  file {
    path => "/infra/elk/unixtoelk_inv/aix_inv.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    type => unixinv

filter {
  if [type] == "unixinv" {
    csv {
      separator => ","
      columns => ["Date","Location","HostName","IP_Address","OS","OS_Version","Model","CPU_Type","Serial","Firmware","Kernel","Logical CPUs","Entitled CPUs","Vir_Min_CPU","Vir_Max_CPU","Phy_Min_CPU","Phy_Max_CPU","Capacity_Weight","Mode","Memory","Min_Mem","Max_Mem","Swap"]
    date {
      match => ["Date", "ddMMYYYYHHmmss"]
    mutate {
      remove_field => ["Date", "type", "path", "host", "message"]

output {
  if [type] == "unixinv" {
    elasticsearch {
      index => "inv-unix-%{+dd-MM-YYYY}"
      hosts => [ "" ]     

It's actually a mistake with the permissions :stuck_out_tongue: The debug showed the below.

[2017-07-05T11:35:30,176][DEBUG][logstash.inputs.file     ] _globbed_files: /infra/elk/unixtoelk_inv/*.csv: glob is: ["/infra/elk/unixtoelk_inv/05072017_aix_inv.csv"]
[2017-07-05T11:47:29,710][DEBUG][logstash.inputs.file     ] _globbed_files: /infra/elk/unixtoelk_inv/*.csv: glob is: []
[2017-07-05T11:47:43,742][DEBUG][logstash.inputs.file     ] _globbed_files: /infra/elk/unixtoelk_inv/*.csv: glob is: [] 

the first line is when we ran it in the foreground. Then suddenly we realized that Logstash runs as "logstash" user when started using systemctl.
I changed the ownership of /infra/elk/unixtoelk_inv/aix_inv.csv to logstash and it worked.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.