Multiple logstash indexes to be created from multiple application log files

[ashiqab]

Hello,

Still considerably new to ELK, I have successfully set up Logstash index for one of my application log files as below conf file:

server1:/etc/logstash/conf.d # cat /etc/logstash/conf.d/accessinfologs.conf
input {
  file {
    path => "/path/accessinfologs"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
filter {
  csv {
    separator => ","
    columns => ["accessinfologs_input_timestamp","accessinfologs_script","accessinfologs_invoked_by","accessinfologs_server"]
  }
  date {
        match => [ "accessinfologs_input_timestamp", "yyyy-MM-dd-HH:mm:ss" ]
  }
}
output {
  elasticsearch {
    hosts => "http://localhost:9200"
    index => "accessinfolog"
    user => elastic
    password => passwd
  }
  stdout {}
}
server1:/etc/logstash/conf.d #

Now, I am planning to add a new index for another application log file by adding a new conf file:

server1:/etc/logstash/conf.d # cat /etc/logstash/conf.d/upgradeaccess.conf
input {
  file {
    path => "/path/upgradeaccess"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
filter {
  csv {
    separator => ","
    columns => ["upgradeaccess_server","upgradeaccess_SID","upgradeaccess_path","upgradeaccess_invoker","upgradeaccess_input_timestamp","upgradeaccess_DB_TYPE","upgradeaccess_UPGRADETYPE"]
  }
  date {
        match => [ "upgradeaccess_input_timestamp", "yyyyMMddHHmm" ]
  }
}
output {
  elasticsearch {
    hosts => "http://localhost:9200"
    index => "upgradeaccess"
    user => elastic
    password => passwd
  }
  stdout {}
}
server1:/etc/logstash/conf.d #

Although the index "upgradeaccess" got created, I could see that the data is corrupted in the indexes. There are some "upgradeaccess" data in "accessinfolog" index and vice versa.

Could you please point me to the right direction as to how to configure multiple unrelated log files to multiple indexes in Logstash?

Any additional details can be shared if needed.

Thanks in advance,
Ashiq Aboo Backer

[Badger]

@ashiqab

Could you please point me to the right direction as to how to configure multiple unrelated log files to multiple indexes in Logstash?

Do not use path.config (or -f) use pipelines.yml. See this thread.

[ashiqab]

Thank you so much for the inputs regarding pipelines.
I was able to make a solution using How to handle Multiple Config files in logstash this conditional processing. I will also try out pipelines.yml and use the easier one for our deployment.

Thanks and regards,
Ashiq