Logstash - Configuration

[Sunillinus (Sunil)]

Hi,
I am using elasticsearch, kibana and logstash for log analysis. All are of 6.2.4 versions.

My Question is :
I have different log files for a different purpose in one folder. Let's say "Sample" folder.
Under Sample folder, I have log files as shown below
D:/Sample/RS20180513.log
D:/Sample/OS20180513.log
D:/Sample/RS20180514.log
D:/Sample/OS20180514.log
D:/Sample/RS20180515.log
D:/Sample/OS20180515.log
etc..

All log filenames are created with a particular date.

How should be the configuration file so that when I run logstash from command prompt, it should create different indexes with the name as log filename for each log file in kibana.

[Dvikas (Vikas)]

r u not using filebeat?

[Sunillinus (Sunil)]

No I'm not using filebeat.

[magnusbaeck (Magnus Bäck)]

Use a grok or dissect filter to extract the filename from the field containing the input filename (source, is it?), examples have been posted in the past, then reference that field in the index option of your elasticsearch output.

[Sunillinus (Sunil)]

Hi magnus,

Actually I'm new to this ELK. Here is my configuration file. Can you please edit this configuration to match my requirement.

input {
file {
path => ["D:/Sample/OS*"]
start_position => "beginning"
sincedb_path => "/dev/null"
}
file{
path => ["D:/Sample/RS*"]
start_position => "beginning"
sincedb_path => "/dev/null"
}
}

output {
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash-%{+YYYY.MM.dd}"
}
stdout {}
}

[Sunillinus (Sunil)]

Hi @magnusbaeck

Can you please help me on how to achieve this.

[magnusbaeck (Magnus Bäck)]

I don't have time to write a config from scratch, but I can help iron out minor problems with a reasonably complete configuration.