Sorry if its repeated question about parsing logs from diffrent directories and diffrent indexes

[rajsolanki (rajesh)]

Hi

on my web server I have following configuration for access log

location : /var/log/customerA/access.log
location : /var/log/customerB/access.log
location : /var/log/customerC/access.log

So i think i need only one log stash agent(install) and config file like following ?

input {
file{
path =>"/var/log/customerA/.log"
start_position => "beginning"
}
file{
path =>"/var/log/customerB/.log"
start_position => "beginning"
}
file{
path =>"/var/log/customerC/.log"
start_position => "beginning"
}
}
filter{
if [path]== "/var/log/customerA/.log"
{
my grok patterns

}

else if [path] == "/var/log/customerB/*.log"
{
my grok patterns
}

else if [path] == "/var/log/customerC/*.log"
{
my grok patterns
}

output {
if [path] == "/var/log/customerA/.log"
{
elasticsearch {
hosts => "http://localhost:9200"
index => "customerA_index"
}
}
else if [path] == "/var/log/customerB/.log"
{
elasticsearch{
hosts => "http://localhost:9200"
index => "customerB_index"
}
}
else if [path] == "/var/log/customerB/*.log"
{
elasticsearch{
hosts => "http://localhost:9200"
index => "customerC_index"
}
}
stdout {
codec => dots {}

}
}

[paz]

It should work, without a problem. In order to maintain some parts easier if the different customers increase, you could use a template for the output, like so:

input { file { path => "/var/log/customerA/*.log" start_position => "beginning" add_field => { "customer" => "customerA" } } file { path => "/var/log/customerB/*.log" start_position => "beginning" add_field => { "customer" => "customerB" } } file { path => "/var/log/customerC/*.log" start_position => "beginning" add_field => { "customer" => "customerC" } } } filter { if [customer] == "customerA" { my grok patterns } else if [customer] == "customerB" { my grok patterns } else if [customer] == "customerC" { my grok patterns } } output { elasticsearch { hosts => "http://localhost:9200" index => "%{customer}_index" } stdout { codec => dots {} } }

[magnusbaeck (Magnus Bäck)]

if [path] == "/var/log/customerA/*.log"{

Well, this won't work since the path field will contain the actual path to the log file.

I suggest you set a separate field (called e.g. "customer") in each file input plugin and use that field here.