Sorry if its repeated question about parsing logs from diffrent directories and diffrent indexes

Hi

on my web server I have following configuration for access log

location : /var/log/customerA/access.log
location : /var/log/customerB/access.log
location : /var/log/customerC/access.log

So i think i need only one log stash agent(install) and config file like following ?

input {
file{
path =>"/var/log/customerA/.log"
start_position => "beginning"
}
file{
path =>"/var/log/customerB/
.log"
start_position => "beginning"
}
file{
path =>"/var/log/customerC/.log"
start_position => "beginning"
}
}
filter{
if [path]== "/var/log/customerA/
.log"
{
my grok patterns

}

else if [path] == "/var/log/customerB/*.log"
{
my grok patterns
}

else if [path] == "/var/log/customerC/*.log"
{
my grok patterns
}

output {
if [path] == "/var/log/customerA/.log"
{
elasticsearch {
hosts => "http://localhost:9200"
index => "customerA_index"
}
}
else if [path] == "/var/log/customerB/
.log"
{
elasticsearch{
hosts => "http://localhost:9200"
index => "customerB_index"
}
}
else if [path] == "/var/log/customerB/*.log"
{
elasticsearch{
hosts => "http://localhost:9200"
index => "customerC_index"
}
}
stdout {
codec => dots {}

}
}

It should work, without a problem. In order to maintain some parts easier if the different customers increase, you could use a template for the output, like so:

input { file { path => "/var/log/customerA/*.log" start_position => "beginning" add_field => { "customer" => "customerA" } } file { path => "/var/log/customerB/*.log" start_position => "beginning" add_field => { "customer" => "customerB" } } file { path => "/var/log/customerC/*.log" start_position => "beginning" add_field => { "customer" => "customerC" } } } filter { if [customer] == "customerA" { my grok patterns } else if [customer] == "customerB" { my grok patterns } else if [customer] == "customerC" { my grok patterns } } output { elasticsearch { hosts => "http://localhost:9200" index => "%{customer}_index" } stdout { codec => dots {} } }

if [path] == "/var/log/customerA/*.log"{

Well, this won't work since the path field will contain the actual path to the log file.

I suggest you set a separate field (called e.g. "customer") in each file input plugin and use that field here.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.