Someone please clarify me on my configuration

[Raja1]

Hi All,

I am getting bit confusion on type,field. Could you please someone confirm me by checking below configuration.

My intention is to create the index by environment wise and remove some fields from the output. Please have a look and clarify me if any mistakes.

input
{
file
{
path => "/etc/logs/POClogs/.log"
add_field => {"environment": "POC"}
start_position => "beginning"
}
{
path => "//etc/logstash/Devlogs/.log"
add_field => {"environment": "DEV"}
start_position => "beginning"
}
}
filter
{
grok
{
match => {"message" => "[%WARNING,%INFO]%{GREEDYDATA}[%WARNING]"}
add_field => { "ApplicationName" => "Devapp" }
add_field => { "ApplicationName" => "POCapp" }
remove_tag => ["_grokparsefailure"]
remove_field => ["clientHostname"]
remove_field => ["_type"]
remove_field => ["_score"]
}
}
output
{
if [type] == "Devlog"
{
elasticsearch
{
hosts => ["x.x.x.x:9200"]
index => "Devlog-%{+YYYY.MM.dd}"
}
}
else if [type] == "POClog"
{
elasticsearch
{
hosts => ["x.x.x.x:9200"]
index => "POClog-%{+YYYY.MM.dd}"
}
}
else
{
elasticsearch
{
hosts => ["x.x.x.x:9200"]
}
stdout { codec => rubydebug }
}
}

Regards
Raja

[magnusbaeck (Magnus Bäck)]

I don't see you ever setting the type field to anything so your conditions won't work as expected.

add_field => {"environment": "POC"}

Wrong syntax; use => and not :.

grok
{
match => {"message" => "[%WARNING,%INFO]%{GREEDYDATA}[%WARNING]"}
add_field => { "ApplicationName" => "Devapp" }
add_field => { "ApplicationName" => "POCapp" }
remove_tag => ["_grokparsefailure"]
remove_field => ["clientHostname"]
remove_field => ["_type"]
remove_field => ["_score"]
}

What's the point of this filter? You're not capturing any fields in the grok expression and you're deleting a number of fields that'll never exist at that point in the pipeline.

[Raja1]

Thank you magnus.

I did the changes in my configuration and now it is working as expected.
But i have different folders having logs so how would i configure and also how can i make the index for each.

Ex: /etc/dev/.log
/etc/uat/.log
/etc/int/*.log

I want to set the index for each folder location so that i can able to find environment wise logs. Write now my configuration for only one file path like below. Please guide me to complete this.

input
{
file
{
path => ["/etc/Dev/*.log"]
codec => multiline {
pattern => "[%WARNING,%INFO]%{GREEDYDATA}[%WARNING]"
negate => "true"
what => "previous"
}
sincedb_path => "/dev/null"
start_position => beginning
ignore_older => 0
}
}
filter
{
grok
{
match => {"message" => "[%WARNING,%INFO]%{GREEDYDATA}[%WARNING]"}
}
}
output
{
elasticsearch
{
hosts => ["x.x.x.x:9200"]
action => "index"
}
stdout {codec => rubydebug }
}

[magnusbaeck (Magnus Bäck)]

I want to set the index for each folder location so that i can able to find environment wise logs.

One index per environment is most likely a bad idea. If you're doing this only to be able to distinguish different kinds of logs just use another field for that. Like the environment field in the configuration you posted previously.

Write now my configuration for only one file path like below.

Why not follow the pattern from the previous post? Apart from what I wrote it looked fine.

grok
{
match => {"message" => "[%WARNING,%INFO]%{GREEDYDATA}[%WARNING]"}
}

Again, what's the purpose of this filter?

[Raja1]

Sorry it should be removed from my conf.

Actually my logs are starting with some status message like WARNING,INFO,FATAL i.e., i used multiline pattern.

How could i run mutiple conf files in single shot. Will that recommended to create multiple configuration files?

[Raja1]

I am getting below error after adding the add_field. Please help me to resolve.

Cannot load an invalid configuration {:reason=>"Expected one of #, => at line 7, column 36 (byte 124) after input\n{\nfile\n{\n path => ["/etc/logstash/INTlogs/*.log"]\n type => "INTlog"\n add_field => {"environment"", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:47:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:139:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:277:in create_pipeline'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:95:inregister_pipeline'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:264:in execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:inrun'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:183:in run'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:inrun'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in `(root)'"]}

[Raja1]

resolved this issue...small mistake in configuration.

Now i need to know executing multiple configuration files.