Grokk Patterning output to respective indexes based on [agent.type] (or any conditionals) not working

[Rohan-boogeyman (Rohan)]

Hello,

Sorry but i've got a question as I have searched alot for an answer but could not find anything.
My winlogbeat and filebeat are all sent to logstash on 5044. I want logstash to filter using Grokk and output them to dedicated indexes. I am using the following config for logstash but it appears the condition if [agent][type] == "filebeat" is not working. Please tell me what have I got wrong, thanks!

Logstash Config

input {
beats {
port => 5044
}
}

filter {
if [agent][type] == "filebeat"{
if "404" in [message] {
grok {
match => {"message" => "%{TIMESTAMP_ISO8601:Grokkstamp} %{WORD:Grokkiis} %{WORD:Grokksite} %{IP:GrokkclientIP} %{WORD:Grokkhttp} %{GREEDYDATA:Grokkurl} %{NUMBER:Grokkport} %{NOTSPACE:Grokkusername} %{IPORHOST:Grokkclienthost} %{NOTSPACE:Grokkuseragent} %{NOTSPACE:Grokkreferer} %{DATA:Grokkhosturl} %{NUMBER:Grokkresponse} %{NUMBER:Grokksubresponse} %{NUMBER:Grokkscstatus} %{NUMBER:Grokkint}"}
}
}
else if "403" in [message] {
grok {
match => {"message" => "%{TIMESTAMP_ISO8601:Grokkstamp} %{WORD:Grokkiis} %{WORD:Grokksite} %{IP:GrokkclientIP} %{WORD:Grokkhttp} %{GREEDYDATA:Grokkurl} %{NUMBER:Grokkport} %{NOTSPACE:Grokkusername} %{IPORHOST:Grokkclienthost} %{NOTSPACE:Grokkuseragent} %{NOTSPACE:Grokkreferer} %{DATA:Grokkhosturl} %{NUMBER:Grokkresponse} %{NUMBER:Grokksubresponse} %{NUMBER:Grokkscstatus} %{NUMBER:Grokkint}"}
}
}
else if "400" in [message] {
grok {
match => {"message" => "%{TIMESTAMP_ISO8601:Grokkstamp} %{WORD:Grokkiis} %{WORD:Grokksite} %{IP:GrokkclientIP} %{WORD:Grokkhttp} %{GREEDYDATA:Grokkurl} %{NUMBER:Grokkport} %{NOTSPACE:Grokkusername} %{IPORHOST:Grokkclienthost} %{NOTSPACE:Grokkuseragent} %{NOTSPACE:Grokkreferer} %{DATA:Grokkhosturl} %{NUMBER:Grokkresponse} %{NUMBER:Grokksubresponse} %{NUMBER:Grokkscstatus} %{NUMBER:Grokkint}"}
}
}
else if "500" in [message] {
grok {
match => {"message" => "%{TIMESTAMP_ISO8601:Grokkstamp} %{WORD:Grokkiis} %{WORD:Grokksite} %{IP:GrokkclientIP} %{WORD:Grokkhttp} %{GREEDYDATA:Grokkurl} %{NUMBER:Grokkport} %{NOTSPACE:Grokkusername} %{IPORHOST:Grokkclienthost} %{NOTSPACE:Grokkuseragent} %{NOTSPACE:Grokkreferer} %{DATA:Grokkhosturl} %{NUMBER:Grokkresponse} %{NUMBER:Grokksubresponse} %{NUMBER:Grokkscstatus} %{NUMBER:Grokkint}"}
}
}
else {
}
}

else if [agent][type] == "winlogbeat"{
if "Service: THP" in [message] {
grok {
match => {"message" => "%{GREEDYDATA:message1} %{WORD:error}"}
}
}
else {
}
}
}

output
{
if [agent][type] == "filebeat"{
if "dd00-sap-iislogs" in [tags]{
elasticsearch
{
hosts => "${xxxx}"
user => "${xxxx}"
password => "${xxxx}"
index => "n1o0-rbptet-sre-windows-filebeat-%{+YYYY.MM}"
manage_template => false
ssl => true
ssl_certificate_verification => false
cacert => "/xxx_xxx/xxx/xxx-xxx.crt"
ilm_rollover_alias => "xxx-xxx-xxx-windows-filebeat"
ilm_pattern => "000001"
ilm_policy => "xxx-xxx-xxx-xxx"
}
}
}
else if [agent][type] == "winlogbeat"]{
elasticsearch
{
hosts => "${xxxx}"
user => "${xxxx}"
password => "${xxxx}"
index => "xxx-xxx-xxx-windows-winlogbeat-%{+YYYY.MM}"
manage_template => false
ssl => true
ssl_certificate_verification => false
cacert => "/xxx_xxx/xxx/xxx-xxx.crt"
ilm_rollover_alias => "xxx-xxx-xxx-windows-winlogbeat"
ilm_pattern => "000001"
ilm_policy => "xxx-xxx-xxx-xxx"
}
}
}

[A_B]

Hi @Rohan-boogeyman,

would it be possible for you to format your config using Preformated text instead of Block quote. It would be much easier to read

Which version of Filebeat are you using? Can't see anything immediately wrong with the if [agent][type] == "filebeat" bit.

What is the result now? Is else if [agent][type] == "winlogbeat" working as expected?