Multiple if else problem in output in Logstash

[Varun_Mahajan (Vayne)]

Hi Team,

I am new to elastic search and logstash so i have one basic small query, Hope i will get quick reply here

I am trying to apply multiple if else in my logstash but i am failed to do it , Here's the example

output {

if [fields][log_type] == "Promotions" {

if "##ERROR CODE" in [message] {
elasticsearch {

hosts => "localhost:9200"
manage_template => false
index => "elkrft-ind-%{+YYYY.MM.dd}" 
action => "update"
document_id => "%{Correlation_Id}"
doc_as_upsert => true
}

}
else{
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "elkrft-ind-%{+YYYY.MM.dd}"
action => "update"
document_id => "%{PromotionID}"
doc_as_upsert => true
}
}

stdout { codec => rubydebug }
}
}

else part is working fine and data is getting updated in case of promotionID but when it comes to if part my data against correlation id is not getting updated but it get's inserted ,

Hope to get a quick reply. Thanks

[paz]

If I undestand this correctly, the part that's not working is

		if "##ERROR CODE" in [message] {
			elasticsearch {
                            ...
			}
		}

If so, that is probably the syntax is a bit misleading. By default the message is the preset field that holds the entire log, so unless you overwrite it that how it stays till the end.

Thing is, if "something" in [message] checks for a literal match, not a regex of any kind, so

if "##ERROR CODE" in [message] {

will work if the message is "##ERROR CODE", but will fail if the message is "##ERROR CODE foo"

You probably want the regex operator for that, like

		if [message] =~ /##ERROR CODE/ {
			elasticsearch {
                            ...
			}
		}

Check the conditionals guide for more information

Edit: As it was pointed out to me by @Badger, the in clause apparently supports substring matching (if the conditional order is value -> field, but not field -> value).
Also, the issue is apparently quite unrelated to what I said above since the document is initially inserted as you said (but I'll keep the post up). Do you get any relevant errors in Logstash?

[A_B]

I used to have If statements in the Logstash output section but have removed them all and use @metadata fields to do the same thing these days. I think it could work for you as well

I would, in the filter section set %{[@metadata][document_id]} to what you want based on whatever conditions you want and use document_id => "%{[@metadata][document_id]}" in the output section.

If that makes sense...

[Varun_Mahajan (Vayne)]

No errors get in logstash , however i will check regex and let you know

[Varun_Mahajan (Vayne)]

can you please tell in this case how metadata is used , i have no idea how to use meta data

[A_B]

This is an example from my Logstash filter section

# Adding @metadata needed for index sharding to Filebeat logs
mutate {
  copy => {
   "[fields][log_prefix]" => "[@metadata][log_prefix]"
   "[fields][log_idx]" => "[@metadata][index]"
  }
}

If you have the information already in the logs that you want to use for index naming you can assign it as above. If it is not in the logs you can statically set it with e.g. add_field.

This you can do based on conditionals.

To return to the original question, which you might still need to solve for this to work as you want it to (multiple if conditions)

You should be able to nest if statements but with the right logic that should not be needed.

If using else if doesn't give you the control you need you can give one if evaluation more than one thing to evaluate with and, or, nand, xor.

Take some time to read the documentation. I find Elastic documentation to be very well written and informative.

[Varun_Mahajan (Vayne)]

Here's my full logstash

input {

beats {
port => 5044
}
}
filter {
if [fields][log_type] == "Promotions" {
if "SOURCE MESSAGEJMS Message In" in [message] {
dissect {
mapping => {
message => "%{Timestamp} ##%{HostName}##%{Service_Name}##%{RICEF}##%{Filler1}##%{Correlation_Id}##%{Filler2}##START SERVICE MESSAGE SERVICE NAME : %{StartServiceMessage}##%{Filler3}##%{Filler4}##%{Filler5}##%{Filler6}##%{ServiceContextId}##%{UserName}##%{Filler7}##SOURCE MESSAGEJMS Message In : %{SourceMessage}" }
}
xml {
source => "SourceMessage"
store_xml => false
xpath => ["/RetailEvent/ID/text()", "input_promo_id"]
}
mutate {
remove_field => [ "SourceMessage" ]
add_field => { "txnstatus" => "SAGReceived" }
add_field => { "PromotionID" => "%{input_promo_id}" }
}
}
if "TARGET MESSAGEJMS Message Out" in [message] {
dissect {
mapping => {
message => "%{Timestamp} ##%{HostName}##%{Service_Name}##%{RICEF}##%{Filler1}##%{Correlation_Id}##%{Filler2}##END SERVICE MESSAGE - SERVICE NAME : %{EndServiceMessage}##%{Filler3}##%{Filler4}##%{Filler5}##%{Filler6}##%{ServiceContextId}##%{UserName}##%{Filler7}##TARGET MESSAGEJMS Message Out : %{targetxmlfile}"}

		}
		xml {
			source => "targetxmlfile"
			store_xml => false
			xpath => ["/ns:RetailEventReplicationRequest/RetailEvent/ID/text()", "target_promo_id"]
		}
		mutate { 
			remove_field => [ "targetxmlfile" ]
			add_field => { "txnstatus" => "SAGProcessed" }
			add_field => { "PromotionID" => "%{target_promo_id}" }
			
		}
				
	}
	if "##ERROR CODE" in [message] {
		dissect {
			mapping => {
				message => "%{Timestamp} ##%{HostName}##%{Service_Name}##%{RICEF}##%{Filler1}##%{Correlation_Id}##%{Filler2}##END SERVICE MESSAGE - SERVICE NAME : %{EndServiceMessage}##%{Filler3}##%{Filler4}##%{Filler5}##%{Filler6}##%{ServiceContextId}##%{UserName}##%{Filler7}##%{ERRORMESSAGE}"
			}
		}
		mutate { 
			add_field => { "txnstatus" => "SAGError" }
		    			}
		
	}
	}
	}

output
{

if [fields][log_type] == "Promotions"
{
if "TARGET MESSAGEJMS Message Out" in [message] or "SOURCE MESSAGEJMS Message In" in [message]
{
elasticsearch
{
hosts => "localhost:9200"
manage_template => false
index => "elkrft-ind-%{+YYYY.MM.dd}"
action => "update"
document_id => "%{PromotionID}"
doc_as_upsert => true
}
}
else
{
elasticsearch
{
hosts => "localhost:9200"
manage_template => false
index => "elkrft-ind-%{+YYYY.MM.dd}"
action => "update"
document_id => "%{Correlation_Id}"
doc_as_upsert => true
}
}

stdout { codec => rubydebug }

}
}

Questions :-

1. Tried using if [message] =~ /##ERROR CODE/ but not able to do the same
2. if "TARGET MESSAGEJMS Message Out" in [message] or "SOURCE MESSAGEJMS Message In" in [message] TRIED THIS but in case of my Error ideally it should go in else loop but logs doesn't get update in case of co-relation id , it get's inserted
   3 Summary :- There are 3 logs which i have tried to break one is "IN", one is "OUT" and third one is "ERROR" , In my IN and OUT logs i have field Promo ID and Correlation iD and i am updating WITH key Promo Id but in case of my Error Logs there is no Promo id in my logs and i am updating with key "Coorealtion ID".
   4 Not able to find meta data applied in my case , if there's any good video and documentation please help