Need to compare a field with other field using logstash

[Ganesh2303 (Ganesh)]

HI Team,
I have one scenario can anyone suggest me solution for that,

My scenario is i have one sample which contain some dcn id and some more fields like this

2019-05-17 [DCN 001] - started
2019-05-17 [DCN 001] - processing,

in some point my log will contain group id and if i found the group id in my log that dcn id is restricted value.

2019-05-17 [DCN 001] <grp_number>P001</grp_number>- started

So 001 id is restricted.

so i need to compare that restricted id into upcoming events in that index. how can i achieve that.

[pastechecker (Charlie)]

@Ganesh2303

in some point my log will contain group id and if i found the group id in my log that dcn id

Please also share some code snippet you have right now for us to come with some suggestions / improvements.

From what you have written and my understanding is:

• you have logs coming to your system and you somehow parse them
• you want to have a field lookup for the past value if existed and do some action

You can accomplish that by placing in a filter section elasticsearch {}, execute a query and work with the return value. Make sure that you control the ID of the data insertion as well that could allow you to do the proper data lookup.

Side note:
Could you put next time at least minimal effort into writing properly your question that follows some logical order? I am not a English native speaker but at least obeying some basic grammar rules would be great from respect reasons.

[Ganesh2303 (Ganesh)]

filter{
	grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} \[%{DATA:resource}\] \[?(?<loglevel>[a-zA-Z ]+)\] \[DCN %{DATA:dcn_id}\] %{DATA:info} - ?(?<description>[a-zA-Z0-9\n -`!@#$%^&*':\".,(){}\[\]~]+)" }    
  }
  grok {
    match => { "description" => "<cts:GroupNumber>%{DATA:grp_id}<" }    
  }
   date {
    match => [ "timestamp", "yyyy-MM-dd HH:mm:ss.SSS" ]
	target => ["timestamp"]
  }
   
   if![restricted_data]
   {
	
	 elasticsearch {
		  hosts => ["localhost:9200"]
		  index => "validate"
		  user => "elastic"
		  password => "elastic"
		  query => "dcn_id:%{[dcn_id]}"
		  fields => { "restricted_data" => "restricted" }
	   }
	}
    
   if [grp_id]
   {
	   elasticsearch {
		  hosts => ["localhost:9200"]
		  index => "restricted_data"
		  user => "elastic"
		  password => "elastic"
		  query => "type:restricted AND grp_number:%{[grp_id]}"
		  fields => { "restricted_status" => "restricted_data" }
	   }
	 
   } 
}
output {
	if "test" in [tags]{
	  elasticsearch {
		hosts => ["http://localhost:9200"]
		index => "test_log"
		user => "elastic"
		password => "elastic"
	}
	if [grp_id] 
	{
		
		elasticsearch {
		hosts => ["http://localhost:9200"]
		index => "validate"
		user => "elastic"
		password => "elastic"
	}

	}	
  }
  stdout{
	codec => rubydebug
	}
}

Please find my code snippet for your kind reference and sorry about my prev message. All my insertion and comparing is happening on same index.