HI Team,
I have some tricky challenge, Anyone can help me on this concept,
I have one restricted field and that will extract from my log. That keyword will not occur on every message. If my log contain restricted value(Group Field) i need to store that event id(DCN field Value) in cache memory/some index/some file to compare that id with up coming events. If my DCN id gets matched mean, need to add one field like restricted_data: "yes". All this operations are happen in same index.
I have tried various steps but its all helpless and i need your help to achieve this. I share my code snippet here for your reference,
Sample log(which contain the restricted data field and normal log)
2019-05-06 22:41:09.174 [WMQJCAResourceAdapter : 4] [DEBUG] [DCN 0201912617621990C] ClaimTranslatorUtil - <?xml ><cts:GroupNumber>000PE1513</cts:GroupNumber>
2019-05-06 22:41:09.174 [WMQJCAResourceAdapter : 4] [DEBUG] [DCN 0201912617621990C] VenAdjUtil - Splunk log for Request : Starts
I need to use DCN value to compare each events
input {
beats{
port => 5047
}
}
filter{
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} \[%{DATA:resource}\] \[?(?<loglevel>[a-zA-Z ]+)\] \[DCN %{DATA:dcn_id}\] %{DATA:info} - ?(?<description>[a-zA-Z0-9\n -`!@#$%^&*':\".,(){}\[\]~]+)" }
}
grok {
match => { "description" => "<cts:GroupNumber>%{DATA:grp_id}<" }
}
date {
match => [ "timestamp", "yyyy-MM-dd HH:mm:ss.SSS" ]
target => ["timestamp"]
}
if![restricted_data]
{
#translate {
# field => "dcn_id"
# destination => "restricted_data"
# override => true
# exact => true
# dictionary_path => "C:/Ganesh/ELK/Latest/check.yaml"
#}
elasticsearch {
hosts => ["localhost:9200"]
index => "validate"
user => "elastic"
password => "elastic"
query => "dcn_id:%{[dcn_id]}"
fields => { "restricted_data" => "restricted" }
}
}
if [grp_id]
{
elasticsearch {
hosts => ["localhost:9200"]
index => "restricted_data"
user => "elastic"
password => "elastic"
query => "type:restricted AND grp_number:%{[grp_id]}"
fields => { "restricted_status" => "restricted_data" }
}
# mutate{
# add_field => {"test" => "%{dcn_id}: Success"}
#
# }
# mutate{
# gsub => ["test", "[\\]", ""]
# }
}
}
output {
if "test" in [tags]{
elasticsearch {
hosts => ["http://localhost:9200"]
index => "test_log"
user => "elastic"
password => "elastic"
}
if [grp_id]
{
# file {
# codec => line { format => "%{test}"}
# path => "C:\Ganesh\ELK\Latest\check.yaml"
#}
elasticsearch {
hosts => ["http://localhost:9200"]
index => "validate"
user => "elastic"
password => "elastic"
}
}
}
stdout{
codec => rubydebug
}
}