Facing issues on translate filter

Ganesh2303 · May 13, 2019, 12:18pm

HI Team,
I'm using translate filter to match value with yaml, but those translate is doesnt happen,

Conf:

translate {
field => "dcn_id"
destination => "restricted_data"
override => true
dictionary_path => "C:\Ganesh\ELK\Latest\check.yaml"
}

Yaml:(I have tried with various combination)

0201912617621990C: Yes
"0201912617621990C": "Yes"
"0201912617621990C": Yes

Example:
dcn_id:0201912617621990C

Badger · May 13, 2019, 1:26pm

0201912617621990C: Yes

gets me

"restricted_data" => true,

Can you try using forward slash instead of backslash in dictionary_path?

Ganesh2303 · May 13, 2019, 1:33pm

where i have to add this line ?

inside of translate filter

Badger · May 13, 2019, 1:36pm

No, that's the output of a rubydebug codec after the translate filter has executed and done the lookup.

Ganesh2303 · May 13, 2019, 1:39pm

This is my full config file

input {
 beats{
	port => 5047
 }
}
filter{

	grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} \[%{DATA:resource}\] \[?(?<loglevel>[a-zA-Z ]+)\] \[DCN %{DATA:dcn_id}\] %{DATA:info} - ?(?<description>[a-zA-Z0-9\n -`!@#$%^&*':\".,(){}\[\]~]+)" }    
  }
  grok {
match => { "description" => "<cts:GroupNumber>%{DATA:grp_id}<" }    
  }
   date {
match => [ "timestamp", "yyyy-MM-dd HH:mm:ss.SSS" ]
	target => ["timestamp"]
  }
   
   if![restricted_data]
   {
	translate {
		field => "dcn_id"
		destination => "restricted_data"
		override => true
		dictionary_path => "C:/Ganesh/ELK/Latest/check.yaml"
	}
   }

   if [grp_id]
   {
	   elasticsearch {
		  hosts => ["localhost:9200"]
		  index => "restricted_data"
		  query => "type:restricted AND grp_number:%{[grp_id]}"
		  fields => { "restricted_status" => "restricted_data" }
	   }
	   mutate{
			add_field => {"test" => "%{dcn_id}: Yes"}

	   }
   } 
}
output {
	if "test" in [tags]{
	  elasticsearch {
		hosts => ["http://localhost:9200"]
		index => "test_log"
	}
	if [grp_id] 
	{
		       file { 
        codec => line { format => "%{test}"}
        path => "C:\Ganesh\ELK\Latest\check.yaml" 
   }

	}
  }
}

Badger · May 13, 2019, 1:56pm

Can you change the codec on your file output to be rubydebug and show us what an event looks like?

Ganesh2303 · May 13, 2019, 2:39pm

Please find the stdout result for your reference

{
         "source" => "C:\\Ganesh\\ELK\\Latest\\hsbclogs\\VendoreAdj.log",
      "timestamp" => 2019-05-06T17:11:09.184Z,
          "input" => {
        "type" => "log"
    },
       "resource" => "xxxx: 4",
            "log" => {
        "file" => {
            "path" => "C:\\Ganesh\\ELK\\Latest\\hsbclogs\\VendoreAdj.log"
        }
    },
       "@version" => "1",
     "@timestamp" => 2019-05-13T14:38:17.256Z,
           "tags" => [
        [0] "test",
        [1] "beats_input_codec_plain_applied",
        [2] "_grokparsefailure"
    ],
           "info" => "MessageSenderTemplate",
       "loglevel" => "DEBUG",
           "beat" => {
            "name" => "xxx",
        "hostname" => "xxx",
         "version" => "6.7.0"
    },
        "message" => "2019-05-06 22:41:09.184 [ResourceAdapter : 4] [DEBUG] [DCN 0201912617621990C] MessageSenderTemplate - ResponseProducer Generated Message:",
         "offset" => 102299,
     "prospector" => {
        "type" => "log"
    },
           "host" => {
        "architecture" => "x86_64",
                  "os" => {
              "family" => "windows",
            "platform" => "windows",
                "name" => "Windows 10 Enterprise",
             "version" => "10.0",
               "build" => "17763.437"
        },
                  "id" => "0245ced2-6c59-41aa-9f75-7a2bd7aadfed",
                "name" => "xxx"
    },
         "dcn_id" => "0201912617621990C",
    "description" => "ResponseProducer Generated Message:"
}

Badger · May 13, 2019, 2:54pm

When I run that message through that configuration I do get the restricted_data field added to the message. Are you sure you have a matching entry in check.yaml?

Ganesh2303 · May 14, 2019, 8:02am

yes i've match field in my yaml file.

Ganesh2303 · May 14, 2019, 10:06am

My concept is in my message i ll get one grp id once if i find the grp id i write the dcn value into my yaml like this,
0201912617621990C: Yes

after that when ever this dcn id comes in my log , i want to write yes value into restricted_data field using translate condition but it doesnt happen why

Badger · May 14, 2019, 2:06pm

The problem is that you read the dictionary, then process events and update the dictionary. Due to batching and caching by both ruby and the filesystem, there may be a considerable delay in updating the dictionary. I think you need an in-memory persistent cache, and that can be done using an aggregate filter.

This might work

    aggregate {
        task_id => "%{dcn_id}"
        code => '
            if ! map["seen"]
                map["seen"] = true
            else
                event.set("restrictedData", true)
            end
        '
        aggregate_maps_path => "/home/user/foo.maps"
        timeout => 3600 # Expire entries after 1 hour
    }

Note that you must have '--pipeline.workers 1' set for whichever pipeline runs this.

Ganesh2303 · May 14, 2019, 2:55pm

Am i using correct in my conf

input {
 beats{
	port => 5047
 }
}
filter{

	grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} \[%{DATA:resource}\] \[?(?<loglevel>[a-zA-Z ]+)\] \[DCN %{DATA:dcn_id}\] %{DATA:info} - ?(?<description>[a-zA-Z0-9\n -`!@#$%^&*':\".,(){}\[\]~]+)" }    
  }
  grok {
    match => { "description" => "<cts:GroupNumber>%{DATA:grp_id}<" }    
  }
   date {
    match => [ "timestamp", "yyyy-MM-dd HH:mm:ss.SSS" ]
	target => ["timestamp"]
  }
   
   if![restricted_data]
   {
	aggregate {
    task_id => "%{dcn_id}"
    code => '
        if ! map["seen"]
            map["seen"] = true
        else
            event.set("restrictedData", true)
        end
    '
    aggregate_maps_path => "/home/user/foo.maps"
    timeout => 3600 # Expire entries after 1 hour
}
   }
    
   if [grp_id]
   {
	   elasticsearch {
		  hosts => ["localhost:9200"]
		  index => "restricted_data"
		  query => "type:restricted AND grp_number:%{[grp_id]}"
		  fields => { "restricted_status" => "restricted_data" }
	   }
	   mutate{
			add_field => {"test" => "%{dcn_id}: Success"}

	   }
	   mutate{
			gsub => ["test", "[\\]", ""]
	   }
   } 
}
output {
	if "test" in [tags]{
	  elasticsearch {
		hosts => ["http://localhost:9200"]
		index => "test_log"
	}
	if [grp_id] 
	{
		       file { 
            codec => line { format => "%{test}"}
            path => "C:\Ganesh\ELK\Latest\check.yaml" 
       }

	}	
  }
  stdout{
	codec => rubydebug
	}
}

Badger · May 14, 2019, 2:59pm

That looks reasonable.

Ganesh2303 · May 14, 2019, 3:05pm

i'm using above configuration, i'm getting aggregation exception

system · June 11, 2019, 3:15pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.