# Facing issues on translate filter

**URL:** <https://discuss.elastic.co/t/facing-issues-on-translate-filter/180808>\
**Category:** Logstash\
**Created:** [May 13, 2019, 12:18pm UTC](https://discuss.elastic.co/t/facing-issues-on-translate-filter/180808 "2019-05-13T12:18:18Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ganesh2303](https://avatars.discourse-cdn.com/v4/letter/g/57b2e6/32.png) [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\
**Post date:** [May 13, 2019, 12:18pm UTC](https://discuss.elastic.co/t/facing-issues-on-translate-filter/180808/1 "2019-05-13T12:18:18Z")

</div>

HI Team,  
I'm using translate filter to match value with yaml, but those translate is doesnt happen,

Conf:

translate {  
field =\> "dcn\_id"  
destination =\> "restricted\_data"  
override =\> true  
dictionary\_path =\> "C:\Ganesh\ELK\Latest\check.yaml"  
}

Yaml:(I have tried with various combination)

0201912617621990C: Yes  
"0201912617621990C": "Yes"  
"0201912617621990C": Yes

Example:  
dcn\_id:0201912617621990C

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 13, 2019, 1:26pm UTC](https://discuss.elastic.co/t/facing-issues-on-translate-filter/180808/2 "2019-05-13T13:26:54Z")

</div>

```auto
0201912617621990C: Yes

```

gets me

```
"restricted_data" => true,

```

Can you try using forward slash instead of backslash in dictionary\_path?

---

<div class="post-metadata">

**Author:** ![Ganesh2303](https://avatars.discourse-cdn.com/v4/letter/g/57b2e6/32.png) [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\
**Post date:** [May 13, 2019, 1:33pm UTC](https://discuss.elastic.co/t/facing-issues-on-translate-filter/180808/3 "2019-05-13T13:33:05Z")

</div>

> [@Badger](#):
>
> "restricted\_data" =\> true,

where i have to add this line ?

inside of translate filter

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 13, 2019, 1:36pm UTC](https://discuss.elastic.co/t/facing-issues-on-translate-filter/180808/4 "2019-05-13T13:36:14Z")

</div>

No, that's the output of a rubydebug codec after the translate filter has executed and done the lookup.

---

<div class="post-metadata">

**Author:** ![Ganesh2303](https://avatars.discourse-cdn.com/v4/letter/g/57b2e6/32.png) [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\
**Post date:** [May 13, 2019, 1:39pm UTC](https://discuss.elastic.co/t/facing-issues-on-translate-filter/180808/5 "2019-05-13T13:39:27Z")

</div>

This is my full config file

```
input {
 beats{
	port => 5047
 }
}
filter{

	grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} \[%{DATA:resource}\] \[?(?<loglevel>[a-zA-Z]+)\] \[DCN %{DATA:dcn_id}\] %{DATA:info} - ?(?<description>[a-zA-Z0-9\n -`!@#$%^&*':\".,(){}\[\]~]+)" }    
  }
  grok {
match => { "description" => "<cts:GroupNumber>%{DATA:grp_id}<" }    
  }
   date {
match => ["timestamp", "yyyy-MM-dd HH:mm:ss.SSS"]
	target => ["timestamp"]
  }
   
   if![restricted_data]
   {
	translate {
		field => "dcn_id"
		destination => "restricted_data"
		override => true
		dictionary_path => "C:/Ganesh/ELK/Latest/check.yaml"
	}
   }

   if [grp_id]
   {
	   elasticsearch {
		  hosts => ["localhost:9200"]
		  index => "restricted_data"
		  query => "type:restricted AND grp_number:%{[grp_id]}"
		  fields => { "restricted_status" => "restricted_data" }
	   }
	   mutate{
			add_field => {"test" => "%{dcn_id}: Yes"}

	   }
   } 
}
output {
	if "test" in [tags]{
	  elasticsearch {
		hosts => ["http://localhost:9200"]
		index => "test_log"
	}
	if [grp_id] 
	{
		       file { 
        codec => line { format => "%{test}"}
        path => "C:\Ganesh\ELK\Latest\check.yaml" 
   }

	}
  }
}
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 13, 2019, 1:56pm UTC](https://discuss.elastic.co/t/facing-issues-on-translate-filter/180808/6 "2019-05-13T13:56:34Z")

</div>

Can you change the codec on your file output to be rubydebug and show us what an event looks like?

---

<div class="post-metadata">

**Author:** ![Ganesh2303](https://avatars.discourse-cdn.com/v4/letter/g/57b2e6/32.png) [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\
**Post date:** [May 13, 2019, 2:39pm UTC](https://discuss.elastic.co/t/facing-issues-on-translate-filter/180808/7 "2019-05-13T14:39:19Z")

</div>

Please find the stdout result for your reference

```
{
         "source" => "C:\\Ganesh\\ELK\\Latest\\hsbclogs\\VendoreAdj.log",
      "timestamp" => 2019-05-06T17:11:09.184Z,
          "input" => {
        "type" => "log"
    },
       "resource" => "xxxx: 4",
            "log" => {
        "file" => {
            "path" => "C:\\Ganesh\\ELK\\Latest\\hsbclogs\\VendoreAdj.log"
        }
    },
       "@version" => "1",
     "@timestamp" => 2019-05-13T14:38:17.256Z,
           "tags" => [
        [0] "test",
        [1] "beats_input_codec_plain_applied",
        [2] "_grokparsefailure"
    ],
           "info" => "MessageSenderTemplate",
       "loglevel" => "DEBUG",
           "beat" => {
            "name" => "xxx",
        "hostname" => "xxx",
         "version" => "6.7.0"
    },
        "message" => "2019-05-06 22:41:09.184 [ResourceAdapter : 4] [DEBUG] [DCN 0201912617621990C] MessageSenderTemplate - ResponseProducer Generated Message:",
         "offset" => 102299,
     "prospector" => {
        "type" => "log"
    },
           "host" => {
        "architecture" => "x86_64",
                  "os" => {
              "family" => "windows",
            "platform" => "windows",
                "name" => "Windows 10 Enterprise",
             "version" => "10.0",
               "build" => "17763.437"
        },
                  "id" => "0245ced2-6c59-41aa-9f75-7a2bd7aadfed",
                "name" => "xxx"
    },
         "dcn_id" => "0201912617621990C",
    "description" => "ResponseProducer Generated Message:"
}
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 13, 2019, 2:54pm UTC](https://discuss.elastic.co/t/facing-issues-on-translate-filter/180808/8 "2019-05-13T14:54:07Z")

</div>

> [@Ganesh2303](#):
>
> ```
> "message" => "2019-05-06 22:41:09.184 [ResourceAdapter : 4] [DEBUG] [DCN 0201912617621990C] MessageSenderTemplate - ResponseProducer Generated Message:",
> 
> ```

When I run that message through that configuration I do get the restricted\_data field added to the message. Are you sure you have a matching entry in check.yaml?

---

<div class="post-metadata">

**Author:** ![Ganesh2303](https://avatars.discourse-cdn.com/v4/letter/g/57b2e6/32.png) [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\
**Post date:** [May 14, 2019, 8:02am UTC](https://discuss.elastic.co/t/facing-issues-on-translate-filter/180808/9 "2019-05-14T08:02:49Z")

</div>

> [@Badger](#):
>
> When I run that message through that configuration I do get the restricted\_data field added to the message. Are you sure you have a matching entry in check.yaml?

yes i've match field in my yaml file.

---

<div class="post-metadata">

**Author:** ![Ganesh2303](https://avatars.discourse-cdn.com/v4/letter/g/57b2e6/32.png) [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\
**Post date:** [May 14, 2019, 10:06am UTC](https://discuss.elastic.co/t/facing-issues-on-translate-filter/180808/10 "2019-05-14T10:06:30Z")

</div>

> [@Badger](#):
>
> When I run that message through that configuration I do get the restricted\_data field added to the message. Are you sure you have a matching entry in check.yaml?

My concept is in my message i ll get one grp id once if i find the grp id i write the dcn value into my yaml like this,  
0201912617621990C: Yes

after that when ever this dcn id comes in my log , i want to write yes value into restricted\_data field using translate condition but it doesnt happen why

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 14, 2019, 2:06pm UTC](https://discuss.elastic.co/t/facing-issues-on-translate-filter/180808/11 "2019-05-14T14:06:40Z")

</div>

The problem is that you read the dictionary, then process events and update the dictionary. Due to batching and caching by both ruby and the filesystem, there may be a considerable delay in updating the dictionary. I think you need an in-memory persistent cache, and that can be done using an aggregate filter.

This might work

```
    aggregate {
        task_id => "%{dcn_id}"
        code => '
            if ! map["seen"]
                map["seen"] = true
            else
                event.set("restrictedData", true)
            end
        '
        aggregate_maps_path => "/home/user/foo.maps"
        timeout => 3600 # Expire entries after 1 hour
    }

```

Note that you must have '--pipeline.workers 1' set for whichever pipeline runs this.

---

<div class="post-metadata">

**Author:** ![Ganesh2303](https://avatars.discourse-cdn.com/v4/letter/g/57b2e6/32.png) [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\
**Post date:** [May 14, 2019, 2:55pm UTC](https://discuss.elastic.co/t/facing-issues-on-translate-filter/180808/12 "2019-05-14T14:55:23Z")

</div>

> [@Badger](#):
>
> aggregate { task\_id =\> "%{dcn\_id}" code =\> ' if ! map["seen"] map["seen"] = true else event.set("restrictedData", true) end ' aggregate\_maps\_path =\> "/home/user/foo.maps" timeout =\> 3600 # Expire entries after 1 hour }

Am i using correct in my conf

```
input {
 beats{
	port => 5047
 }
}
filter{

	grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} \[%{DATA:resource}\] \[?(?<loglevel>[a-zA-Z]+)\] \[DCN %{DATA:dcn_id}\] %{DATA:info} - ?(?<description>[a-zA-Z0-9\n -`!@#$%^&*':\".,(){}\[\]~]+)" }    
  }
  grok {
    match => { "description" => "<cts:GroupNumber>%{DATA:grp_id}<" }    
  }
   date {
    match => ["timestamp", "yyyy-MM-dd HH:mm:ss.SSS"]
	target => ["timestamp"]
  }
   
   if![restricted_data]
   {
	aggregate {
    task_id => "%{dcn_id}"
    code => '
        if ! map["seen"]
            map["seen"] = true
        else
            event.set("restrictedData", true)
        end
    '
    aggregate_maps_path => "/home/user/foo.maps"
    timeout => 3600 # Expire entries after 1 hour
}
   }
    
   if [grp_id]
   {
	   elasticsearch {
		  hosts => ["localhost:9200"]
		  index => "restricted_data"
		  query => "type:restricted AND grp_number:%{[grp_id]}"
		  fields => { "restricted_status" => "restricted_data" }
	   }
	   mutate{
			add_field => {"test" => "%{dcn_id}: Success"}

	   }
	   mutate{
			gsub => ["test", "[\\]", ""]
	   }
   } 
}
output {
	if "test" in [tags]{
	  elasticsearch {
		hosts => ["http://localhost:9200"]
		index => "test_log"
	}
	if [grp_id] 
	{
		       file { 
            codec => line { format => "%{test}"}
            path => "C:\Ganesh\ELK\Latest\check.yaml" 
       }

	}	
  }
  stdout{
	codec => rubydebug
	}
}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 14, 2019, 2:59pm UTC](https://discuss.elastic.co/t/facing-issues-on-translate-filter/180808/13 "2019-05-14T14:59:03Z")

</div>

That looks reasonable.

---

<div class="post-metadata">

**Author:** ![Ganesh2303](https://avatars.discourse-cdn.com/v4/letter/g/57b2e6/32.png) [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\
**Post date:** [May 14, 2019, 3:05pm UTC](https://discuss.elastic.co/t/facing-issues-on-translate-filter/180808/14 "2019-05-14T15:05:58Z")

</div>

i'm using above configuration, i'm getting aggregation exception ☹

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 11, 2019, 3:15pm UTC](https://discuss.elastic.co/t/facing-issues-on-translate-filter/180808/15 "2019-06-11T15:15:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.