# Huge dictionary in logstash translate filter **URL:** \ **Category:** Logstash\ **Created:** [October 30, 2020, 7:27pm UTC](https://discuss.elastic.co/t/huge-dictionary-in-logstash-translate-filter/253857 "2020-10-30T19:27:10Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![heric](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/heric/32/70748_2.png) [@heric](https://discuss.elastic.co/u/heric)\ **Post date:** [October 30, 2020, 7:27pm UTC](https://discuss.elastic.co/t/huge-dictionary-in-logstash-translate-filter/253857/1 "2020-10-30T19:27:10Z")
I have huge dictionary file to be used in logstash translate filter , around 180k entries. I tried to reduce it to around 80k but still always getting error when starting the pipeline , as below : ```auto [2020-10-30T22:11:47,115][ERROR][logstash.agent] Failed to execute action {:id=>:"dump-subsc", :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil} ``` Translate filter in logstash pipeline as below : ```auto translate { field => "[imeiTac]" destination => "[deviceName]" dictionary_path => "/usr/share/logstash/pipeline/imei_tac.csv" fallback => "unknown device" refresh_interval => 0 } ``` Below are sample of the dictionary entries: ```auto "01124500","iPhone A1203" "01130000","iPhone A1203" "01130100","iPhone A1203" "01136400","iPhone A1203" "01136500","iPhone A1203" "01143400","iPhone A1203" "01147200","iPhone A1203" "01154600","iPhone-A1203" "01161200","iPhone 3G A1241" "01161300","iPhone 3G A1241" "01161400","iPhone 3G A1241" "01165400","iPhone-A1203" "01171200","iPhone 3G A1241" "01171300","iPhone 3G A1241" "01171400","iPhone 3G A1241" "01174200","iPhone 3G A1241" ``` I have tried to use around 5k entries only in dictionary , pipeline is working fine and i can get the result. However when i added full dictionary ( around 80k entries ) pipeline is not started. I have tried to increase the JVM from 1GB , 2GB to 4GB without success. According to logstash translate filter documentation, it has been tested with around 100k dictionary entries. How can i make my pipeline working with this huge dictionary entries ? Thanks, ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 30, 2020, 8:28pm UTC](https://discuss.elastic.co/t/huge-dictionary-in-logstash-translate-filter/253857/2 "2020-10-30T20:28:09Z")
Enable log.level debug and see if you get a more informative error message than that "Failed to execute action". ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [October 30, 2020, 9:10pm UTC](https://discuss.elastic.co/t/huge-dictionary-in-logstash-translate-filter/253857/3 "2020-10-30T21:10:59Z")
What is the size of the dictionary file? Try increasing the number of entries until it stops working. Go from 5k to 10k, then 15k, 20k until it fails. I had a similar problem a couple of years ago, I've had a huge dictionary and logstash took to long to start, blocking the pipeline, and the same thing happened during the scheduled refresh. To solve this, instead of the translate filter I used the [memcached](https://www.elastic.co/guide/en/logstash/current/plugins-filters-memcached.html) filter and stored my dictionary in a memcached server. ---
**Author:** ![heric](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/heric/32/70748_2.png) [@heric](https://discuss.elastic.co/u/heric)\ **Post date:** [October 31, 2020, 9:52am UTC](https://discuss.elastic.co/t/huge-dictionary-in-logstash-translate-filter/253857/4 "2020-10-31T09:52:17Z")
Thank you , i found the problem after activating debug. The dictionary file is not sanitized properly , there are some error in the middle of the file. ```auto [2020-10-31T11:59:46,942][DEBUG][logstash.javapipeline][dump-subsc] Pipeline terminated by worker error {:pipeline_id=>"dump-subsc", :exception=>#, :backtrace=>["uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/csv.rb:1927:in `block in shift'" ``` Which is looks like below: ```auto "35166808","PIXI 4 7" 4G android" "35168607","DEXP Ixion ES2 5"" "35218207","DEXP Ixion M 4"" "35249807","DEXP Ixion ML2 5"" "35288007","DEXP Ixion ML 4.5"" "35295808","POP4 6" 4G android" "35296308","POP4 6" 4G android" "35296408","POP4 6" 4G android" ``` After fixing it , pipeline is working. ---
**Author:** ![heric](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/heric/32/70748_2.png) [@heric](https://discuss.elastic.co/u/heric)\ **Post date:** [October 31, 2020, 9:55am UTC](https://discuss.elastic.co/t/huge-dictionary-in-logstash-translate-filter/253857/5 "2020-10-31T09:55:19Z")
size of dictionary is around 4MB. I found the error after activation debug as suggested by @Badger I tried with around 68k entries right now and will increase to full 180k Thanks for your suggestion about memcached @leandrojmp , i will experiment with that later if this translate filter is causing some slowness / delay in the throughput. ---
**Author:** ![ylasri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ylasri/32/86120_2.png) [@ylasri](https://discuss.elastic.co/u/ylasri)\ **Post date:** [October 31, 2020, 9:57am UTC](https://discuss.elastic.co/t/huge-dictionary-in-logstash-translate-filter/253857/6 "2020-10-31T09:57:35Z")
You may consider using also a [ingest pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest-enriching-data.html) directly in elasticsearch for a huge dump (like subscribers infos) ---
**Author:** ![heric](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/heric/32/70748_2.png) [@heric](https://discuss.elastic.co/u/heric)\ **Post date:** [October 31, 2020, 10:07am UTC](https://discuss.elastic.co/t/huge-dictionary-in-logstash-translate-filter/253857/7 "2020-10-31T10:07:57Z")
Thank you @ylasri seems many things to be explored 🙂 ---
**Author:** ![ylasri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ylasri/32/86120_2.png) [@ylasri](https://discuss.elastic.co/u/ylasri)\ **Post date:** [October 31, 2020, 10:08am UTC](https://discuss.elastic.co/t/huge-dictionary-in-logstash-translate-filter/253857/8 "2020-10-31T10:08:51Z")
Yes, all depend on how frequently dictionnary data will be updated 🙂 ---
**Author:** ![heric](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/heric/32/70748_2.png) [@heric](https://discuss.elastic.co/u/heric)\ **Post date:** [November 2, 2020, 3:43pm UTC](https://discuss.elastic.co/t/huge-dictionary-in-logstash-translate-filter/253857/9 "2020-11-02T15:43:46Z")
Just to update with the solution for this problem, i was able to load 180k in translate dictionary, however it is not giving me consistent result, many key values that exist in the dictionary but logstash is giving me "unknown device" ( fallback value ). I ended up setting memcached and using it instead of dictionary and it is working perfectly. i indexed around 4M records and adding 1 field from memcached and it took less than 30min. ---
**Author:** ![Peter\_Nelissen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/peter_nelissen/32/78587_2.png) [@Peter\_Nelissen](https://discuss.elastic.co/u/Peter_Nelissen)\ **Post date:** [November 6, 2020, 6:43pm UTC](https://discuss.elastic.co/t/huge-dictionary-in-logstash-translate-filter/253857/10 "2020-11-06T18:43:06Z")
What do you mean? When would you use translate and when ingest pipeline? The translate filter has a nice refresh\_interval option, so that looks good if the dictionary often changes. ---
**Author:** ![Peter\_Nelissen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/peter_nelissen/32/78587_2.png) [@Peter\_Nelissen](https://discuss.elastic.co/u/Peter_Nelissen)\ **Post date:** [November 6, 2020, 6:51pm UTC](https://discuss.elastic.co/t/huge-dictionary-in-logstash-translate-filter/253857/11 "2020-11-06T18:51:17Z")
@heric Can you share the config needed for this memcache, given your dictionary in /usr/share/logstash/pipeline/imei\_tac.csv ? Thx! ---
**Author:** ![heric](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/heric/32/70748_2.png) [@heric](https://discuss.elastic.co/u/heric)\ **Post date:** [November 8, 2020, 8:41am UTC](https://discuss.elastic.co/t/huge-dictionary-in-logstash-translate-filter/253857/12 "2020-11-08T08:41:25Z")
Hi Peter, Configuration for memcached is simple, in my case it is shown below I am mapping imeiTac to deviceName ```auto memcached { hosts => ["memcached_server_ip"] get => { "%{imeiTac}" => "[deviceName]" } } ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 6, 2020, 8:41am UTC](https://discuss.elastic.co/t/huge-dictionary-in-logstash-translate-filter/253857/13 "2020-12-06T08:41:33Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.