# Regarding big dictionary (2GB) in filter translate plug-in

**URL:** <https://discuss.elastic.co/t/regarding-big-dictionary-2gb-in-filter-translate-plug-in/187423>\
**Category:** Logstash\
**Created:** [June 25, 2019, 7:38pm UTC](https://discuss.elastic.co/t/regarding-big-dictionary-2gb-in-filter-translate-plug-in/187423 "2019-06-25T19:38:27Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![leeyu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leeyu/32/48561_2.png) [@leeyu](https://discuss.elastic.co/u/leeyu)\
**Post date:** [June 25, 2019, 7:38pm UTC](https://discuss.elastic.co/t/regarding-big-dictionary-2gb-in-filter-translate-plug-in/187423/1 "2019-06-25T19:38:27Z")

</div>

Hi all,

Is a 2GB dictionary considered too big for the filter translate plug-in ?

Is there any alternative option (such as access the dictionary from elastic search)?

Thanks.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 25, 2019, 8:15pm UTC](https://discuss.elastic.co/t/regarding-big-dictionary-2gb-in-filter-translate-plug-in/187423/2 "2019-06-25T20:15:02Z")

</div>

Is it too big? That depends. I run logstash with a 512 MB heap, it would definitely be too big for that. If I had a 2 TB heap (or even a 8 GB heap) then it might not be too big. Does it have two 1 GB entries or 2 billion single bit entries? That could also determine whether it is "too big".

There are other filters that might provide this functionality such as elasticsearch, jdbc\_streaming, http, or memcached.

---

<div class="post-metadata">

**Author:** ![leeyu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leeyu/32/48561_2.png) [@leeyu](https://discuss.elastic.co/u/leeyu)\
**Post date:** [June 25, 2019, 9:03pm UTC](https://discuss.elastic.co/t/regarding-big-dictionary-2gb-in-filter-translate-plug-in/187423/3 "2019-06-25T21:03:06Z")

</div>

Hi Badger,

Thanks for the information. I will try to increase heap and see what happens.

Is there any material that I can look at regarding using elasticsearch as dictionary?

---

<div class="post-metadata">

**Author:** ![leeyu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leeyu/32/48561_2.png) [@leeyu](https://discuss.elastic.co/u/leeyu)\
**Post date:** [June 26, 2019, 11:04pm UTC](https://discuss.elastic.co/t/regarding-big-dictionary-2gb-in-filter-translate-plug-in/187423/4 "2019-06-26T23:04:33Z")

</div>

Hi Badger,

I tried to use elasticsearch plug-in in filter plug-in.

Here is what I got

```
input{
  file{
      path => "/data/threat_event/data/2018-01-01/all_1204346130001631.csv"
      start_position => "beginning"
      sincedb_path => "/dev/null"
      max_open_files => 65535
  }
}

filter{

    csv {
      autodetect_column_names => "true"
      autogenerate_column_names => "true"
      skip_header => "true"
      separator => ","
    }

    elasticsearch{
        hosts => ["localhost:9200"]
        index => "severity-mapping"
        query => "number.keyword:%{[severity]}"
        result_size => 1
        fields => {"level" => "severity_level"}
    }
}

output
{
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "es_dictionary_test"
  }
}

```

It creates field but the field contains nothing.

We also tried many combinations of the query part.

Any idea? Thanks

severity-mapping

```
number,level
1,Low
2,Medium
3,High
4,Critical
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 27, 2019, 12:30am UTC](https://discuss.elastic.co/t/regarding-big-dictionary-2gb-in-filter-translate-plug-in/187423/5 "2019-06-27T00:30:06Z")

</div>

I do not have an elasticsearch instance running, so I cannot help. I suggest you ask a new question in the logstash forum, about how to do a simple lookup using an elasticsearch filter, and mention that the dataset is too large to use translate. I would include a sample of a couple of documents from the severity-mapping index.

Are you certain that the thing you want to translate is in [severity]?

It seems a little odd that you are using both autodetect\_column\_names and autogenerate\_column\_names. There are use cases where you need both but they are unusual.

---

<div class="post-metadata">

**Author:** ![leeyu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leeyu/32/48561_2.png) [@leeyu](https://discuss.elastic.co/u/leeyu)\
**Post date:** [June 27, 2019, 12:51am UTC](https://discuss.elastic.co/t/regarding-big-dictionary-2gb-in-filter-translate-plug-in/187423/6 "2019-06-27T00:51:55Z")

</div>

Hi Badger,

Thank for the suggestion,

We are trying to use json instead, because my friend got it work with json.

So it might be some problem to use csv this way.

I'll open a new post if anything thanks a lot.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 25, 2019, 12:52am UTC](https://discuss.elastic.co/t/regarding-big-dictionary-2gb-in-filter-translate-plug-in/187423/7 "2019-07-25T00:52:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.