# Updating the dictionary file used in translate filter while logs are being processed

**URL:** <https://discuss.elastic.co/t/updating-the-dictionary-file-used-in-translate-filter-while-logs-are-being-processed/214494>\
**Category:** Logstash\
**Created:** [January 9, 2020, 9:34pm UTC](https://discuss.elastic.co/t/updating-the-dictionary-file-used-in-translate-filter-while-logs-are-being-processed/214494 "2020-01-09T21:34:45Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![dmalhan](https://avatars.discourse-cdn.com/v4/letter/d/c77e96/32.png) [@dmalhan](https://discuss.elastic.co/u/dmalhan)\
**Post date:** [January 9, 2020, 9:34pm UTC](https://discuss.elastic.co/t/updating-the-dictionary-file-used-in-translate-filter-while-logs-are-being-processed/214494/1 "2020-01-09T21:34:45Z")

</div>

Currently the Logstash instance that we're running basically runs 24 hours a day (processes millions of logs everyday). It's using a file as a mapping dictionary through a Translate filter with the default refresh interval. The docs [here](https://www.elastic.co/guide/en/logstash/current/plugins-filters-translate.html#plugins-filters-translate-refresh_interval) mention a default rate for refreshing the file and that's what we're using in the configuration file.

My question is regarding the architecture of the refresh process: **If logs are continuously being processed, does it mean the file being used for mapping is under lock and being read continuously or is it read into memory every refresh interval and that's what's used for mapping?**

My use case is that the file will be updated on a weekly basis but the process will only be able to update the file if it's not being read by another process (i.e. Logstash in this case). Any help is appreciated!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 9, 2020, 10:28pm UTC](https://discuss.elastic.co/t/updating-the-dictionary-file-used-in-translate-filter-while-logs-are-being-processed/214494/2 "2020-01-09T22:28:25Z")

</div>

> [@dmalhan](#):
>
> does it mean the file being used for mapping is under lock and being read continuously or is it read into memory every refresh interval and that's what's used for mapping?

The file is read into a hash every refresh interval.

---

<div class="post-metadata">

**Author:** ![dmalhan](https://avatars.discourse-cdn.com/v4/letter/d/c77e96/32.png) [@dmalhan](https://discuss.elastic.co/u/dmalhan)\
**Post date:** [February 3, 2020, 7:28pm UTC](https://discuss.elastic.co/t/updating-the-dictionary-file-used-in-translate-filter-while-logs-are-being-processed/214494/3 "2020-02-03T19:28:50Z")

</div>

Thanks @Badger, you mean the file contents itself? So then if the file is being updated it doesn't matter because the previous contents are already in a hash in-memory right?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 3, 2020, 7:37pm UTC](https://discuss.elastic.co/t/updating-the-dictionary-file-used-in-translate-filter-while-logs-are-being-processed/214494/4 "2020-02-03T19:37:31Z")

</div>

I believe that is correct.

---

<div class="post-metadata">

**Author:** ![dmalhan](https://avatars.discourse-cdn.com/v4/letter/d/c77e96/32.png) [@dmalhan](https://discuss.elastic.co/u/dmalhan)\
**Post date:** [February 4, 2020, 8:39pm UTC](https://discuss.elastic.co/t/updating-the-dictionary-file-used-in-translate-filter-while-logs-are-being-processed/214494/5 "2020-02-04T20:39:06Z")

</div>

That makes sense, thanks! Would you know why I could be running into the following error:

> [main] Pipeline aborted due to error {:pipeline\_id=\>"main", :exception=\>#\<LogStash::Filters::Dictionary::DictionaryFileError: Translate: Unquoted fields do not allow \r or \n (line 1). when loading dictionary file at ...

The CSV looks like this:

```
KEY,ID
"GDC123","000355"
"GDC154","000355"
"GDC165","000355"
"GDC1786","018265"
"GDC1987","005543"

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 4, 2020, 8:58pm UTC](https://discuss.elastic.co/t/updating-the-dictionary-file-used-in-translate-filter-while-logs-are-being-processed/214494/6 "2020-02-04T20:58:36Z")

</div>

A Windows format text file on a UNIX machine?

---

<div class="post-metadata">

**Author:** ![dmalhan](https://avatars.discourse-cdn.com/v4/letter/d/c77e96/32.png) [@dmalhan](https://discuss.elastic.co/u/dmalhan)\
**Post date:** [February 4, 2020, 9:27pm UTC](https://discuss.elastic.co/t/updating-the-dictionary-file-used-in-translate-filter-while-logs-are-being-processed/214494/7 "2020-02-04T21:27:05Z")

</div>

> [@Badger](#):
>
> Windows format text file

What do you mean, isn't that the normal way a csv is defined? We tried it without the double quotes and it's the same exact error which is why we tried with the quotes.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 4, 2020, 9:59pm UTC](https://discuss.elastic.co/t/updating-the-dictionary-file-used-in-translate-filter-while-logs-are-being-processed/214494/8 "2020-02-04T21:59:08Z")

</div>

> [@dmalhan](#):
>
> What do you mean

I mean a file that uses \r\n as a line ending on a machine that uses \n as a line ending. That would result in the ruby CSV parser seeing a trailing \r on a field.

---

<div class="post-metadata">

**Author:** ![dmalhan](https://avatars.discourse-cdn.com/v4/letter/d/c77e96/32.png) [@dmalhan](https://discuss.elastic.co/u/dmalhan)\
**Post date:** [February 4, 2020, 10:00pm UTC](https://discuss.elastic.co/t/updating-the-dictionary-file-used-in-translate-filter-while-logs-are-being-processed/214494/9 "2020-02-04T22:00:53Z")

</div>

It only has `\n` at the end of each line which is why I'm confused. What you're saying makes sense though and I wish that was it.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 3, 2020, 10:00pm UTC](https://discuss.elastic.co/t/updating-the-dictionary-file-used-in-translate-filter-while-logs-are-being-processed/214494/10 "2020-03-03T22:00:55Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.