Updating the dictionary file used in translate filter while logs are being processed

Elastic Stack Logstash
1

Currently the Logstash instance that we're running basically runs 24 hours a day (processes millions of logs everyday). It's using a file as a mapping dictionary through a Translate filter with the default refresh interval. The docs here mention a default rate for refreshing the file and that's what we're using in the configuration file.

My question is regarding the architecture of the refresh process: If logs are continuously being processed, does it mean the file being used for mapping is under lock and being read continuously or is it read into memory every refresh interval and that's what's used for mapping?

My use case is that the file will be updated on a weekly basis but the process will only be able to update the file if it's not being read by another process (i.e. Logstash in this case). Any help is appreciated!

2

The file is read into a hash every refresh interval.

3

Thanks @Badger, you mean the file contents itself? So then if the file is being updated it doesn't matter because the previous contents are already in a hash in-memory right?

4

I believe that is correct.

5

That makes sense, thanks! Would you know why I could be running into the following error:

[main] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<LogStash::Filters::Dictionary::DictionaryFileError: Translate: Unquoted fields do not allow \r or \n (line 1). when loading dictionary file at ...

The CSV looks like this:

KEY,ID
"GDC123","000355"
"GDC154","000355"
"GDC165","000355"
"GDC1786","018265"
"GDC1987","005543"
6

A Windows format text file on a UNIX machine?

7

What do you mean, isn't that the normal way a csv is defined? We tried it without the double quotes and it's the same exact error which is why we tried with the quotes.

8

I mean a file that uses \r\n as a line ending on a machine that uses \n as a line ending. That would result in the ruby CSV parser seeing a trailing \r on a field.

9

It only has \n at the end of each line which is why I'm confused. What you're saying makes sense though and I wish that was it.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.