Translate filter with +2.5 million dictionary entries

Elastic Stack Logstash
1

Hi,
I have a case when I need to use a translate filter with a very large YAML dictionary (around 2.5 million of entries).
I was reading this post

where it recommends to use memcached in order to speed up the translation.
But my case is I use the regex => true

                                    translate{
                                            field => "a_key"
                                            destination => "data_A_Billing"
                                            regex => true
                                            exact => true
                                            dictionary_path => '/data/sbc/tables/rates_a_billing.yaml'
                                            fallback => "NF|NF|-1|Destination Not Found|Origin Not Found|NF"
                                    }

where a_key is a combination of the destination prefix and an originating number in a phone call
For example:

34655#376688785

That will match the following line in the yaml

'^34655#376[0-9]*$': "34655|376|0,083200|Spain -Mob ORANGE|ZONE 3|A"

The dictionary file is sorted from more specific to more general regex.

Originally the dictionary was of 400.000 entries, but now is 7 times bigger and latency in processing events are now a drawback.

Is there a better way to implement this?
Any feddback will be appreciated

Thank you!

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.