Best Match & Exact Match

Elastic Stack Logstash
1

Hi,

I have input coming in from a database , specifically call detail records , I need to parse the calling party number to identify the country from a dictionary file.

The problem I am facing is that unless the input field is exactly equal to the value in dictionary , it doesn't match , For example:

The input field is "calling_mut":937066

dictionary file is :

"9370": "Afghanistan"
"93706": "Afghanistan - AWC"

The filter I am using is :

mutate {
             convert => {
                    calling_mut   => "integer"
                    }
            }
    translate {
            field => "calling_mut"
            dictionary_path => "/etc/logstash/conf.d/dialcode-country.yaml"
   		add_field => ["country","%{translation}"]
            fallback => "no match"
   		}

The expected result is that it should match 93706 - Afghanistan - AWC as best match , but it doesn't , as the full number 937066 is not in dictionary , Is it possible to force it to use best match instead of exact match , or is there any other way I can match against a dictionary using best match?

2

Operationally, if the event field specified in the field configuration matches the EXACT contents of a dictionary entry key (or matches a regex if regex configuration item has been enabled), the field’s value will be substituted with the matched key’s value from the dictionary.

Depending on how your caller ids are coming in then you may be able use a regex.

In the case above, if you only have five significant characters in your dictionary then you could mutate the calling_mut to five chars and search on that.

3

Thanks a lot for the reply , The problem is that the dictionary has some countries with detailed in-country operator breakdown as well and some of them are just country codes.

I am looking up the first 5 digits , if i used exact => false , It does match it but iterates through each character in the input field separately and matches so

937066 results in AfghanistanRussia0Thailand
93 - afghanistan
7 - Russia
0 - Not in dictionary
66 - Thailand

Is there any way to trigger this match , as its pretty critical to what i am trying to do.

4

Have you tried anchoring your regexes at the start of the string, e.g. using ^66 instead of 66?

5

I am using the translate filter and not regexes.

How can i achieve this with regexes without writing one for the whole dictionary ( 45000 entries ).

6

The translate filter supports regex matching, so you can enable regex mode and store regexes as key. With that many regex entries it could however possibly be slow, but you will need to benchmark it.

7

Hi,

Just reverting back here , for my specific use - case , identifying country code from number , I achieved it by using custom ruby code and phonelib , which is based on libphonenumber from google.

Thanks a lot for replying everyone!