Translate return multiple fields

Elastic Stack Logstash
#1

Hi, I have been using the translate plugin that works great however I have multiple fields that all reference to the one field used for the translate. So is it possible to use the translate plugin to lookup a value against a file and return multiple fields that is then loaded into ES?

Here is what I am currently using in the filter:
...
translate {
field => "Calling"
destination => "Calling_Name"
dictionary_path => "/opt/.../calling_name.yaml"
}
...

The translate file looks currently as follows:

...
"1001": "OK"
"1002": "failed"

Here is what I would like to achive:

Lookup multiple fields e.g.

translate field
...
"1001": "field1,field2,field3,..."
...

The 1001 is the reference field then return all the different field1 to field3.

#2

It's not entirely clear what result you want to achieve for an event with Calling=1001. That said, the translate lookup returns a single value, but that value could be processed with other filters like csv, json, or kv.

#3

Apologies, yes I would like to process the value returned from the translate lookup. Do you have an example maybe how to process a returned value separated by a comma?

#4

That depends on how you want to process it.

#5

When the fields are coma separated like a csv file. How would you do this?

#6

Last chance: What is the desired result if "1001" maps to "field1,field2,field3"? If the event looks like

{
  "Calling": "1001"
}

before the translate filter, what should it look like after the translate filter?

#7

Apologies, I was not able to respond earlier. Thank you for the last chance, I will try and make it as clear as possible.

The current section and what I would like to achieve section could be the current challenge that is causing confusion.

When using filter [quote="Hans, post:1, topic:64590"]
...
translate {
field => "Calling"
destination => "Calling_Name"
dictionary_path => "/opt/.../calling_name.yaml"
}
...
[/quote]

The file being called [quote="Hans, post:1, topic:64590"]
dictionary_path => "/opt/.../calling_name.yaml"
[/quote] should looks something like this:

Now the field [quote="Hans, post:1, topic:64590"]
destination => "Calling_Name"
[/quote] field has multiples values of information separated by ",". How can I now split the field [quote="Hans, post:1, topic:64590"]
Calling_Name
[/quote] to index the fields individually in elasticsearch and not as one long string? Also can I use a CSV filter on this field to add the field name?

#8

to index the fields individually in elasticsearch and not as one long string?

You're still not answering the question! What does "index individually" mean?

One field per substring?

{
  "a": "field1",
  "b": "field2",
  "c": "field3"
}

An array?

{
  "a": ["field1", "field2", "field3"]
}

Three separate events?

{
  "a": "field1"
}
{
  "a": "field2"
}
{
  "a": "field3"
}

Something else?

Do not attempt to describe what you want with words. Use examples.

Also can I use a CSV filter on this field to add the field name?

Yes.

#9

OK when translating [quote="Hans, post:1, topic:64590"]
"1001": "field1,field2,field3,..."
[/quote]

I would like to have the:

Name of field 1 = field1
Name of field 2 = field2
Name of field 3 = field3
.....

So only one lookup 1001 returns multiples fields relating to this reference 1001

#10

In that case use the csv filter on the field containing "field1,field2,field3".

Reindex Elasticsearch using new mapping
#11

Hi, I have tried the following script however I only receive the first field and none of the others?

`            translate {
                                field => "MS_CC"
                                destination => "SUBDATA_ALL"
                                dictionary_path => "/etc/logstash/conf.d/translate/SUBSDATA.yaml"
                }
                csv {
        columns => [
                                        "S_ID",
                                        "C_ID",
                                        "C_NAME",
                                        "A_ID",
                                        "S_TYPE",
                                        "S_NAME",
                                        "MN",
                                        "ISI",
                                        "S_STATUS",
                                        "OFFER",
                                        "EFF_DATE",
                                        "P_P"
        ]
                    separator => ";"
                    remove_field => ["SUBDATA_ALL"]
        }`

SUBSDATA.yaml file sample:

...
"276476070": "501254424;500440257;PE PACKS;300467279;PREP;;276476070;655031001867176;Two Block;Olga;25/12/2015 00:00:00;Olga"
...
#12

Got it just forgot the source

#13