Filter and mapping by multiple fields in translate filter and dictionary

Elastic Stack Logstash
1

I have a logstash event which e.g. looks like this:

{
  "field1": "value1",
  "field2": "value2"
}

Now I want to perform a translate filter which filters and maps by the two fields ("field1" or/and "field2"). So, when this event will be imported to ES, logstash should only take the value of the mapping dictionary, if "field1" or/and "field2" are matching. So it is like a double filtering. At the moment it is only possible to name one field to filter and only one field for mapping in the dictionary.

The original logstash config looks like this:

translate{
  field => "[field1]" 
  destination => "[newfield]"
  dictionary_path="/tmp/mapping.json"
  fallback => ""
  exact => true
  refresh_behaviour => "replace"
}

and the mapping dictionary like this:

{
  "value1": "newvalue1",
  "value2": "newvalue2"
}

This would result in:

{
  "field1": "value1",
  "field2": "value2",
  "newfield": "newvalue1"
}

But only filtered and mapped by "field1". So I can't filter and map first by "field1" and after by "field2" of vice versa, because of the mapping dictionary and the filter.

My idea is that the mapping dictionary e.g. looks like this:

{
  "value1": 
  {
    "value2": "newvalue"
  }
}

and maybe this the logstash config:

translate{
  field1 => "[field1]" 
  field2 => "[field2]"
  operator => and/or 
  destination => "[newfield]"
  dictionary_path="/tmp/mapping.json"
  ...
}

the result event should look like this:

{
  "field1": "value1",
  "field2": "value2",
  "newfield": "newvalue"
}

But filtered and mapped by "field1" and/or "field2". I hope you get it what I mean.

I already searched for this idea and found this GitHub issue which is a kind of merging fields to reduce the number of mapping fields and does not fit my idea. All other discussions I found were similar or the same.

Another idea, if the translate filter does not fit my idea, is to merge the values of "field1" and "field2" to one new field which new value is separated by any letter, e.g. after adding a new tag the event could look like this:

{
  "field1": "value1",
  "field2": "value2",
  "merged_fields": "value1_value2"
}

And have a mapping dictionary which maps by "value1_value2":

{
  "value1_value2": "newvalue"
}

A second idea I got was to write a ruby script which fits it in any way..Do you have any suggestions on this topic?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.