Why does translate not work for me?

Elastic Stack Logstash
1

Hi,

I can't wrap my head around why I don't get this translate filter to work.
I have a bunch of IoT logfiles (csv) that I want to import.

  • One of the fields (KO-ID) does contain an internal ID of the old log engine, for example "3051"
  • another field (GrpAdr) contains the ID I need to work with in the future. Because it is not available for all log entries, I want to substitute this GrpAdr for a range of old IDs (3050 to 3101). In other words: if KO-ID is in the range above, use the lookup table to overwrite whatever there is in GrpAdr (it should be empty anyway).

KO-ID and GrpAdr both populate the correct fields when I import the log files. However, I cannot get the translation to work (also tried a dictionary instead of lookup file). GrpAdr is always empty for the values in the range, but it has values when they are in the original log.

current config:

filter {
 ... 
  if "knximport" in [tags] {

    csv {
      columns => [ "eventtime", "Funktion", "Geschoss", "Gewerk", "KNX-Name", "KNX-Wert", "KNX-Wert-Float", "Name", "Raum", "GrpAdr", "KOID", "PA", "Typ" ]
      separator => ","
    }

    translate {
      source => "[KOID]"
      target => "[GrpAdr]"
      dictionary => {
        "3050" => "7/0/0"
        "3051" => "7/0/1"
        "3052" => "7/0/2"
        ...
        "654"  => "11/2/3"
        "669"  => "11/2/4"
      }
      fallback => "%{GrpAdr}" 
    }

I tried the field name with and without brackets, and several other things. However, the result is always the same:

grafik
grafik2298×634 55.6 KB

Here is a snippet from what I am importing:

2023-09-10T10:30:00.150Z,,,,Zirkulation,-29.9,,no match,,-,3056,,INTERN
2023-09-10T10:30:00.156Z,,,,Brauchwasser Solltemperatur,50,,no match,,-,3060,,INTERN
2023-09-10T10:30:00.160Z,,,,Betriebsart,3,,no match,,-,3064,,INTERN
2023-09-10T10:30:00.165Z,,,,Fusspunkt,35,,no match,,-,3068,,INTERN
2023-09-10T10:30:00.170Z,,,,Heizgrenze Absenken,10,,no match,,-,3072,,INTERN
2023-09-10T10:30:00.175Z,,,,Spreizung Heizkreis,20,,no match,,-,3076,,INTERN

What am I missing?

2

GrpAdr will always exist, even if it is empty, so you need to set the override option on the translate filter, otherwise the filter will not replace it.

3

That was it, thank you very much!