Reindex + enrich data

Elastic Stack Logstash
1

Hi, I am trying to reindex sensor data in order to enrich it via lookup. My goal is to add multiple fields, as shown in the csv plugin config below.
While the reindexing itself starts, I see no difference in the new index. Here is my configuration:

reindex.conf
    input {
      elasticsearch {
        hosts => [ "localhost:9200" ]
        query => '{ "query": { "query_string": { "query": "*" } } }'
        index => "knx-*"
        docinfo => true
      }
    }
    filter {
      translate {
        field => "edomi-GA_KO"
    #    destination => "alldata"
        dictionary_path => "/etc/logstash/conf.d/lookup/knxlookup.csv"
    #    exact => false
        fallback => "no match"
    #    start_position => "beginning"
        }
      csv {
        source => "translation"
        columns => [ 
        "GA",
        "ID",
        "Name",
        "DPT",
        "Geschoss",
        "Raum",
        "Gewerk",
        "Funktion"
        ]
        separator => ";"
        }
    }
    output {
      elasticsearch {
        hosts => [ "localhost:9200" ]
        index => "knx2-%{+YYYY.MM}"
        document_id => "%{[@metadata][_id]}"
      }
    }

the lookup file is present and has this value:

1/1/0;133;Raum 1 Licht schalten;1;EG;Raum 1;Licht;ein / aus
1/1/1;134;Raum 2 Licht schalten;1;EG;Raum 2;Licht;ein / aus

The Field "edomi-GA_KO" is present in the source index and has data matching the first column in the lookup file.
Following the documentation the translate plugin would add a field named "translation" by default, which also isn't created. Logstash logs don't show errors but indicate that the reindex is starting which I can use afterwards.

Thanks in advance!
André

EDIT:

Now after letting it sit for some minutes and the new index is created, has the same number of documents as the source index. after refreshing the index pattern I see the "translation" field and the first column "GA" showing up. But they aren't visible in Kibana Discover. I notice that still something is going on, and cat on the indices gives me this (docs.deleted is growing while I am writing this):

health status index        uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   knx2-2017.05 QfwJbTbZSaqD2FprBcndKQ   1   0    6122337      2443710      2.2gb          2.2gb
green  open   knx-2017.05a iZ-NZ1ojRb2qXuDOp-z-zQ   1   0    6122337            0    956.8mb        956.8mb
2

Okay after a while the translation field shows up, unfortunately it always shows "no match" so there's something wrong with my translate config.

3

Solved it by myself. I had to recreate the lookupto searate the first column with a comma (because translate doesn't seen to accept the ";" and then all other column with it.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.