Hi, I am trying to reindex sensor data in order to enrich it via lookup. My goal is to add multiple fields, as shown in the csv plugin config below.
While the reindexing itself starts, I see no difference in the new index. Here is my configuration:
reindex.conf
input {
elasticsearch {
hosts => [ "localhost:9200" ]
query => '{ "query": { "query_string": { "query": "*" } } }'
index => "knx-*"
docinfo => true
}
}
filter {
translate {
field => "edomi-GA_KO"
# destination => "alldata"
dictionary_path => "/etc/logstash/conf.d/lookup/knxlookup.csv"
# exact => false
fallback => "no match"
# start_position => "beginning"
}
csv {
source => "translation"
columns => [
"GA",
"ID",
"Name",
"DPT",
"Geschoss",
"Raum",
"Gewerk",
"Funktion"
]
separator => ";"
}
}
output {
elasticsearch {
hosts => [ "localhost:9200" ]
index => "knx2-%{+YYYY.MM}"
document_id => "%{[@metadata][_id]}"
}
}
the lookup file is present and has this value:
1/1/0;133;Raum 1 Licht schalten;1;EG;Raum 1;Licht;ein / aus
1/1/1;134;Raum 2 Licht schalten;1;EG;Raum 2;Licht;ein / aus
The Field "edomi-GA_KO" is present in the source index and has data matching the first column in the lookup file.
Following the documentation the translate plugin would add a field named "translation" by default, which also isn't created. Logstash logs don't show errors but indicate that the reindex is starting which I can use afterwards.
Thanks in advance!
André
EDIT:
Now after letting it sit for some minutes and the new index is created, has the same number of documents as the source index. after refreshing the index pattern I see the "translation" field and the first column "GA" showing up. But they aren't visible in Kibana Discover. I notice that still something is going on, and cat on the indices gives me this (docs.deleted is growing while I am writing this):
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open knx2-2017.05 QfwJbTbZSaqD2FprBcndKQ 1 0 6122337 2443710 2.2gb 2.2gb
green open knx-2017.05a iZ-NZ1ojRb2qXuDOp-z-zQ 1 0 6122337 0 956.8mb 956.8mb