Hello,
I have
input {
kafka {
zk_connect => "list of zoo keepers"
white_list => "topbeat-wc1|filebeat-wc1|winlogbeat-wc1"
decorate_events => true
codec => plain
}
}
This is the standard out
{
"message" => "2016-04-22T23:01:43.437Z blah %{message}",
"@version" => "1",
"@timestamp" => "2016-04-22T23:01:44.747Z",
"kafka" => {
"msg_size" => 50,
"topic" => "topbeat-wc1",
"consumer_group" => "logstash",
"partition" => 0,
"key" => nil
}
}
{
"message" => "2016-04-22T23:01:40.680Z blah 04/22/2016 16:01:38|spLIST_GetUpdatedListings_153|Listing Server|78|0",
"@version" => "1",
"@timestamp" => "2016-04-22T23:01:44.747Z",
"kafka" => {
"msg_size" => 110,
"topic" => "filebeat-wc1",
"consumer_group" => "logstash",
"partition" => 6,
"key" => nil
}
}
I am trying to use the value of the topic and assign it to index in the output section
output {
elasticsearch {
hosts => [ list of data nodes]
index => %{topic}
}
It's currently not working. Should I do this using metadata, or there is a simpler way that I am not seeing at the moment