I'm in the middle of shipping our logs from beats to a Redis queue using pattern_channel. I've read that the metadata is not not available through the Redis input. Currently our logs ship directly from beats to logstash. So in our output, indexes were created via "index => "%{[@metadata][beat]}-%{+YYYY.MM}"" which doesn't work using Redis. I see that when I PSUBSCRIBE in Redis that the index information "filebeat-test-server1" and "filebeat-test-server2" are in Redis.
Example of data stream:
-
"pmessage"
-
"filebeat*"
-
"filebeat-test-server1"
-
"{"@timestamp":"2016-08-26T21:05:46.639Z","beat":{"hostname":"
-
"pmessage"
-
"filebeat*"
-
"filebeat-test-server2"
-
"{"@timestamp":"2016-08-26T21:05:46.639Z","beat":{"hostname":\
My question is, how to I retrieve that information? I'm looking to get that info and add it as a new field then using that new field to create the index in ES.