I currently downloaded all the current software for elasticsearch, logstash, redis and topbeats.
I currently have my topbeats going into redis, that goes to my logstash, that goes into elasticsearch. Everything is working pretty much right out of the box setup.
When I go to view the data via Kibana, I see all the data coming in is using the logstash index and not the topbeat one I specified in the config.
I am wondering why? Is there an issue with redis that changes the index or some other setting that might redefine the message?
An example message I am writing into ES that came through the system. Should I be seeing the index of topbeat within the message from the topbeat app?
{"@timestamp":"2016-07-18T17:05:53.841Z","beat":{"hostname":"hostname123","name":"hostname123"},"count":1,"fs":{"device_name":"/dev/mapper/rootvg-varcrash","total":4227530752,"used":142598144,"used_p":0.03,"free":4084932608,"avail":3870187520,"files":262144,"free_files":262133,"mount_point":"/var/crash"},"type":"filesystem","@version":"1"}
Logstash config file.
input {
redis{
host => "host123"
port => 6380
data_type => "channel" # use pub/sub, like we do with rsyslog
key => "logstash" # use the same key as in rsyslog
batch_count => 100 # how many messages to fetch at once
}
redis{
host => "host123"
port => 6380
data_type => "list" # use pub/sub, like we do with rsyslog
key => "topbeat" # use the same key as in rsyslog
batch_count => 10 # how many messages to fetch at once
}
}
The filter part of this file is commented out to indicate that it is
optional.
filter {
}
output {
elasticsearch {
hosts => ["host123:9210"]
#index => "%{type}_indexer"
}
#stdout {}
#file {
path => "/tmp/text.log"
#}
}