Topbeat index not propagating to Elasticsearch


(Stephen) #1

I currently downloaded all the current software for elasticsearch, logstash, redis and topbeats.

I currently have my topbeats going into redis, that goes to my logstash, that goes into elasticsearch. Everything is working pretty much right out of the box setup.

When I go to view the data via Kibana, I see all the data coming in is using the logstash index and not the topbeat one I specified in the config.

I am wondering why? Is there an issue with redis that changes the index or some other setting that might redefine the message?

An example message I am writing into ES that came through the system. Should I be seeing the index of topbeat within the message from the topbeat app?

{"@timestamp":"2016-07-18T17:05:53.841Z","beat":{"hostname":"hostname123","name":"hostname123"},"count":1,"fs":{"device_name":"/dev/mapper/rootvg-varcrash","total":4227530752,"used":142598144,"used_p":0.03,"free":4084932608,"avail":3870187520,"files":262144,"free_files":262133,"mount_point":"/var/crash"},"type":"filesystem","@version":"1"}

Logstash config file.

input {
redis{
host => "host123"
port => 6380
data_type => "channel" # use pub/sub, like we do with rsyslog
key => "logstash" # use the same key as in rsyslog
batch_count => 100 # how many messages to fetch at once
}
redis{
host => "host123"
port => 6380
data_type => "list" # use pub/sub, like we do with rsyslog
key => "topbeat" # use the same key as in rsyslog
batch_count => 10 # how many messages to fetch at once
}
}

The filter part of this file is commented out to indicate that it is

optional.

filter {

}

output {
elasticsearch {
hosts => ["host123:9210"]
#index => "%{type}_indexer"
}
#stdout {}

#file {

path => "/tmp/text.log"

#}
}


(Steffen Siering) #2

please properly format config files, logs either using 3 back-ticks \`` or use the</>` button.

You didn't configure any index in logstash elasticsearch output. So logstash will forward all events to index logstash-<data> for you.


(system) #3

This topic was automatically closed after 21 days. New replies are no longer allowed.