Topbeat Kibana Dashboard Index not working

bluethundr · February 11, 2016, 7:15pm

hey all,

I'm trying to get the example beats dashboards working in Kibana. These are the ones I'm hoping to work with:

So I followed the instructions here:

And I tried uploading the topbeat index to my elasticsearch cluster like so:

[root@web1:~] #curl -u admin:$ES_PASS  -XPUT 'http://es1.example.com:9200/_template/topbeat' -d@/etc/topbeat/topbeat.template.json
{"acknowledged":true}

But when I go to the first elasticsearch server (which is also doing Kibana and Logstash) I don't see the new index listed:

[root@logs:~] #curator --http_auth admin:$ES_PASS show indices --all-indices
2016-02-11 14:07:21,115 INFO      Job starting: show indices
2016-02-11 14:07:21,139 INFO      Matching all indices. Ignoring flags other than --exclude.
2016-02-11 14:07:21,139 INFO      Action show will be performed on the following indices: [u'.kibana', u'.marvel-es-2016.02.10', u'.marvel-es-2016.02.11', u'.marvel-es-data', u'logstash-2016.02.10', u'logstash-2016.02.11']
2016-02-11 14:07:21,139 INFO      Matching indices:
.kibana
.marvel-es-2016.02.10
.marvel-es-2016.02.11
.marvel-es-data
logstash-2016.02.10
logstash-2016.02.11

And when I go to load the dashboard, this is what I see:

Visualize: [index_not_found_exception] no such index, with: {"index":"[topbeat-*]"}

I'm attaching the json template file that I used to upload the index to ES.

So why is that index not available, even tho ES clearly tells me 'acknowledged' after I upload the index? How can I get this to work?

Thanks

Joe_Fleming · February 11, 2016, 7:26pm

Have you started running topbeat yet? The command you listed only creates the index template, not the index itself. You won't have a topbeat index until topbeat starts writing some data. That template only helps to ensure that the fields get the right mappings.

bluethundr · February 11, 2016, 7:38pm

Hi Joe,

Yeah, I've started running topbeat on one of my servers so far. I'm currently using filebeat with great success! So I thought that if I configure topbeat similarly to filebeat, I would have a working topbeat setup. However that doesn't seem to be the case so far. Because at this point no topbeat index has been created as of yet.

This is probably not a kibana issue at this point, but here's my topbeat config:

[root@web1:/etc/topbeat] #egrep -v "^$|^#|^(.*)#" topbeat.yml
input:
  period: 10
  procs: [".*"]
  stats:
    system: true
    proc: true
    filesystem: true
    cpu_per_core: false
output:
  elasticsearch:
    hosts: ["logs.example.com:5000"]
shipper:
  tags: ["service-X", "web-tier", "jf-dev"]
logging:
  files:

This is my beats input on the Logstash side of things:

beats {
     port => 5000
     tags => "beats"
     codec => "json"
     #ssl  => true
     #ssl_certificate => "/opt/filebeats/logs.example.com.crt"
     #ssl_key => "/opt/filebeats/logs.example.com.key"
     }

And I've been able to verify that topbeat is running on my one web server.

[root@web1:/etc/topbeat] #ps -ef | grep topbeat |grep -v grep
root     27146     1  0 Feb10 ?        00:04:55 /usr/bin/topbeat -c /etc/topbeat/topbeat.yml

So how can I verify that TopBeat is working on the Logstash side? Or am I better off sending topbeat directly into Elasticsearch for processing?

Thanks

Joe_Fleming · February 11, 2016, 7:50pm

I don't know about better off, but it's worth trying just to make sure topbeat is in fact sending data along. If you point topbeat at your ES cluster and you start getting data, then at least you know the issue has something to do with Logstash.

There's also a debug flag you can use with topbeat that'll report what it's (trying) to send, which might be helpful. I believe -d "publish" is good enough, and I think you can specify the selectors in the config file too.

bluethundr · February 11, 2016, 7:54pm

I also just tried reconfiguring topbeat to output to elasticsearch, just for giggles and to see if that would create the topbeat index.

[root@web1:/etc/topbeat] #egrep -v "^$|^#|^(.*)#" topbeat.yml
input:
  period: 10
  procs: [".*"]
  stats:
    system: true
    proc: true
    filesystem: true
    cpu_per_core: false
output:
  elasticsearch:
    hosts: ["es1.example.com:9200"]
    protocol: "https"
    username: "admin"
    password: "secret"
    worker: 1
    index: "topbeat"
shipper:
  tags: ["service-X", "web-tier", "jf-dev"]
logging:
  files:

No luck there. The index still doesn't get created and the error remains:

[root@logs:/etc/logstash/conf.d] #curator --http_auth admin:$ES_PASS show indices --all-indices             2016-02-11 14:53:47,504 INFO      Job starting: show indices
2016-02-11 14:53:47,529 INFO      Matching all indices. Ignoring flags other than --exclude.
2016-02-11 14:53:47,529 INFO      Action show will be performed on the following indices: [u'.kibana', u'.marvel-es-2016.02.10', u'.marvel-es-2016.02.11', u'.marvel-es-data', u'logstash-2016.02.10', u'logstash-2016.02.11']
2016-02-11 14:53:47,529 INFO      Matching indices:
.kibana
.marvel-es-2016.02.10
.marvel-es-2016.02.11
.marvel-es-data
logstash-2016.02.10
logstash-2016.02.11

Do think that creating the topbeat-* index manually could work?

Joe_Fleming · February 11, 2016, 8:04pm

OK, to me, it doesn't seem like topbeat is working correctly. Try running it with some debugging and see if you're actually trying to do anything, and check for errors in the output.

bluethundr · February 11, 2016, 8:33pm

Hey Joe,

I actually got this working. I got started by outputting to Elasticsearch first. The mistake I had made first time around was in using the HTTPS protocol instead of the HTTP protocol. Once I used HTTP for the output to ES, the dashboard started working. I then stopped the topbeat agent, and configured it for Logstash. And when I started it up again, it continued working fine!

Thanks for your help! Looking forward to checking out packetbeats next. Now that I seem to have filebeat and topbeat working correctly.

Thanks

Kevin_Rego · February 25, 2016, 8:22pm

I made the mistake of configuring the host information in the elasticsearch output section of the config. commented that all out, and configured the logstash output section and it my index got created.

Lucas_Pottersky · June 2, 2016, 9:41pm

My problem was on my logstash.conf. By default it was writing to "logstash" index. So I changed it.

You probably want to configure the "index" and "document_type" settings to read from the @metadata fields, like so:

# logstash.conf
output {
  elasticsearch {
    host => "localhost"
    port => "9200"
    protocol => "http"
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }

This way, LogStash will use the metadata sent from the TopBeat agent.

I figured [@metadata][beat] has the value that you define on your topbeat.yml "index" key. The default is "topbeat", so [@metadata][beat] will be equal to "topbeat".