Metricbeat counter logstash API

[Jo_De_Troy]

Hello,

I was wondering if metricbeat can pull in metrics from logstash instances which are available though the logstash API (e.g. node stats API) ? Or is there another way to get insight on the performance/throughput of logstash instances?

Thanks in advance,
Jo

[ruflin]

There is currently no Logstash Module in Metricbeat. But Logstashbeat could be what you are looking for: https://github.com/consulthys/logstashbeat

[Jo_De_Troy]

Thx for the pointer.

I've tried it and was able to get in compiled.
But after starting it up it crashed with the error below:

 ./logstashbeat -c ./logstashbeat.yml -e -d "*"

2016/12/13 09:02:46.440125 beat.go:267: INFO Home path: [/usr/local/go/src/github.com/consulthys/logstashbeat] Config path: [/usr/local/go/src/github.com/consulthys/logstashbeat] Data path: [/usr/local/go/src/github.com/consulthys/logstashbeat/data ] Logs path: [/usr/local/go/src/github.com/consulthys/logstashbeat/logs]
2016/12/13 09:02:46.440201 beat.go:177: INFO Setup Beat: logstashbeat; Version: 6.0.0-alpha1
2016/12/13 09:02:46.440226 processor.go:43: DBG Processors:
2016/12/13 09:02:46.440237 beat.go:183: DBG Initializing output plugins
2016/12/13 09:02:46.440247 logp.go:219: INFO Metrics logging every 30s
2016/12/13 09:02:46.440363 output.go:167: INFO Loading template enabled. Reading template file: /usr/local/go/src/github.com /consulthys/logstashbeat/etc/logstashbeat.template.json
2016/12/13 09:02:46.440598 output.go:178: INFO Loading template enabled for Elasticsearch 2.x. Reading template file: /usr/l ocal/go/src/github.com/consulthys/logstashbeat/logstashbeat.template-es2x.json
2016/12/13 09:02:46.440788 client.go:120: INFO Elasticsearch url: http://elastic:9200
2016/12/13 09:02:46.440827 outputs.go:106: INFO Activated elasticsearch as output plugin.
2016/12/13 09:02:46.440835 publish.go:234: DBG Create output worker
2016/12/13 09:02:46.440894 publish.go:276: DBG No output is defined to store the topology. The server fields might not be f illed.
2016/12/13 09:02:46.440917 publish.go:291: INFO Publisher name: logstash.mgt.company.com
2016/12/13 09:02:46.441129 async.go:63: INFO Flush Interval set to: 1s
2016/12/13 09:02:46.441139 async.go:64: INFO Max Bulk Size set to: 50
2016/12/13 09:02:46.441147 async.go:72: DBG create bulk processing worker (interval=1s, bulk size=50)
2016/12/13 09:02:46.441260 logstashbeat.go:87: DBG Init logstashbeat
2016/12/13 09:02:46.441271 logstashbeat.go:88: DBG Period 10s
2016/12/13 09:02:46.441280 logstashbeat.go:89: DBG Watch [http://127.0.0.1:9600]
2016/12/13 09:02:46.441288 logstashbeat.go:90: DBG Capture 3 hot threads
2016/12/13 09:02:46.441294 logstashbeat.go:91: DBG JVM statistics true
2016/12/13 09:02:46.441311 logstashbeat.go:92: DBG Process statistics true
2016/12/13 09:02:46.441326 logstashbeat.go:93: DBG Pipeline statistics true
2016/12/13 09:02:46.441434 beat.go:207: INFO logstashbeat start running.
2016/12/13 09:02:46.441457 logstashbeat.go:99: INFO logstashbeat is running! Hit CTRL-C to stop it.
2016/12/13 09:02:56.441682 logstashbeat.go:116: DBG Hot threads for url: http://127.0.0.1:9600
2016/12/13 09:02:57.126712 logstashbeat.go:122: DBG Hot threads detail: &{Host:logstash Version:5.0.2 Http_ address:127.0.0.1:9600 Hot_threads:{Time:2016-12-13T10:02:56+01:00 Busiest_threads:3 Threads:[0xc42001adc0 0xc42001af80 0xc4 2001b140]}}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x46b5a2]

goroutine 30 [running]:
panic(0x8a2920, 0xc4200120f0)
/usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/consulthys/logstashbeat/beater.(*Logstashbeat).Run.func1(0xc42015b380, 0xc42015b400)
/usr/local/go/src/github.com/consulthys/logstashbeat/beater/logstashbeat.go:131 +0x17d2
created by github.com/consulthys/logstashbeat/beater.(*Logstashbeat).Run
/usr/local/go/src/github.com/consulthys/logstashbeat/beater/logstashbeat.go:211 +0x9d

Any idea if there are plans to incorporate the logstash metrics into a metricbeat module?

Best Regards,
Jo

[ruflin]

Best is to open a Github issue in logstashbeat so the problem can be fixed. There are no immediate plans to add a Logstash module. It is planned that Logstash can send its own metrics directly to elasticsearch.