Logstash X-Pack monitoring troubles

Slurpgeit · February 1, 2017, 7:21pm

Hello,

I'm trying to setup monitoring of some logstash machines, but I'm running into a few issues:

The documentation (https://www.elastic.co/guide/en/x-pack/5.2/monitoring-logstash.html) states that the "xpack.monitoring.elasticsearch.url" setting takes either a string or array. I would like to use the latter (for load balancing purposes, my logstash machines talk to the cluster through nginx). However, if I copy the example setting from the documentation page logstash startup fails with the following error:

[2017-02-01T20:03:17,321][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: Setting "xpack.monitoring.elasticsearch.url" must be a String. Received: ["http://es-prod-node-1:9200", "http://es-prod-node-2:9200"] (Array)>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:208:invalidate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:384:in validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:171:inset'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:61:in set_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:80:inmerge'", "org/jruby/RubyHash.java:1342:in each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:80:inmerge'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:115:in validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:210:inexecute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:183:inrun'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in(root)'"]}`

So does that mean that the documentatio is wrong, or that there is a bug somewhere?

How can I change the index that logstash uses for monitoring data?
I need to use TLS client certificates for authenticating to the cluster, but I can only seem to find options for setting a CA certificat,e not client certificates. What options would I use for client certificates?

Thanks in advance!

omarzing · February 2, 2017, 11:08pm

I'm having the same issue! We switched to a string, but I don't see any monitoring data.

EDIT: We solved our issue by using a string and removing any trailing slashes. Good luck!

Slurpgeit · February 7, 2017, 5:13pm

Thank you, however this is not the issue i'm having. It connects just fine when I specity the URL as a string, however, the documentation states you should be able to use an array of URL's. This does not seem to work though.

In other news, I have "solved" my problem with client certificate authentication. It seems the plugin developers neglected to add the option to the plugin at all. So I created two patch files to add it:

gist.github.com

https://gist.github.com/Slurpgeit/c9be2ad3a04659bc8c485826e2b89b1b

template.cfg.erb_keystore.patch

--- template.cfg.erb    2017-02-07 16:48:05.162151750 +0100
+++ template.cfg.erb    2017-02-07 17:35:33.359201872 +0100
@@ -24,6 +24,10 @@
 <% if ca_path %>
     cacert => "<%= ca_path %>"
 <% end %>
+<% if keystore? %>
+    keystore => "<%= keystore_path %>"
+    keystore_password => "<%= keystore_password %>"
+<% end %>

This file has been truncated. show original

gist.github.com

https://gist.github.com/Slurpgeit/c728b9464d4176b1e725a82d9244cf2a

monitoring.rb_keystore.patch

--- monitoring.rb   2017-02-07 16:48:27.129098149 +0100
+++ monitoring.rb   2017-02-07 17:52:47.013679654 +0100
@@ -16,6 +16,8 @@
                      ca_path,
                      truststore_path,
                      truststore_password,
+                     keystore_path,
+                     keystore_password,
                      collection_interval,
                      collection_timeout_interval)

This file has been truncated. show original

Hopefully someone from Elastic will pick up on this and make sure it gets fixed in a future release of X-Pack.

alvin · February 7, 2017, 6:32pm

With regards to setting the ES hosts to ship metrics to (xpack.monitoring.elasticsearch.url), we only support a single host (string) in 5.2.0. We've updated the documentation to reflect this: https://www.elastic.co/guide/en/x-pack/current/monitoring-logstash.html.

With that said, we'll be adding the ability to pass in an array of hosts soon.

Additionally, we do currently only support truststore (CA cert) and not yet keystore (client cert). I'll make sure to track this one on the Elastic end.

Let us know if you find anything else, thanks!

Slurpgeit · February 8, 2017, 9:02pm

Thank you! It's the same as with the CA certificate setting btw

system · March 8, 2017, 9:02pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.