Logstash and Jboss

Elastic Stack Logstash
1

I have newly setup a JBOSS JVM server and would like to use ELK to visualize the Server Host Usage Metrics. Though I can see there are server logs but seem are not showing the Host Usage Metrics, i.g. cpu usage, disk usage, heap usage, etc.

Would anyone can tell me how can I collect these metrics with logstash and any simple conf file to collect them?

2

Logstash doesn't collect host metrics by itself. Look into e.g. Topbeat or collectd.

3

So is there anyway to collect the metrics and feed into logstash?

4

Did you look into Topbeat or collectd as I suggested?

5

Yes, but seem they are collecting the whole server metrics instead of a specific application. I can collect metrics of the host but not JBOSS appplication because I am running Jboss as an application in my Redhat Server

6

Oh. I'm not familiar with a tool that collects metrics for particular applications, unless you're using JMX or if you're running Docker containers that can be monitored with cAdvisor.

7

Is that JMX can be used to read the metrics so?

8

JMX is an interface for reading metrics about what goes on in a JVM. Logstash has a JMX input plugin.

9

I have install jmx inpuut and make a jboss configuration file in logstash as follow:

input {
jmx{
path => "/home/lau/Downloads/logstash/jmx"
polling_frequency => 30
type => "jmx"
nb_thread => 4
}
}

output {
elasticsearch { hosts => slocalhost }
}

Then, inside /home/lau/Downloads/logstash/jmx, I have another jmx.conf file as the follow

{
"url" : "service:jmx:remoting-jmx://localhost:9999",
"username" : "jboss",
"password": "123456",
"alias" : "jmx.instance1.elasticsearch",
"host" : "localhost",
"port" : 9999,
"queries" : [
{
"object_name" : "java.lang:type=Memory",
"attributes" : [ "HeapMemoryUsage", "NonHeapMemoryUsage" ],
"object_alias" : "Memory"
}, {
"object_name" : "java.lang:type=Runtime",
"attributes" : [ "Uptime", "StartTime" ],
"object_alias" : "Runtime"
}, {
"object_name" : "java.lang:type=Threading",
"attributes" : [ "ThreadCount", "TotalStartedThreadCount", "DaemonThreadCount", "PeakThreadCount" ],
"object_alias" : "Threading"
}, {
"object_name" : "java.lang:type=OperatingSystem",
"attributes" : [ "OpenFileDescriptorCount", "FreePhysicalMemorySize", "CommittedVirtualMemorySize", "FreeSwapSpaceSize", "ProcessCpuLoad", "ProcessCpuTime", "SystemCpuLoad", "TotalPhysicalMemorySize", "TotalSwapSpaceSize", "SystemLoadAverage" ],
"object_alias" : "OperatingSystem"
} ]
}

When I run the logstash conf file, it gives me the following result:

Create queue dispatching JMX requests to threads {:level=>:info}
Compile regexp for group alias object replacement {:level=>:info}
Initialize 4 threads for JMX metrics collection {:level=>:info}
Loading configuration files in path {:path=>"/home/lau/Downloads/logstash/jmx", :level=>:info}
Issue with configuration file {:file=>"/home/lau/Downloads/logstash/jmx/jmx.conf~", :validation_errors=>["Bad type for parameter 'port', expecting 'Fixnum', found 'String'."], :level=>:warn}
Unsupported protocol: remoting-jmx {:level=>:error}
javax.management.remote.JMXConnectorFactory.newJMXConnector(javax/management/remote/JMXConnectorFactory.java:357)
javax.management.remote.JMXConnectorFactory.connect(javax/management/remote/JMXConnectorFactory.java:267)
java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:606)
RUBY.create_connection(/home/lau/Downloads/logstash/vendor/bundle/jruby/1.9/gems/jmx4r-0.1.4/lib/jmx4r.rb:221)
RUBY.connection(/home/lau/Downloads/logstash/vendor/bundle/jruby/1.9/gems/jmx4r-0.1.4/lib/jmx4r.rb:138)
RUBY.thread_jmx(/home/lau/Downloads/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-jmx-2.0.4/lib/logstash/inputs/jmx.rb:210)
RUBY.run(/home/lau/Downloads/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-jmx-2.0.4/lib/logstash/inputs/jmx.rb:321)
java.lang.Thread.run(java/lang/Thread.java:745) {:level=>:error}
Using mapping template from {:path=>nil, :level=>:info}
Attempting to install template {:manage_template=>{"template"=>"logstash-*", "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "omit_norms"=>true}, "dynamic_templates"=>[{"message_field"=>{"match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}, "fields"=>{"raw"=>{"type"=>"string", "index"=>"not_analyzed", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"string", "index"=>"not_analyzed"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"float"}, "longitude"=>{"type"=>"float"}}}}}}}, :level=>:info}
slocalhost: Name or service not known {:class=>"Manticore::ResolutionFailure", :level=>:error}
New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["slocalhost"], :level=>:info}
Starting pipeline {:id=>"main", :pipeline_workers=>8, :batch_size=>125, :batch_delay=>5, :max_inflight=>1000, :level=>:info}

I have no idea what's wrong with the file. Please help. Thanks sincerely.

10

elasticsearch { hosts => slocalhost }

Is your ES host really named "slocalhost"?

11

Yes, I have correct this mistake but the problems are still same on except the line of error for this dismissed
Other mistakes are still there

12

I have tried to workout some mistakes, but it keeps giving me the following issues? Is there any solution for it?

Great Thanks.

Unsupported protocol: remoting-jmx {:level=>:error}
javax.management.remote.JMXConnectorFactory.newJMXConnector(javax/management/remote/JMXConnectorFactory.java:357)
javax.management.remote.JMXConnectorFactory.connect(javax/management/remote/JMXConnectorFactory.java:267)
java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:606)
RUBY.create_connection(/home/lau/Downloads/logstash/vendor/bundle/jruby/1.9/gems/jmx4r-0.1.4/lib/jmx4r.rb:221)
RUBY.connection(/home/lau/Downloads/logstash/vendor/bundle/jruby/1.9/gems/jmx4r-0.1.4/lib/jmx4r.rb:138)
RUBY.thread_jmx(/home/lau/Downloads/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-jmx-2.0.4/lib/logstash/inputs/jmx.rb:210)
RUBY.run(/home/lau/Downloads/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-jmx-2.0.4/lib/logstash/inputs/jmx.rb:321)
java.lang.Thread.run(java/lang/Thread.java:745) {:level=>:error}
13

I see this is an old thread but maybe there has been some progress on this?

I want to do exactly that, get VM/APP metrics from Wildfly 10 (jboss) using jmx into logstash:

Setting:

  • Wildfly running on a VM on 10.0.0.2.
  • ELK running on a VM on 10.0.0.5 ( see Vagrant config on github).
  • I'm using logstash-input-jmx plugin.
  • All ports are exported, so 10.0.0.5 can see 10.0.0.2 (tested with curl), specially port 9990 where jmx is on.
  • I can connect to wildfly using the jconsole on the URL service:jmx:remote+http://localhost:9990 or using the deprecated URL-style: service:jmx:http-remoting-jmx://localhost:9990 (will complain at the beginning though).

Problem:

  • when using one of those URLs, logstash complains with
[2017-06-22T17:31:40,724][ERROR][logstash.inputs.jmx]  Unsupported protocol: remote+http

or

[2017-06-22T17:33:25,929][ERROR][logstash.inputs.jmx ] Unsupported protocol: http-remoting-jmx`
  • when using no url in the config, but only host and port it says:
[2017-06-22T17:31:10,755][ERROR][logstash.inputs.jmx] Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
        java.net.SocketTimeoutException: Read timed out]
[2017-06-22T17:31:10,756][ERROR][logstash.inputs.jmx      ] javax.management.remote.rmi.RMIConnector.connect(javax/management/remote/rmi/RMIConnector.java:369)
javax.management.remote.JMXConnectorFactory.connect(javax/management/remote/JMXConnectorFactory.java:270)
java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)
RUBY.create_connection(/opt/logstash/vendor/bundle/jruby/1.9/gems/jmx4r-0.1.4/lib/jmx4r.rb:220)
RUBY.connection(/opt/logstash/vendor/bundle/jruby/1.9/gems/jmx4r-0.1.4/lib/jmx4r.rb:137)
RUBY.thread_jmx(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-jmx-3.0.1/lib/logstash/inputs/jmx.rb:210)
RUBY.run(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-jmx-3.0.1/lib/logstash/inputs/jmx.rb:321)
java.lang.Thread.run(java/lang/Thread.java:745)

Which means to me, that they are not speaking the right protocol. Since I get Jconsole to connect to the VM, I assume, that logstash.inputs.jmx is using a different protocol (when it obviously hates the URL-Styles above). So how should the URL look like?

Another possibility is, that I am missing something in the Java side, since the Jconsole only works when jboss-cli-client.jar is in the class path (see this article). It could be possible, that there is some vendor magic involved. Could it be, that the plugin is using java jmx connectors, and I can just put the lib somewhere so it is in the class path?

14

UPDATE:

I tested it putting the jboss-cli-client.jar it into /opt/logstash/vendor/jruby/lib/jboss-cli-client.jar and it worked.

This looks a little too hacky for me. Is there a proper way to add some libraries to logstash's classpath from a different directory?

I have no knowledge of ruby, so make your answer child-proof please.

I also updated my camminati/vagrant-docker-elk if you want to try stuff.

15

I'm running into the same issue, but after following your configuration steps I'm receiving "Unsupported protocol: remote+http". I'm running it on Windows 10, using version 6.2.3 of Logstash, with the jboss-cli-client.jar from v10.0.0.0 of Wildfly.

16

Any one found a solution for this one?

I followed camminati steps. Still can't get a way around it. Unsupported protocol error.