Logstash jmx input plugin

hi all, I am using jmx plugin to ship WebLogic's MBeans into elasticsearch. for this purpose, following configuration has been considered in the logstash:

input {
jmx {
  path => "D:\logstash\jmx_config"
  polling_frequency => 5
  type => "jmx"
  nb_thread => 4
}
}
output {
  elasticsearch { 
    hosts => ["http://localhost:9200"]
    index => "wlbean-%{+YYYY.MM.dd}"
 }
  stdout { codec => rubydebug }
}

where the configuration file in jmx_config folder is as following:

{
  "host" : "localhost",
  "port" : 5555,
  "alias" : "MBean",
  "queries" : [
  {
    "object_name" : "java.lang:type=Threading",
	"attributes" : [ "TotalStartedThreadCount","PeakThreadCount" ],
    "object_alias" : "Threading"
  }]
}

when i starts logstash, the MBean data are shipping into elasticsearch but for each attributes there is a request, while it is expected that a request includes all attributes of object.
for example the json of one of request is as following:

{
  "_index": "wlbean-2019.05.29",
  "_type": "doc",
  "_id": "uYAwAmsB7qwfzxhygdf9",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-05-29T06:04:33.185Z",
    "@version": "1",
    "host": "localhost",
    "path": "D:\\logstash\\jmx_config",
    "metric_value_number": 90,
    "metric_path": "MBean.Threading.PeakThreadCount",
    "type": "jmx"
  },
  "fields": {
    "@timestamp": [
      "2019-05-29T06:04:33.185Z"
    ]
  },
  "sort": [
    1559109873185
  ]
}

and another json is as following:

{
  "_index": "wlbean-2019.05.29",
  "_type": "doc",
  "_id": "v4AwAvfd7uklffZhhSrlW9",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-05-29T06:04:33.185Z",
    "@version": "1",
    "host": "localhost",
    "path": "D:\\logstash\\jmx_config",
    "metric_value_number": 12536,
    "metric_path": "MBean.Threading.TotalStartedThreadCount",
    "type": "jmx"
  },
  "fields": {
    "@timestamp": [
      "2019-05-29T06:04:33.185Z"
    ]
  },
  "sort": [
    1559109873185
  ]
}

while i want to be as following, it means that all attributes reported in one event.

{
      "_index": "wlbean-2019.05.29",
      "_type": "doc",
      "_id": "v4AwAvfd7uklffZhhSrlW9",
      "_version": 1,
      "_score": null,
      "_source": {
        "@timestamp": "2019-05-29T06:04:33.185Z",
        "@version": "1",
        "host": "localhost",
        "path": "D:\\logstash\\jmx_config",
        "MBean.Threading": 
{
"TotalStartedThreadCount" : 12536
"PeakThreadCount" : 90
},
        "type": "jmx"
      },
      "fields": {
        "@timestamp": [
          "2019-05-29T06:04:33.185Z"
        ]
      },
      "sort": [
        1559109873185
      ]
    }

is it possible to have such output?

The answer to such "can you do it in logstash" questions is always "yes". You can do arbitrary transformations in a ruby filter. If you want to turn logstash into a c++ compiler it can definitely be done :smiley:

I have always thought the way the jmx filter names fields in its output is terrible and I do not like using it. I just cannot suggest a better way.

You have two issues here. One is renaming the fields. I would start with this:

ruby {
    code => '
        p = event.get("metric_path")
        vn = event.get("metric_value_number")
        vs = event.get("metric_value_string")
        if vn
            event.set(p, vn)
        else
            event.set(p, vs)
        end
    '
}
de_dot { nested => true }

The other is combining multiple events into one, which you can probably do with an aggregate filter.

Many thanks for your comprehensive comment.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.