Logstash jmx input plugin

[sahere37 (LR)]

hi all, I am using jmx plugin to ship WebLogic's MBeans into elasticsearch. for this purpose, following configuration has been considered in the logstash:

input {
jmx {
  path => "D:\logstash\jmx_config"
  polling_frequency => 5
  type => "jmx"
  nb_thread => 4
}
}
output {
  elasticsearch { 
    hosts => ["http://localhost:9200"]
    index => "wlbean-%{+YYYY.MM.dd}"
 }
  stdout { codec => rubydebug }
}

where the configuration file in jmx_config folder is as following:

{
  "host" : "localhost",
  "port" : 5555,
  "alias" : "MBean",
  "queries" : [
  {
    "object_name" : "java.lang:type=Threading",
	"attributes" : [ "TotalStartedThreadCount","PeakThreadCount" ],
    "object_alias" : "Threading"
  }]
}

when i starts logstash, the MBean data are shipping into elasticsearch but for each attributes there is a request, while it is expected that a request includes all attributes of object.
for example the json of one of request is as following:

{
  "_index": "wlbean-2019.05.29",
  "_type": "doc",
  "_id": "uYAwAmsB7qwfzxhygdf9",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-05-29T06:04:33.185Z",
    "@version": "1",
    "host": "localhost",
    "path": "D:\\logstash\\jmx_config",
    "metric_value_number": 90,
    "metric_path": "MBean.Threading.PeakThreadCount",
    "type": "jmx"
  },
  "fields": {
    "@timestamp": [
      "2019-05-29T06:04:33.185Z"
    ]
  },
  "sort": [
    1559109873185
  ]
}

and another json is as following:

{
  "_index": "wlbean-2019.05.29",
  "_type": "doc",
  "_id": "v4AwAvfd7uklffZhhSrlW9",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-05-29T06:04:33.185Z",
    "@version": "1",
    "host": "localhost",
    "path": "D:\\logstash\\jmx_config",
    "metric_value_number": 12536,
    "metric_path": "MBean.Threading.TotalStartedThreadCount",
    "type": "jmx"
  },
  "fields": {
    "@timestamp": [
      "2019-05-29T06:04:33.185Z"
    ]
  },
  "sort": [
    1559109873185
  ]
}

while i want to be as following, it means that all attributes reported in one event.

{
      "_index": "wlbean-2019.05.29",
      "_type": "doc",
      "_id": "v4AwAvfd7uklffZhhSrlW9",
      "_version": 1,
      "_score": null,
      "_source": {
        "@timestamp": "2019-05-29T06:04:33.185Z",
        "@version": "1",
        "host": "localhost",
        "path": "D:\\logstash\\jmx_config",
        "MBean.Threading": 
{
"TotalStartedThreadCount" : 12536
"PeakThreadCount" : 90
},
        "type": "jmx"
      },
      "fields": {
        "@timestamp": [
          "2019-05-29T06:04:33.185Z"
        ]
      },
      "sort": [
        1559109873185
      ]
    }

is it possible to have such output?

[Badger]

@sahere37

is it possible to have such output?

The answer to such "can you do it in logstash" questions is always "yes". You can do arbitrary transformations in a ruby filter. If you want to turn logstash into a c++ compiler it can definitely be done

I have always thought the way the jmx filter names fields in its output is terrible and I do not like using it. I just cannot suggest a better way.

You have two issues here. One is renaming the fields. I would start with this:

ruby {
    code => '
        p = event.get("metric_path")
        vn = event.get("metric_value_number")
        vs = event.get("metric_value_string")
        if vn
            event.set(p, vn)
        else
            event.set(p, vs)
        end
    '
}
de_dot { nested => true }

The other is combining multiple events into one, which you can probably do with an aggregate filter.

[sahere37 (LR)]

Many thanks for your comprehensive comment.