Support for JSON arrays in Metricbeat HTTP Module

[BennyInc (Benny)]

Continuing the discussion from Configuring multiple paths in http module:

In the old discussion I explained my usecase, which is gathering metrics from IBM Liberty.

Yesterday I downloaded a 6.3 Snapshot to test the new json.is_array behavior. While this did work in a way, it did not work fully as expected.

My input as captured from https://localhost:9443/IBMJMXConnectorREST/mbeans/WebSphere%3Aname%3DLargeThreadPool%2Ctype%3DThreadPoolStats/attributes is this:

[
{
name: "PoolSize",
value: {
value: "1563",
type: "java.lang.Integer"
}
},
{
name: "PoolName",
value: {
value: "LargeThreadPool",
type: "java.lang.String"
}
},
{
name: "ActiveThreads",
value: {
value: "1",
type: "java.lang.Integer"
}
}
]

I expected these three data points to be contained in one event sent to my elasticsearch cluster. Unfortunately, I actually receive three events, containing the payload like this:

"http": {
  "ThreadPoolStats": {
    "value": {
      "value": "LargeThreadPool",
      "type": "java.lang.String"
    },
    "name": "PoolName"
  }
}

In another event's _source:

"http": {
  "ThreadPoolStats": {
    "name": "PoolSize",
    "value": {
      "value": "2500",
      "type": "java.lang.Integer"
    }
  }
}

Is there a way to have them contained in an array or object within one single event?

[exekias (Carlos Pérez Aradros)]

Hi @BennyInc,

That is the expected behavior, storing the 3 documents as an array in Elasticsearch would make querying the data more difficult.

I guess you could merge them together later in the chain (Logstash or Ingest node), or use Logstash to do the HTTP fetching.

May I ask why do you want them in an array?

Best regards

[BennyInc (Benny)]

Having them together would allow me to compare the PoolSize with the number of ActiveThreads directly in a graph. With multiple events, one of the values will always be zero which brought up errors in my graph.

I haven't tried it again however, now that the events at last have the same timestamp.

[exekias (Carlos Pérez Aradros)]

You should be able to draw the 2 different series I think, did you give that a try?