[HTTP-Module] json: cannot unmarshal object into Go value of type []common.MapStr

[PeterPain]

Hey there.
Trying to pull data from a device which supports http protocol and has the option to query for
different parameters in JSON-Format. Because the device puts the data in an array i have json.is_array enabled.

Example Device URL:
http://localhost:80/json.do?_P_SUM3,_SYSTIME
Output looks like this:
{ "_P_SUM3": [413.7307,"W"], "_SYSTIME": [1557142986.0000,"sec"]}

Here are my http.configs:

- module: http
  metricsets:
    - json
  period: 10s
  hosts: ["localhost:80/json.do?_P_SUM3,_SYSTIME"]
  namespace: "Janitza_Monti2"
  path: ""
  #body: ""
  method: "GET"
  #username: "user"
  #password: "secret"
  #request.enabled: true
  #response.enabled: true
  json.is_array: true
  #dedot.enabled: false

Im getting this error message:

"error": {
      "message": "json: cannot unmarshal object into Go value of type []common.MapStr"
    },

thx for any help

[martinr_ubi (Martin R.)]

Hi,

I think you're misinterpreting the json.is_array setting in that context.
ref

What you get:

{ "_P_SUM3": [413.7307,"W"], "_SYSTIME": [1557142986.0000,"sec"]}

Is not an array.
This would be an array for json.is_array:

[ {"name": "David" }, { "name": "Robert"} ]

You should remove json.is_array.
But I also recommend you read this:
https://www.elastic.co/guide/en/elasticsearch/reference/current/array.html

Because as far as I'm concerned you will not be able to work with that document because the values are arrays of different types. In ES you can easily push an array for the value of a field like that:

{"field1": ["V","W"], "field2": [1,2] }

But they(values in the array) have to be of the same type, as above, where in your case they are arrays containing values of different types. I do recommend you make your own tests, maybe there is a combination of settings to make ES accept the first value of the array and then refuse the second one... but ignore the error on the bad type of the second one and still index the document? Not sure, I have never tried to index such array of differing types(because it doesn't work).

If that is indeed a dead end because of the types in the arrays the next step would depend on where it is easier for you to intervene in a custom manner... One option is to ship this into an ingest pipeline like this:
https://www.elastic.co/guide/en/elasticsearch/reference/7.0/script-processor.html
Where possibilities are infinite*(TM). And fix the problem there.

Maybe someone else has a suggestion too.

[PeterPain]

Hi Martin,
thanks alot for the information.
Yeah i tried pushing the data without the json.is_array-setting.
when using the same settings as above, i cant see any data output to elastic/kibana.
I think because these types of array doesnt work, as you mentioned.
I also tried pushing only one parameter but this doesn't work aswell.

I should get this array:
{ "_P_SUM3": [413.7307,"W"],
But this is what the http-module push to elastic.

{
  "_index": "metricbeat-7.0.0-2019.04.17-000001",
  "_type": "_doc",
  "_id": "i4cckWoB5TWw_WhLMMKM",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-05-07T07:05:03.738Z",
    "ecs": {
      "version": "1.0.0"
    },
    "host": {
      "name": "hostname"
    },
    "agent": {
      "type": "metricbeat",
      "ephemeral_id": "2f0b9fd0-d781-40fc-9982-bb4bbe467d0e",
      "hostname": "hostname",
      "id": "de164d55-8cb5-4d01-ad5a-c68afe290c08",
      "version": "7.0.0"
    },
    "http": {
      "Janitza_Monti2": {
        "_P_SUM3=": [
          "var not found !",
          ""
        ]
      }
    },
    "event": {
      "dataset": "http.Janitza_Monti2",
      "module": "http",
      "duration": 25931200
    },
    "metricset": {
      "name": "json"
    },
    "service": {
      "address": "lokalhost:80",
      "type": "http"
    }
  },
  "fields": {
    "@timestamp": [
      "2019-05-07T07:05:03.738Z"
    ]
  },
  "sort": [
    1557212703738
  ]
}

Im not sure how the http module parse the data. As you can see the Parameter "_P_SUM3" says ""var not found !", this is the message you get from the device when the parameter name is wrong. I was wondering where the equal sign in _P_SUM3= in the doc comes from.
I will give the script processor a try.
If anyone else has a suggestion too, I would be very grateful.

[PeterPain]

Hey Martin,

i was successful with using the script Processor.
Thanks again for the help.