MetricBeat HTTP module Return Code

RunningSmurf · November 27, 2019, 5:00pm

Good morning,

I want to call an API that will run a query, but only want to know the return code of the request, not the actual returned data. This is to monitor that the API is up and running. I have the following http.yml configuration:

- module: http
  metricsets:
    - json
  period: 10s
  hosts: ["https://urlofservice"]
  namespace: "json_namespace"
  path: "pathtoAPIendpoint"
  ssl.verification_mode: none
  method: "GET"
  headers:
    API_KEY: "keyfor authentication"
  json.is_array: true

Since it returns data, I get the following error:
json: cannot unmarshal string into Go value of type common.MapStr

How do I configure http.yml to just return the Return Code please? OR how do I get it to parse the return data, at which point Kibana will show me the return code?

Thanks in advance!

RunningSmurf · November 27, 2019, 8:30pm

Doing some research (unfortunately, I am not a developer, so all of this is new to me), it looks like the returned values are numbers, and cannot be converted to strings, is that it?

In addition to the questions above, is there a way to get around this conversion issue?

Kaiyan_Sheng · December 2, 2019, 11:25pm

I see http module returns http.response.code which is a string. This requires response.enabled to be configured to true: https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-http-json.html#_response_enabled

Is that what you are looking for?

RunningSmurf · December 3, 2019, 7:49pm

Hello Kaiyan,

Thank you very much for your response. I think that this gets me a step closer to what I need. However, I run into 2 issues when I include the request.enabled to true.

The first is that for one call, the returned value is single number. Apparently, the module expects a string, as it gives me a message stating that: "json: cannot unmarshal number into Go value of type common.MapStr"... How do I get around that?

The second is for a second call, which returns an array for strings, like this:
[
"K-RFIX-20GB-70cm-8W",
"K-RFIX-40GB-70cm-16W",
"K-RMOB-40GB-70cm-8W",
"K-RMOB-20GB-70cm-16W",
"K-GMAR-40GB-70cm-16W",
"K-RMAR-80GB-70cm-16W",
"K-P3Mo-10GB-70cm-8W"
]
Then the module returns the error: json: cannot unmarshal array into Go value of type common.MapStr, even though I set json.is_array: true. What am I doing wrong?

Thank you so much for your help!

Kaiyan_Sheng · December 3, 2019, 7:54pm

Could you copy paste the log messages from Metricbeat here please? Thanks!

RunningSmurf · December 3, 2019, 8:44pm

Hello Kaiyan,

Unfortunately, the log under /var/log/metricbeat does not get updated, ever since I installed it. I did set the directory and log files to 777. Maybe the log are somewhere else?

I was reading about the adding processors to convert the output from one format to another. Would that work? Something like this:

processors:
  - convert:
      fields:
        - {from: "src_port", to: "source.port", type: "integer"}
      ignore_missing: true
      fail_on_error: false

I guess that I would need to know what the field name would be. Would you know what the field name that the HTTP module uses to store the returned data? Do you think that this would work?

Thank you!

RunningSmurf · December 3, 2019, 10:33pm

I guess that I would need the from and to names of the fields, if that is indeed the way to go. Thanks again, Kaiyan!

Kaiyan_Sheng · December 4, 2019, 12:53am

For logs, I think that is the default path hmmm But if you are having issues seeing metricbeat logs, please give https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-logging.html#configuration-logging a try to setup logging in metricbeat.yml. Hopefully that will help!

RunningSmurf · December 5, 2019, 2:16pm

Thank you, Kaiyan, I was able to get logs going with this... This is what the log file returns... I put xxxx and yyyy for values for our host server and target http url - I hope that it provides enough information.

Dec 5 14:06:59 : 2019-12-05T14:06:59.778Z#011INFO#011template/load.go:88#011Template metricbeat-7.4.2 already exists and will not be overwritten.
Dec 5 14:06:59 : 2019-12-05T14:06:59.778Z#011INFO#011template/load.go:88#011Template metricbeat-7.4.2 already exists and will not be overwritten.
Dec 5 14:06:59 : 2019-12-05T14:06:59.778Z#011INFO#011[index-management]#011idxmgmt/std.go:289#011Loaded index template.
Dec 5 14:06:59 : 2019-12-05T14:06:59.778Z#011INFO#011[index-management]#011idxmgmt/std.go:289#011Loaded index template.
Dec 5 14:06:59 : 2019-12-05T14:06:59.778Z#011DEBUG#011[elasticsearch]#011elasticsearch/client.go:761#011GET http://localhost:9200/_alias/metricbeat-7.4.2
Dec 5 14:06:59 : 2019-12-05T14:06:59.778Z#011DEBUG#011[elasticsearch]#011elasticsearch/client.go:761#011GET http://localhost:9200/_alias/metricbeat-7.4.2
Dec 5 14:06:59 : 2019-12-05T14:06:59.778Z#011INFO#011[index-management]#011idxmgmt/std.go:300#011Write alias successfully generated.
Dec 5 14:06:59 : 2019-12-05T14:06:59.778Z#011INFO#011[index-management]#011idxmgmt/std.go:300#011Write alias successfully generated.
Dec 5 14:06:59 : 2019-12-05T14:06:59.778Z#011DEBUG#011[elasticsearch]#011elasticsearch/client.go:761#011GET http://localhost:9200/
Dec 5 14:06:59 : 2019-12-05T14:06:59.778Z#011DEBUG#011[elasticsearch]#011elasticsearch/client.go:761#011GET http://localhost:9200/
Dec 5 14:06:59 : 2019-12-05T14:06:59.779Z#011INFO#011pipeline/output.go:105#011Connection to backoff(elasticsearch(http://localhost:9200)) established
Dec 5 14:06:59 : 2019-12-05T14:06:59.779Z#011INFO#011pipeline/output.go:105#011Connection to backoff(elasticsearch(http://localhost:9200)) established
Dec 5 14:06:59 : 2019-12-05T14:06:59.784Z#011DEBUG#011[elasticsearch]#011elasticsearch/client.go:347#011PublishEvents: 1 events have been published to elasticsearch in 5.116281ms.
Dec 5 14:06:59 : 2019-12-05T14:06:59.784Z#011DEBUG#011[elasticsearch]#011elasticsearch/client.go:347#011PublishEvents: 1 events have been published to elasticsearch in 5.116281ms.
Dec 5 14:06:59 : 2019-12-05T14:06:59.784Z#011DEBUG#011[publisher]#011memqueue/ackloop.go:160#011ackloop: receive ack [0: 0, 1]
Dec 5 14:06:59 : 2019-12-05T14:06:59.784Z#011DEBUG#011[publisher]#011memqueue/ackloop.go:160#011ackloop: receive ack [0: 0, 1]
Dec 5 14:06:59 : 2019-12-05T14:06:59.784Z#011DEBUG#011[publisher]#011memqueue/eventloop.go:535#011broker ACK events: count=1, start-seq=1, end-seq=1
Dec 5 14:06:59 : 2019-12-05T14:06:59.784Z#011DEBUG#011[publisher]#011memqueue/eventloop.go:535#011broker ACK events: count=1, start-seq=1, end-seq=1
Dec 5 14:06:59 : 2019-12-05T14:06:59.784Z#011DEBUG#011[publisher]#011memqueue/ackloop.go:128#011ackloop: return ack to broker loop:1
Dec 5 14:06:59 : 2019-12-05T14:06:59.784Z#011DEBUG#011[publisher]#011memqueue/ackloop.go:131#011ackloop: done send ack
Dec 5 14:06:59 : 2019-12-05T14:06:59.784Z#011DEBUG#011[publisher]#011memqueue/ackloop.go:128#011ackloop: return ack to broker loop:1
Dec 5 14:06:59 : 2019-12-05T14:06:59.784Z#011DEBUG#011[publisher]#011memqueue/ackloop.go:131#011ackloop: done send ack
Dec 5 14:07:08 : 2019-12-05T14:07:08.763Z#011INFO#011module/wrapper.go:252#011Error fetching data for metricset http.json: json: cannot unmarshal number into Go value of type common.MapStr
Dec 5 14:07:08 : 2019-12-05T14:07:08.763Z#011INFO#011module/wrapper.go:252#011Error fetching data for metricset http.json: json: cannot unmarshal number into Go value of type common.MapStr
Dec 5 14:07:08 : 2019-12-05T14:07:08.764Z#011DEBUG#011[processors]#011processing/processors.go:183#011Publish event: {
Dec 5 14:07:08 : 2019-12-05T14:07:08.764Z#011DEBUG#011[processors]#011processing/processors.go:183#011Publish event: {
Dec 5 14:07:08 : "@timestamp": "2019-12-05T14:07:08.719Z",
Dec 5 14:07:08 : "@timestamp": "2019-12-05T14:07:08.719Z",
Dec 5 14:07:08 : "@metadata": {
Dec 5 14:07:08 : "@metadata": {
Dec 5 14:07:08 : "beat": "metricbeat",
Dec 5 14:07:08 : "beat": "metricbeat",
Dec 5 14:07:08 : "type": "_doc",
Dec 5 14:07:08 : "type": "_doc",
Dec 5 14:07:08 : "version": "7.4.2"
Dec 5 14:07:08 : "version": "7.4.2"
Dec 5 14:07:08 : },
Dec 5 14:07:08 : },
Dec 5 14:07:08 : "host": {
Dec 5 14:07:08 : "host": {
Dec 5 14:07:08 : "name": "xxxxx",
Dec 5 14:07:08 : "name": "xxxxx",
Dec 5 14:07:08 : "os": {
Dec 5 14:07:08 : "os": {
Dec 5 14:07:08 : "family": "redhat",
Dec 5 14:07:08 : "name": "Red Hat Enterprise Linux Server",
Dec 5 14:07:08 : "family": "redhat",
Dec 5 14:07:08 : "name": "Red Hat Enterprise Linux Server",
Dec 5 14:07:08 : "kernel": "3.10.0-693.2.2.el7.x86_64",
Dec 5 14:07:08 : "kernel": "3.10.0-693.2.2.el7.x86_64",
Dec 5 14:07:08 : "codename": "Maipo",
Dec 5 14:07:08 : "platform": "rhel",
Dec 5 14:07:08 : "codename": "Maipo",
Dec 5 14:07:08 : "platform": "rhel",
Dec 5 14:07:08 : "version": "7.4 (Maipo)"
Dec 5 14:07:08 : "version": "7.4 (Maipo)"
Dec 5 14:07:08 : },
Dec 5 14:07:08 : },
Dec 5 14:07:08 : "id": "951ca9e4d2d941c9b88f44901d916ff9",
Dec 5 14:07:08 : "id": "951ca9e4d2d941c9b88f44901d916ff9",
Dec 5 14:07:08 : "containerized": false,
Dec 5 14:07:08 : "containerized": false,
Dec 5 14:07:08 : "hostname": "xxxxx",
Dec 5 14:07:08 : "hostname": "xxxxx",
Dec 5 14:07:08 : "architecture": "x86_64"
Dec 5 14:07:08 : },
Dec 5 14:07:08 : "event": {
Dec 5 14:07:08 : "module": "http",
Dec 5 14:07:08 : "duration": 44451446,
Dec 5 14:07:08 : "dataset": "http.json"
Dec 5 14:07:08 : },
Dec 5 14:07:08 : "architecture": "x86_64"
Dec 5 14:07:08 : },
Dec 5 14:07:08 : "event": {
Dec 5 14:07:08 : "module": "http",
Dec 5 14:07:08 : "duration": 44451446,
Dec 5 14:07:08 : "dataset": "http.json"
Dec 5 14:07:08 : },
Dec 5 14:07:08 : "metricset": {
Dec 5 14:07:08 : "period": 10000,
Dec 5 14:07:08 : "metricset": {
Dec 5 14:07:08 : "period": 10000,
Dec 5 14:07:08 : "name": "json"
Dec 5 14:07:08 : "name": "json"
Dec 5 14:07:08 : },
Dec 5 14:07:08 : "service": {
Dec 5 14:07:08 : },
Dec 5 14:07:08 : "service": {
Dec 5 14:07:08 : "address": "ipmanagerit.yyyyy.com",
Dec 5 14:07:08 : "type": "http"
Dec 5 14:07:08 : "address": "ipmanagerit.yyyyy.com",
Dec 5 14:07:08 : "type": "http"
Dec 5 14:07:08 : },
Dec 5 14:07:08 : "error": {
Dec 5 14:07:08 : "message": "json: cannot unmarshal number into Go value of type common.MapStr"
Dec 5 14:07:08 : },
Dec 5 14:07:08 : "fields": {
Dec 5 14:07:08 : "app_id": "test_gb"
Dec 5 14:07:08 : },
Dec 5 14:07:08 : },
Dec 5 14:07:08 : "error": {
Dec 5 14:07:08 : "message": "json: cannot unmarshal number into Go value of type common.MapStr"
Dec 5 14:07:08 : },
Dec 5 14:07:08 : "fields": {
Dec 5 14:07:08 : "app_id": "test_gb"
Dec 5 14:07:08 : },
Dec 5 14:07:08 : "agent": {
Dec 5 14:07:08 : "ephemeral_id": "578e1ef1-a6fb-4711-b3e2-595b31d711f3",
Dec 5 14:07:08 : "hostname": "atldev-psgmon.nadev.intsatdev.com",
Dec 5 14:07:08 : "agent": {
Dec 5 14:07:08 : "ephemeral_id": "578e1ef1-a6fb-4711-b3e2-595b31d711f3",
Dec 5 14:07:08 : "hostname": "xxxxxx",
Dec 5 14:07:08 : "id": "6a26f89a-3e1d-44f7-bae6-c5e147cfbfb9",
Dec 5 14:07:08 : "version": "7.4.2",
Dec 5 14:07:08 : "type": "metricbeat"
Dec 5 14:07:08 : "id": "6a26f89a-3e1d-44f7-bae6-c5e147cfbfb9",
Dec 5 14:07:08 : "version": "7.4.2",
Dec 5 14:07:08 : "type": "metricbeat"
Dec 5 14:07:08 : },
Dec 5 14:07:08 : },
Dec 5 14:07:08 : "ecs": {
Dec 5 14:07:08 : "ecs": {
Dec 5 14:07:08 : "version": "1.1.0"
Dec 5 14:07:08 : "version": "1.1.0"
Dec 5 14:07:08 : }
Dec 5 14:07:08 : }
Dec 5 14:07:08 : }
Dec 5 14:07:08 : }
Dec 5 14:07:09 : 2019-12-05T14:07:09.770Z#011DEBUG#011[elasticsearch]#011elasticsearch/client.go:347#011PublishEvents: 1 events have been published to elasticsearch in 6.194876ms.
Dec 5 14:07:09 : 2019-12-05T14:07:09.770Z#011DEBUG#011[elasticsearch]#011elasticsearch/client.go:347#011PublishEvents: 1 events have been published to elasticsearch in 6.194876ms.
Dec 5 14:07:09 : 2019-12-05T14:07:09.770Z#011DEBUG#011[publisher]#011memqueue/ackloop.go:160#011ackloop: receive ack [1: 0, 1]
Dec 5 14:07:09 : 2019-12-05T14:07:09.770Z#011DEBUG#011[publisher]#011memqueue/ackloop.go:160#011ackloop: receive ack [1: 0, 1]
Dec 5 14:07:09 : 2019-12-05T14:07:09.770Z#011DEBUG#011[publisher]#011memqueue/eventloop.go:535#011broker ACK events: count=1, start-seq=2, end-seq=2
Dec 5 14:07:09 : 2019-12-05T14:07:09.770Z#011DEBUG#011[publisher]#011memqueue/eventloop.go:535#011broker ACK events: count=1, start-seq=2, end-seq=2
Dec 5 14:07:09 : 2019-12-05T14:07:09.770Z#011DEBUG#011[publisher]#011memqueue/ackloop.go:128#011ackloop: return ack to broker loop:1
Dec 5 14:07:09 : 2019-12-05T14:07:09.770Z#011DEBUG#011[publisher]#011memqueue/ackloop.go:128#011ackloop: return ack to broker loop:1
Dec 5 14:07:09 : 2019-12-05T14:07:09.770Z#011DEBUG#011[publisher]#011memqueue/ackloop.go:131#011ackloop: done send ack
Dec 5 14:07:09 : 2019-12-05T14:07:09.770Z#011DEBUG#011[publisher]#011memqueue/ackloop.go:131#011ackloop: done send ack
Dec 5 14:07:15 atldev-psgmon sudo: psgmon : TTY=pts/0 ; PWD=/etc/metricbeat ; USER=root ; COMMAND=/bin/systemctl stop metricbeat
Dec 5 14:07:15 atldev-psgmon polkitd[910]: Registered Authentication Agent for unix-process:9784:196377828 (system bus name :1.8254 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)

system · January 2, 2020, 2:16pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
How to query json url with http module Beats metricbeat	3	495	December 13, 2018
Metricbeat http http.json: invalid character '<' looking for beginning of value error Beats metricbeat	3	2661	August 4, 2020
Metricbeat HTTP module - custom request and fetch headers only Beats beats-module , metricbeat	2	401	June 26, 2020
Metricbeat HTTP module - request and process headers only Beats beats-module	2	436	June 21, 2020
Setup json metricset in module http Beats metricbeat	1	286	March 2, 2020

MetricBeat HTTP module Return Code

Related topics