Error metricbeat sending json data to elastic search

Hi

I haven't tried this before. I am trying to send jenkins build data to elastic search. I am seeing some error.

Elastic stack : 7.2

My jenkins url response is like this.

http://master.example.com//job/Federated_Install/lastBuild/api/json?depth=0&username=user&password=********8pretty=true&tree=building,duration,result,timestamp,url,actions[parameters[name,value]]

{
  "_class" : "hudson.model.FreeStyleBuild",
  "actions" : [
    {
      "_class" : "hudson.model.ParametersAction",
      "parameters" : [
        {
          "_class" : "hudson.model.StringParameterValue",
          "name" : "BUILDNUMBER",
          "value" : "12571"
        }
      ]
    },
    {
      "_class" : "hudson.model.CauseAction"
    },
    {
      "_class" : "jenkins.metrics.impl.TimeInQueueAction"
    },
    {
      
    },
    {
      
    },
    {
      
    },
    {
      
    },
    {
      
    },
    {
      
    },
    {
      
    }
  ],
  "building" : false,
  "result" : "SUCCESS",
  "timestamp" : 1564313909399,
  "url" : "https://master.example.com/job/Federated_Install_10.8/lastBuild/api/json?depth=0&pretty=true&tree=building,duration,result,url,actions[parameters]"
}

I want to send these build parameters to elastic search

"name" : "BUILDNUMBER",
"value" : "12571"
"building" : false,
"result" : "SUCCESS",
"timestamp" : 1564313909399,
"url" : "https://master.example.com/job/Federated_Install_10.8/123"

I am seeing this error when I configured my url in to metricbeat http module. You can see this in the images attached. But I am not seeing the data what I am expecting.

**t** error.message invalid character '<' looking for beginning of value

Here is my http.yml configuration. Please suggest if I need to modify or update my configuration.

- module: http
  metricsets:
    - json
  period: 10s
  hosts: ["master.example.com"]
  namespace: "json_test"
  path: "/job/Federated_Install/lastBuild/api/json?depth=0&username=user&password=********&pretty=true&tree=building,duration,result,timestamp,url"
  #body: ""
  method: "GET"

@warkolm Can you please guide or redirect to some one who can help me. Thanks in advance.

First i think the request should be method: "POST". Second i would try to put the params into the body like this {"depth" : "0", "username": "user",... }.

@maddin2016 Thanks for your reply. Let me try that. Will get back to you if it does not work.

I am still seeing the same error. Is it because my json has dictionary in output.

Seems like a duplicate of this

How does you current http.yml config looks like? What you get with curl? And are you able to catch the request on the jenkins side with wireshark, etc.?

@maddin2016 It strange. Same url give me a json output in browser but using curl I get a html. btw its a jenkins url. Any idea how I can parse this output to Elastic search?

curl -vvvv http://master:8080/view/Server%20Setups/job/Archive/lastBuild/api/json?
* About to connect() to setupmaster port 8080 (#0)
*   Trying 1.4.54...
* Connected to master port 8080 (#0)
> GET /view/Server%20Setups/job/Archive/lastBuild/api/json? HTTP/1.1
> User-Agent: curl/7.29.0
> Host: master:8080
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Thu, 01 Aug 2019 16:24:21 GMT
< X-Content-Type-Options: nosniff
< X-Jenkins: 2.150.1
< X-Jenkins-Session: 16f8ec6e
< Content-Type: application/json;charset=utf-8
< Content-Length: 1195
< Server: Jetty(9.4.z-SNAPSHOT)
<
{"_class":"hudson.model.FreeStyleBuild","actions":[{"_class":"hudson.model.ParametersAction","parameters":[{"_class":"hudson.model.StringParameterValue","name":"BUILDNUMBER","value":"12578"}]},{"_class":"hudson.model.CauseAction","causes":[{"_class":"hudson.model.Cause$RemoteCause","shortDescription":"Started by remote host **.**.**.**","addr":"**.**.**.**","note":null}]},{"_class":"jenkins.metrics.impl.TimeInQueueAction","blockedDurationMillis":0,"blockedTimeMillis":0,"buildableDurationMillis":1,"buildableTimeMillis":1,"buildingDurationMillis":81735,"executingTimeMillis":81735,"executorUtilization":1.0,"subTaskCount":0,"waitingDurationMillis":7771,"waitingTimeMillis":7771},{},{},{},{},{},{},{},{},{}],"artifacts":[],"building":false,"description":null,"displayName":"#12578","duration":81735,"estimatedDuration":76463,"executor":null,"fullDisplayName":"Archive #12578","id":"1836","keepLog":false,"number":1836,"queueId":70320,"result":"SUCCESS","timestamp":1564617659914,"url":"http://master:8080/* Connection #0 to host master left intact
view/Server%20Setups/job/Archive/1836/","builtOn":"setupbuild10","changeSet":{"_class":"hudson.scm.EmptyChangeLogSet","items":[],"kind":null},"culprits":[]}

How does your curl command looks like?

Can you try curl -XPOST "yourURLhere" -H "Content-Type: application/json" -d '{"depth": "0", "username": "user", restOfParamsHere}'

I donot get any output. I just used the main url did not add any filters. I am running this command on Linux Redhat Server. This Jenkins server doesn't need any user and password.

My main url will look like this. Not sure how to add these parameters in curl command.
http://master:8080/view/Server%20Setups/job/server/lastBuild/api/json?pretty=true&tree=actions[parameters[name,value]],building,result,timestamp,url

user@318:/tmp ssh$ curl -XPOST "http://master:8080/view/Server%20Setups/job/server/lastBuild/api" -H "Content-Type: application/json" -d '{"depth": "0"}'
user@318:/tmp ssh$

I'm not familiar with Jenkins but you should have a look here. I think it is a problem with the url you present to the http.module.

This gives me a json output not sure how to configure this in Http.yml.

curl -X POST http://master:8080/view/Server%20Setups/job/server/lastBuild/api/json --data-urlencode json='{"parameters":"actions"}'

{"_class":"hudson.model.FreeStyleBuild","actions":[{"_class":"hudson.model.ParametersAction","parameters":[{"_class":"hudson.model.StringParameterValue","name":"BUILDNUMBER","value":"12579"}]},{"_class":"hudson.model.CauseAction","causes":[{"_class":"hudson.model.Cause$RemoteCause","shortDescription":"Started by remote host ..*","addr":"..***","note":null}]},{"_class":"jenkins.metrics.impl.TimeInQueueAction","blockedDurationMillis":0,"blockedTimeMillis":0,"buildableDurationMillis":2,"buildableTimeMillis":2,"buildingDurationMillis":0,"executingTimeMillis":0,"executorUtilization":1.0,"subTaskCount":0,"waitingDurationMillis":7024,"waitingTimeMillis":7024},{},{},{},{},{},{}],"artifacts":,"building":true,"description":null,"displayName":"#12579","duration":0,"estimatedDuration":5972709,"executor":{},"fullDisplayName":"server #12579","id":"1944","keepLog":false,"number":1944,"queueId":70491,"result":null,"timestamp":1564677945650,"url":"http://master:8080/view/Server%20Setups/job/server/1944/","builtOn":"setupbuild12","changeSet":{"_class":"hudson.scm.EmptyChangeLogSet","items":[],"kind":null},"culprits":}a

@maddin2016 I have made some modifications in my http.yml file now I see this result. Looks like its not parsing the json output.

Looks like json is parsing the top level values but unable to parse the values in the bottom tree level.

Do you have set json.is_array in your config set to true?

Currently it is set to false. Let me set to true.

If I set it to true I get a different error.

`

> **t** error.message json: cannot unmarshal object into Go value of type []common.MapStr

`