[Filebeat 8.6.1 - httpjson] chain response.transform question

[marrc.rousseau (Marc Rousseau)]

Hello,

I'm facing an issue with response.transform and after searching during hours I'm still not able to find a solution.

I try to chain some api calls to get a list of bugs per host :

Request 1: To get list of host Ids
	Url :  /hosts

Response:

results: [
      {
	    id: 1, 
		name: server1
	  },
      {
	    id: 2,
		name : server2
	  }
	]



Chain :
Request 2.1 : To get bug list for host 1
	Url :  /hosts/1/errata

Response:

results: [
        {
            bug: 5000
		},
        {
            bug: 5100
		}
	]
}


Request 2.2 : To get bug list for host 2
	Url :  /hosts/2/errata

Response:

results": [
        {
            bug: 6000
		},
        {
            bug: 6100
		}
	]
}



Using response.split, I'm able to obtain 4 seperate documents :
message: {
            bug: 5000
		}
		
message: {
            bug: 5100
		}

message: {
            bug: 6000
		}

message: {
            bug: 6100
		}			



Final result I would like 
message: {
            name: server1,
            bug: 5000
		}
		
message: {
            name: server1,
            bug: 5100
		}

message: {
			name: server2,
			bug: 6000
		}

message: {
			name: server2,
            bug: 6100
		}	

I use this httjson filebeat configuration

filebeat.inputs:
- type: httpjson
  #tracer doesn't works for chain requests...useless
  #request.tracer.filename : /etc/filebeat/debug
  request.url: https://xxxx/api/hosts?per_page=2&thin=2&search=o*
  request.transforms:
    - set:
        target: header.Authorization
        value: 'Basic xxxx=='
  chain:
    - step:
        request.url: https://xxx/api/hosts/$.results[:].id/errata?type=security&severity=Important
        request.method: GET
        replace: $.results[:].id
        request.transforms:
          - set:
              target: header.Authorization
              value: 'Basic xxx=='

        response.split:
          target: body.results
          type: array
          transforms:
            - set:
                target: body.name
#                value: static_value -> OK
#                value: $.results[:].name   -> NOK
#                value: '[[.first_response.body.results[:].name]]'  -> NOK
#                value: '[[.first_response.body.results[0].name]]' -> NOK
#                value: '[[.first_response.body.results.name]]'   -> NOK

I don't find what should I use in value of response.split.transforms to obtain name from first response body .

Any idea ?

Thanks

[exdghost (Shourie Ganguly)]

Hi Marc,

Thanks for posting your query. At the moment the first_response should only be used for storing flat JSON structures and cannot be split upon. It was introduced for a couple of niche use_cases. It also cannot deal with NDJSON, JSON responses with the root object being an array or Gzipped JSONs. This will be worked upon the future and a PR is already under review to make the first_response object a flag based variable, updating the documentation simultaneously.

PR:- [filebeat][httpjson] - Fix first_response false positive error by making it a flag based object by ShourieG · Pull Request #34748 · elastic/beats · GitHub

[marrc.rousseau (Marc Rousseau)]

Thanks for your answer @exdghost