[Filebeat 8.6.1 - httpjson] invalid memory address or nil pointer dereference

[marrc.rousseau (Marc Rousseau)]

Hello,

I try to use httpjson with "chain" and "steps" but I'm stuck on this error "invalid memory address or nil pointer dereference".

Here is what I did:

1. a simple httpjson to call an API and get host IDs

filebeat.inputs:
- type: httpjson
  request.url: https://xxx/api/hosts?per_page=2&thin=true
  request.method: GET
  auth.basic:
    user: xxx
    password: xxx
output.console:
  pretty: true

Result is

{
    "total": 62,
    "subtotal": 62,
    "page": 1,
    "per_page": 2,
    "search": null,
    "sort": {
        "by": null,
        "order": null
    },
    "results": [
        {
            "id": 193,
            "name": "xxxx"
        },
        {
            "id": 101,
            "name": "yyy"
        }
    ]
}

Looks good ....

2. now I want to use these ids to do a second api call

filebeat.inputs:
- type: httpjson
  request.url: https://xxx/api/hosts?per_page=2&thin=true
  request.method: GET
  auth.basic:
    user: xxx
    password: xxx
  chain:
    - step:
        request.url: https://xxx/api/hosts/$.results[:].id/errata
        request.method: GET
        replace: $.results[:].id
output.console:
  pretty: true

And got this error when I start "filebeat -c test.yml -e"

{"log.level":"error","@timestamp":"2023-02-23T19:45:44.138+0100","log.logger":"input.httpjson-stateless","log.origin":{"file.name":"compat/compat.go","file.line":124},"message":"Input 'httpjson-stateless' failed with: runtime error: invalid memory address or nil pointer dereference","service.name":"filebeat","id":"E49CFF25B852F300","ecs.version":"1.6.0"}

Any idea ?
Any help is welcome

 
 
 

Note :

Looks like there's a typo in ELK documentation : https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-httpjson.html#chain-step-replace

(...)
  chain:
    # second call
    - step:
      request.url: https://example.com/services/data/v1.0/$.records[:].id/export_ids
(...)

This code will create an error :

Exiting: Failed to start crawler: starting input failed: error while initializing input: both step & while blocks in a chain cannot be empty accessing 'filebeat.inputs.0' (source:'/etc/filebeat/test4.yml')

You need to add two more white spaces in front of lines under "- step:"

(...)
  chain:
    # second call
    - step:
        request.url: https://example.com/services/data/v1.0/$.records[:].id/export_ids
(...)

[marrc.rousseau (Marc Rousseau)]

Finally found a workaround
Looks like auth.basic is not compatible with chain/step. Bug or documentation misleading ?

This one works

filebeat.inputs:
- type: httpjson
  request.url: https://xxx/api/hosts?per_page=2&thin=2
  request.transforms:
    - set:
        target: header.Authorization
        value: 'Basic xxxxxxxxxxxxxxxx'
  chain:
    - step:
        request.url: https://xxx/api/hosts/$.results[:].id/errata?type=security
        request.method: GET
        replace: $.results[:].id
        request.transforms:
          - set:
              target: header.Authorization
              value: 'Basic xxxxxxxxxx'
output.console:
  pretty: true