How to use Filebeat input HttpJson

Elastic Stack Beats
1

Hi there,

i'm trying to pull data from my gitlab instance. there are 2 kinds of data: commits and projects. this is my first time using httpjson in filebeat so I don't really understand how to use it.

there are 2 APIs I used for this.
https://mygitlab.my.id/api/v4/projects and
https://mygitlab.my.id/api/v4/projects/<project_id>/repository/commits

all responses from those APIs are array [{...},{...}] and if you know, gitlab uses pagination so sometimes in a curl command I add something like this https://mygitlab.my.id/api/v4/projects/<project_id>/repository/commits?per_page=100 to make sure I got all the response from those APIs

this is my filebeat configuration. I'm using chain here because the project_id from the first API will be needed for the second API to fetch the commit

- type: httpjson
  id: fetch_project_and_commits
  enabled: true
  interval: 30s
  request:
    method: GET
    url: "https://mygitlab.my.id/api/v4/projects"
    headers:
      PRIVATE-TOKEN: "xxxxx"
    transforms:
      - set:
          target: url.params.per_page
          value: 100
      - set:
          target: url.params.page
          value: '[[.cursor.page]]'
  response:
    pagination:
      - set:
          target: cursor.page
          value: '[[.last_response.header.X-Next-Page]]'
    split:
      target: "body."
  cursor:
    paginate:
      field: cursor.page
      initial: '1'
  chain:
    # Step 2: Fetch commits for each project ID
    - step:
        request:
          method: GET
          url: "https://mygitlab.my.id/api/v4/projects/$.id/repository/commits"
          headers:
            PRIVATE-TOKEN: "xxxxxx"
          transforms:
            - set:
                target: url.params.per_page
                value: 100
            - set:
                target: url.params.page
                value: '[[.cursor.page]]'
        replace: $.id  
        response:
          pagination:
            - set:
                target: cursor.page
                value: '[[.last_response.header.X-Next-Page]]'
          split:
            target: "body."
        cursor:
          paginate:
            field: cursor.page
            initial: '1'

but I still got the same error in the pagination part.

{"log.level":"error","@timestamp":"2025-03-18T13:49:30.747+0700","log.origin":{"file.name":"instance/beat.go","file.line":1307},"message":"Exiting: Failed to start crawler: starting input failed: error while initializing input: fail to unpack the set configuration: template: :1: bad character U+002D '-' accessing 'filebeat.inputs.0.response.pagination.0.set.value' (source:'/etc/filebeat/filebeat.yml') accessing 'filebeat.inputs.0.response' (source:'/etc/filebeat/filebeat.yml')","service.name":"filebeat","ecs.version":"1.6.0"}

I don't know what to put there. the log says bad character U+002D '-' so I tried to escape it and filebeat won't start. i tried to give double quote on X-Next-Page and [[.last_response.header.X-Next-Page]] but I still got the same error

so can you help me with this? if there's anything you want to ask to give a clear view, just tell me

Thanks

2

maybe this is the reason? i found this in the documentation

The first_response object at the moment can only store flat JSON structures (i.e. no support for JSONS having an array at the root level, NDJSON or Gzipped JSON), hence it should only be used in scenarios where this is the case. Splits cannot be performed on first_response. It must be explicitly enabled by setting the flag response.save_first_response to true in the httpjson config.