# How to use Filebeat input HttpJson

**URL:** <https://discuss.elastic.co/t/how-to-use-filebeat-input-httpjson/376045>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [March 18, 2025, 7:16am UTC](https://discuss.elastic.co/t/how-to-use-filebeat-input-httpjson/376045 "2025-03-18T07:16:53Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![yuswanul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuswanul/32/101662_2.png) [@yuswanul](https://discuss.elastic.co/u/yuswanul)\
**Post date:** [March 18, 2025, 7:16am UTC](https://discuss.elastic.co/t/how-to-use-filebeat-input-httpjson/376045/1 "2025-03-18T07:16:54Z")

</div>

Hi there,

i'm trying to pull data from my gitlab instance. there are 2 kinds of data: commits and projects. this is my first time using httpjson in filebeat so I don't really understand how to use it.

there are 2 APIs I used for this.  
`https://mygitlab.my.id/api/v4/projects` and  
`https://mygitlab.my.id/api/v4/projects/<project_id>/repository/commits`

all responses from those APIs are array `[{...},{...}]` and if you know, gitlab uses pagination so sometimes in a curl command I add something like this `https://mygitlab.my.id/api/v4/projects/<project_id>/repository/commits?per_page=100` to make sure I got all the response from those APIs

this is my filebeat configuration. I'm using chain here because the project\_id from the first API will be needed for the second API to fetch the commit

```auto
- type: httpjson
  id: fetch_project_and_commits
  enabled: true
  interval: 30s
  request:
    method: GET
    url: "https://mygitlab.my.id/api/v4/projects"
    headers:
      PRIVATE-TOKEN: "xxxxx"
    transforms:
      - set:
          target: url.params.per_page
          value: 100
      - set:
          target: url.params.page
          value: '[[.cursor.page]]'
  response:
    pagination:
      - set:
          target: cursor.page
          value: '[[.last_response.header.X-Next-Page]]'
    split:
      target: "body."
  cursor:
    paginate:
      field: cursor.page
      initial: '1'
  chain:
    # Step 2: Fetch commits for each project ID
    - step:
        request:
          method: GET
          url: "https://mygitlab.my.id/api/v4/projects/$.id/repository/commits"
          headers:
            PRIVATE-TOKEN: "xxxxxx"
          transforms:
            - set:
                target: url.params.per_page
                value: 100
            - set:
                target: url.params.page
                value: '[[.cursor.page]]'
        replace: $.id  
        response:
          pagination:
            - set:
                target: cursor.page
                value: '[[.last_response.header.X-Next-Page]]'
          split:
            target: "body."
        cursor:
          paginate:
            field: cursor.page
            initial: '1'

```

but I still got the same error in the pagination part.

```auto
{"log.level":"error","@timestamp":"2025-03-18T13:49:30.747+0700","log.origin":{"file.name":"instance/beat.go","file.line":1307},"message":"Exiting: Failed to start crawler: starting input failed: error while initializing input: fail to unpack the set configuration: template: :1: bad character U+002D '-' accessing 'filebeat.inputs.0.response.pagination.0.set.value' (source:'/etc/filebeat/filebeat.yml') accessing 'filebeat.inputs.0.response' (source:'/etc/filebeat/filebeat.yml')","service.name":"filebeat","ecs.version":"1.6.0"}

```

I don't know what to put there. the log says `bad character U+002D '-'` so I tried to escape it and filebeat won't start. i tried to give double quote on `X-Next-Page` and `[[.last_response.header.X-Next-Page]]` but I still got the same error

so can you help me with this? if there's anything you want to ask to give a clear view, just tell me

Thanks

---

<div class="post-metadata">

**Author:** ![yuswanul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuswanul/32/101662_2.png) [@yuswanul](https://discuss.elastic.co/u/yuswanul)\
**Post date:** [March 18, 2025, 12:17pm UTC](https://discuss.elastic.co/t/how-to-use-filebeat-input-httpjson/376045/2 "2025-03-18T12:17:41Z")

</div>

maybe this is the reason? i found this in the [documentation](https://www.elastic.co/guide/en/beats/filebeat/8.11/filebeat-input-httpjson.html#response-pagination)

> The first\_response object at the moment can only store flat JSON structures (i.e. no support for JSONS having an array at the root level, NDJSON or Gzipped JSON), hence it should only be used in scenarios where this is the case. Splits cannot be performed on first\_response. It must be explicitly enabled by setting the flag response.save\_first\_response to true in the httpjson config.