# Filebeat HTTPJSON - Dynamic query in body **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [April 2, 2025, 5:02pm UTC](https://discuss.elastic.co/t/filebeat-httpjson-dynamic-query-in-body/376707 "2025-04-02T17:02:30Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![Jannus\_C](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jannus_c/32/139984_2.png) [@Jannus\_C](https://discuss.elastic.co/u/Jannus_C)\ **Post date:** [April 2, 2025, 5:02pm UTC](https://discuss.elastic.co/t/filebeat-httpjson-dynamic-query-in-body/376707/1 "2025-04-02T17:02:30Z")
Hello, I'm testing filebeat + httpjson against an elasticsearch instance. I'm able to retrieve all the data via scrolling using a "fixed query", ```auto - type: httpjson config_version: 2 interval: 5m request.url: https://192.168.3.10:9200/test-index/_search?scroll=5m request.method: POST request.ssl.verification_mode: none auth.basic.user: "${ES_USR}" auth.basic.password: "${ES_PWD}" request.body: query: bool: filter: term: month: 2 response.split: target: body.hits.hits processors: - decode_json_fields: fields: ["message"] target: "test" - add_tags: tags: ["5nd","POST","scroll","query"] target: "stage" index: httpjson-test pipeline: clean_httpjson response.pagination: - set: target: url.value value: https://192.168.3.10:9200/_search/scroll - set: target: url.params.scroll_id value: '[[.last_response.body._scroll_id]]' - set: target: body.scroll value: 5m ``` What I want to do now is to specify a dynamic query in the body. Something like this ```auto { "query": { "range": { "@timestamp": { "gte": "now-7d" } } } ``` Is it possible? Thank you very much in advance ---
**Author:** ![exdghost](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exdghost/32/100274_2.png) [@exdghost](https://discuss.elastic.co/u/exdghost)\ **Post date:** [April 9, 2025, 7:01am UTC](https://discuss.elastic.co/t/filebeat-httpjson-dynamic-query-in-body/376707/3 "2025-04-09T07:01:45Z")
> [@Jannus\_C](#): > > `gte` you can do something like ```auto request.transforms: - set: target: body.query.range value: '@timestamp gte [[formatDate (now (parseDuration "-7d")) "2006-01-02T15:04:05.99Z"]]' ``` ---
**Author:** ![Jannus\_C](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jannus_c/32/139984_2.png) [@Jannus\_C](https://discuss.elastic.co/u/Jannus_C)\ **Post date:** [April 13, 2025, 9:26pm UTC](https://discuss.elastic.co/t/filebeat-httpjson-dynamic-query-in-body/376707/4 "2025-04-13T21:26:55Z")
Thank you @exdghost I tried but I got the following error (basically [range] query malformed) ```auto {"log.level":"error","@timestamp":"2025-04-13T23:14:54.165+0200","log.logger":"input.httpjson-stateless","log.origin":{"function":"github.com/elastic/beats/v7/x-pack/filebeat/input/httpjson.run.func1","file.name":"httpjson/input.go","file.line":181},"message":"Error while processing http request: failed to collect first response: failed to execute http POST: server responded with status code 400: {\"error\":{\"root_cause\":[{\"type\":\"parsing_exception\",\"reason\":\"[range] query malformed, no start_object after query name\",\"line\":1,\"col\":19}],\"type\":\"parsing_exception\",\"reason\":\"[range] query malformed, no start_object after query name\",\"line\":1,\"col\":19},\"status\":400}","service.name":"filebeat","id":"168CAE93553DF9E9","input_url":"https://192.168.3.10:9200/nessus-saas/_search?scroll=5m","ecs.version":"1.6.0"} {"log.level":"error","@timestamp":"2025-04-13T23:14:57.086+0200","log.logger":"add_cloud_metadata","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata","file.name":"add_cloud_metadata/providers.go","file.line":190},"message":"add_cloud_metadata: received error for provider gcp: failed requesting gcp metadata: Get \"http://169.254.169.254/computeMetadata/v1/?recursive=true&alt=json\": dial tcp 169.254.169.254:80: i/o timeout","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2025-04-13T23:14:57.087+0200","log.logger":"add_cloud_metadata","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).init.func1","file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":100},"message":"add_cloud_metadata: hosting provider type not detected.","service.name":"filebeat","ecs.version":"1.6.0"} ``` Here is my complete config ```auto - type: httpjson config_version: 2 interval: 5m request.url: https://192.168.3.10:9200/test-index/_search?scroll=5m request.method: POST request.ssl.verification_mode: none auth.basic.user: "${ES_USR}" auth.basic.password: "${ES_PWD}" request.body: query: range: "@timestamp": gte: now response.split: target: body.hits.hits processors: - decode_json_fields: fields: ["message"] target: "test" - add_tags: tags: ["8nd","POST","scroll","query"] target: "stage" index: httpjson-test pipeline: clean_httpjson request.transforms: - set: target: body.query.range value: '@timestamp gte [[formatDate (now (parseDuration "-90d")) "2006-01-02T15:04:05.99Z"]]' response.pagination: - set: target: url.value value: https://192.168.3.10:9200/_search/scroll - set: target: url.params.scroll_id value: '[[.last_response.body._scroll_id]]' - set: target: body.scroll value: 5m ``` Thank you very much in advance ---
**Author:** ![Jannus\_C](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jannus_c/32/139984_2.png) [@Jannus\_C](https://discuss.elastic.co/u/Jannus_C)\ **Post date:** [April 13, 2025, 9:40pm UTC](https://discuss.elastic.co/t/filebeat-httpjson-dynamic-query-in-body/376707/5 "2025-04-13T21:40:12Z")
@exdghost , doing this "partially works" ```auto request.transforms: - set: target: body.query.range.@timestamp.gte value: '[[(now (parseDuration "-900h"))]]' ``` `:\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":[{\"shard\":0,\"index\":\"nessus-saas-2025.02.20-000001\",\"node\":\"SeT8Ru0RSveuDJNbEB-Ajw\",\"reason\":{\"type\":\"parse_exception\",\"reason\":\"failed to parse date field [2025-03-07 09:29:32.853043701 +0000 UTC] with format [strict_date_optional_time||epoch_millis]: [failed to parse date field [2025-03-07 09:29:32.853043701 +0000 UTC] with format [strict_date_optional_time||epoch_millis]]\` The query is well formed but I got an error regarding to the timestamp format. I tried several ways to use the formatDate but none of them worked. ---
**Author:** ![exdghost](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exdghost/32/100274_2.png) [@exdghost](https://discuss.elastic.co/u/exdghost)\ **Post date:** [April 14, 2025, 8:29am UTC](https://discuss.elastic.co/t/filebeat-httpjson-dynamic-query-in-body/376707/6 "2025-04-14T08:29:58Z")
> [@Jannus\_C](#): > > epoch\_millis It seems like a time format mismatch, checking the error it seems it requires epoch millisecond or Unix milli time. We can modify your query as ```auto request.transforms: - set: target: body.query.range.@timestamp.gte value: '[[(now (parseDuration "-900h")).UnixMilli]]' ``` ---
**Author:** ![Jannus\_C](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jannus_c/32/139984_2.png) [@Jannus\_C](https://discuss.elastic.co/u/Jannus_C)\ **Post date:** [April 14, 2025, 12:24pm UTC](https://discuss.elastic.co/t/filebeat-httpjson-dynamic-query-in-body/376707/7 "2025-04-14T12:24:18Z")
Hello @exdghost Still get an error ```auto {"log.level":"error","@timestamp":"2025-04-14T14:11:38.395+0200","log.logger":"input.httpjson-stateless","log.origin":{"function":"github.com/elastic/beats/v7/x-pack/filebeat/input/httpjson.(*publisher).handleError","file.name":"httpjson/request.go","file.line":801},"message":"error processing response: server responded with status code 400: {\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"Unknown parameter [query] in request body or parameter is of the wrong type[START_OBJECT] \"}],\"type\":\"illegal_argument_exception\",\"reason\":\"Unknown parameter [query] in request body or parameter is of the wrong type[START_OBJECT] \"},\"status\":400}","service.name":"filebeat","id":"CEB4F18BBE575622","input_url":"https://192.168.3.10:9200/test-index/_search?scroll=5m","ecs.version":"1.6.0"} ``` ---
**Author:** ![exdghost](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exdghost/32/100274_2.png) [@exdghost](https://discuss.elastic.co/u/exdghost)\ **Post date:** [April 15, 2025, 5:09am UTC](https://discuss.elastic.co/t/filebeat-httpjson-dynamic-query-in-body/376707/8 "2025-04-15T05:09:33Z")
> [@Jannus\_C](#): > > :"all shards failed","phase":"query","grouped":true,"failed\_shards":[{"shard":0,"index":"nessus-saas-2025.02.20-000001","node":"SeT8Ru0RSveuDJNbEB-Ajw","reason":{"type":"parse\_exception","reason":"failed to parse date field [2025-03-07 09:29:32.853043701 +0000 UTC] with format [strict\_date\_optional\_time||epoch\_millis]: [failed to parse date field [2025-03-07 09:29:32.853043701 +0000 UTC] with format [strict\_date\_optional\_time||epoch\_millis]]\ Can you try the following once: ```auto request.transforms: - set: target: body.query.range.@timestamp.gte value: '[[formatDate (now (parseDuration "-900h")) "2006-01-02T15:04:05Z07:00"]]' ``` If this does not work, can you share your current complete httpjson config, so I can personally debug it a bit. ---
**Author:** ![Jannus\_C](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jannus_c/32/139984_2.png) [@Jannus\_C](https://discuss.elastic.co/u/Jannus_C)\ **Post date:** [April 15, 2025, 10:40pm UTC](https://discuss.elastic.co/t/filebeat-httpjson-dynamic-query-in-body/376707/9 "2025-04-15T22:40:29Z")
Hello @exdghost It only fetches the 10 first documents (out of 7300) and then I get this error ```auto {"log.level":"error","@timestamp":"2025-04-16T00:29:03.346+0200","log.logger":"input.httpjson-stateless","log.origin":{"function":"github.com/elastic/beats/v7/x-pack/filebeat/input/httpjson.(*publisher).handleError","file.name":"httpjson/request.go","file.line":801},"message":"error processing response: server responded with status code 400: {\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"Unknown parameter [query] in request body or parameter is of the wrong type[START_OBJECT] \"}],\"type\":\"illegal_argument_exception\",\"reason\":\"Unknown parameter [query] in request body or parameter is of the wrong type[START_OBJECT] \"},\"status\":400}","service.name":"filebeat","id":"351E05331C28D226","input_url":"https://192.168.3.10:9200/nessus-fpnsm/_search?scroll=5m","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2025-04-16T00:29:03.346+0200","log.logger":"input.httpjson-stateless","log.origin":{"function":"github.com/elastic/beats/v7/x-pack/filebeat/input/httpjson.(*requester).doRequest","file.name":"httpjson/request.go","file.line":207},"message":"request finished: 10 events published","service.name":"filebeat","id":"351E05331C28D226","input_url":"https://192.168.3.10:9200/test-index/_search?scroll=5m","ecs.version":"1.6.0"} ``` The full config is ```auto - type: httpjson config_version: 2 interval: 5m request.url: https://192.168.3.10:9200/test-index/_search?scroll=5m request.method: POST request.ssl.verification_mode: none auth.basic.user: "${ES_USR}" auth.basic.password: "${ES_PWD}" request.body: query: range: "@timestamp": gte: now response.split: target: body.hits.hits processors: - decode_json_fields: fields: ["message"] target: "test" - add_tags: tags: ["9nd","POST","scroll","query"] target: "stage2" index: httpjson-test pipeline: clean_httpjson request.transforms: - set: target: body.query.range.@timestamp.gte value: '[[formatDate (now (parseDuration "-900h")) "2006-01-02T15:04:05Z07:00"]]' response.pagination: - set: target: url.value value: https://192.168.3.10:9200/_search/scroll - set: target: url.params.scroll_id value: '[[.last_response.body._scroll_id]]' - set: target: body.scroll value: 5m ``` Thank you ---
**Author:** ![Jannus\_C](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jannus_c/32/139984_2.png) [@Jannus\_C](https://discuss.elastic.co/u/Jannus_C)\ **Post date:** [April 15, 2025, 10:49pm UTC](https://discuss.elastic.co/t/filebeat-httpjson-dynamic-query-in-body/376707/10 "2025-04-15T22:49:14Z")
Fixing the query like here ```auto request.body: query: range: "@timestamp": gte: "now-900h" ``` Returns the 7300 docs ---
**Author:** ![exdghost](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exdghost/32/100274_2.png) [@exdghost](https://discuss.elastic.co/u/exdghost)\ **Post date:** [April 24, 2025, 9:16am UTC](https://discuss.elastic.co/t/filebeat-httpjson-dynamic-query-in-body/376707/11 "2025-04-24T09:16:26Z")
Then it seems to be working as now-900h is a huge time range so something like 7.3k docs can be expected. ---
**Author:** ![Jannus\_C](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jannus_c/32/139984_2.png) [@Jannus\_C](https://discuss.elastic.co/u/Jannus_C)\ **Post date:** [April 30, 2025, 2:04pm UTC](https://discuss.elastic.co/t/filebeat-httpjson-dynamic-query-in-body/376707/13 "2025-04-30T14:04:53Z")
Hello @exdghost, I decided to move directy to the Custom API integration via agent in order to get my IDM audit events. The query (in the body) I need to execute in order to grab the events is the one below, that I've configured it in the **Request Body** section of the integration ```auto { "attributes": "userId" , "limit": 100, "orderByAttribute": { "ascending": true, "name": "eventTime" }, "searchByAttributes": { "name": "startTime", "operator": "GREATER_THAN_OR_EQUAL", "value": "2025-04-29T00:00:00" } } ``` I've also configured the **Response Split** and **Response Pagination** ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/2/e2a1cdb9bf1860fb8e01325b74e7b564a2ed9eb8.png) In the next execution I need to **modify** the startTime and I do not know exactly how to do it. Should I use Request Transforms or Response Transforms? Should I define a cursor that tracks the last event?Any help will be appreciated Thank you ---
**Author:** ![Jannus\_C](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jannus_c/32/139984_2.png) [@Jannus\_C](https://discuss.elastic.co/u/Jannus_C)\ **Post date:** [April 30, 2025, 4:39pm UTC](https://discuss.elastic.co/t/filebeat-httpjson-dynamic-query-in-body/376707/14 "2025-04-30T16:39:11Z")
Regarding to **Response Pagination** how can I add a condition to this`- set:` ` target: body.cursor` ` value: '[[.last_response.body.paging.nextCursor]]'` I only want to set the body.cursor if the nextCursor is not null, otherwise the cycle should finish till next execution I tried something like this but it fails ```auto - set: target: body.cursor value: '[[if (eq (.last_response.body.paging.nextCursor) nil)]][[.last_response.body.paging.nextCursor]][[end]]' ``` ---
**Author:** ![Jannus\_C](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jannus_c/32/139984_2.png) [@Jannus\_C](https://discuss.elastic.co/u/Jannus_C)\ **Post date:** [April 30, 2025, 5:27pm UTC](https://discuss.elastic.co/t/filebeat-httpjson-dynamic-query-in-body/376707/15 "2025-04-30T17:27:03Z")
I solved the pagination issue ```auto - set: target: body.cursor value: '[[if (ne .last_response.body.paging.nextCursor nil)]][[.last_response.body.paging.nextCursor]][[end]]' fail_on_template_error: true ``` ---
**Author:** ![Jannus\_C](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jannus_c/32/139984_2.png) [@Jannus\_C](https://discuss.elastic.co/u/Jannus_C)\ **Post date:** [May 2, 2025, 7:55am UTC](https://discuss.elastic.co/t/filebeat-httpjson-dynamic-query-in-body/376707/16 "2025-05-02T07:55:56Z")
I solved like this via Custom API integration ```auto **Request Transforms** - set: target: body.limit value: 100 - set: target: body.attributes value: "userId" - set: target: body.orderByAttribute.ascending value: "true" - set: target: body.orderByAttribute.name value: "eventTime" - set: target: body.searchByAttributes.name value: "startTime" - set: target: body.searchByAttributes.operator value: "GREATER_THAN_OR_EQUAL" - set: target: body.searchByAttributes.value value: "2025-04-30T16:00:00" **Response Split** target: body.results **Response Pagination** - set: target: body.cursor value: '[[if (ne .last_response.body.paging.nextCursor nil)]][[.last_response.body.paging.nextCursor]][[end]]' fail_on_template_error: true **Custom Request Cursor** last_requested_at: value: '[[.last_event.eventTime]]' ```