# Ingest pipeline is not working for given document
**URL:** https://discuss.elastic.co/t/ingest-pipeline-is-not-working-for-given-document/379099
**Category:** Elasticsearch
**Tags:** painless, ingest-pipeline
**Created:** [June 11, 2025, 2:40pm UTC](https://discuss.elastic.co/t/ingest-pipeline-is-not-working-for-given-document/379099 "2025-06-11T14:40:08Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@kuldeep\_gupta](https://discuss.elastic.co/u/kuldeep_gupta)
#### Post date: [June 11, 2025, 2:40pm UTC](https://discuss.elastic.co/t/ingest-pipeline-is-not-working-for-given-document/379099/1 "2025-06-11T14:40:08Z")
i have below record which i want to process.
```auto
[
{
"_id": "GFk-X5cBY6REVzo7i86y",
"_index": "processor_test",
"_source": {
"event.original":"172.16.102.98 - - [11/Jun/2025:19:05:43 +0530] \"POST /api/elasticsearchData HTTP/1.1\" 200 397 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36 Edg/137.0.0.0\" \"{\\x22query\\x22:\\x22/home/overview/trafficOverview.json\\x22,\\x22querySource\\x22:\\x22file\\x22,\\x22startDate\\x22:\\x222025-06-11T12:41:30.760Z\\x22,\\x22endDate\\x22:\\x222025-06-11T12:56:30.760Z\\x22,\\x22customUserFilter\\x22:false,\\x22tabify\\x22:false,\\x22debug\\x22:false}"
}
}
]
```
And below is my ingress pipeline
```auto
PUT _ingest/pipeline/logs-nginx.access@custom
{
"processors": [
{
"grok": {
"field": "event.original",
"tag": "grok_extract_raw_body",
"patterns": [
"(?:%{IPORHOST} %{HTTPDUSER} %{HTTPDUSER} \\[%{HTTPDATE}\\] \"(?:%{WORD} %{NOTSPACE}(?: HTTP/%{NUMBER})?|%{DATA})\" %{NUMBER} (?:%{NUMBER}|-) %{QS} %{QS}) %{GREEDYDATA:request_body_raw}"
],
"on_failure": [
{
"set": {
"field": "pipeline_failure.processor_type",
"value": "{{ _ingest.on_failure_processor_type }}"
}
},
{
"set": {
"field": "pipeline_failure.processor_tag",
"value": "{{ _ingest.on_failure_processor_tag }}"
}
},
{
"set": {
"field": "pipeline_failure.message",
"value": "Grok failed: {{ _ingest.on_failure_message }}"
}
}
]
}
},
{
"script": {
"lang": "painless",
"tag": "data_sanitization",
"source": """
if (ctx.containsKey('request_body_raw') && ctx.request_body_raw != null) {
String tempBody = ctx.request_body_raw;
// Remove outermost double quotes (if present) captured by Grok
if (tempBody.length() >= 2 && tempBody.startsWith('"') && tempBody.endsWith('"')) {
tempBody = tempBody.substring(1, tempBody.length() - 1);
}
// Correctly replace \\x22 with a single double quote "
ctx.request_body = tempBody.replace("\\\\x22", "\\\""); // Correct Painless escape for literal "
}
"""
}
},
{
"json": {
"field": "request_body",
"tag": "json_parsing",
"target_field": "parsed_request_body",
"if": "ctx.request_body != null && (ctx.request_body.startsWith('{') || ctx.request_body.startsWith('['))",
"ignore_failure": true,
"on_failure": [
{
"set": {
"field": "json_parsing_error",
"value": "JSON parsing failed for 'request_body': {{ _ingest.on_failure_message }}"
}
}
]
}
},
{
"rename": {
"tag": "renaming_parsed_request_body.query",
"field": "parsed_request_body.query",
"target_field": "request.query",
"ignore_failure": true
}
},
{
"rename": {
"tag": "renaming_parsed_request_body.startDate",
"field": "parsed_request_body.startDate",
"target_field": "request.startDate",
"ignore_failure": true
}
},
{
"rename": {
"tag": "renaming_parsed_request_body.endDate",
"field": "parsed_request_body.endDate",
"target_field": "request.endDate",
"ignore_failure": true
}
},
{
"remove": {
"field": "event.original",
"ignore_failure": true
}
},
{
"script": {
"tag": "timerange_calculation",
"lang": "painless",
"source": "\n if (ctx.containsKey('request') && ctx.request.startDate != null && ctx.request.endDate != null) {\n Instant start = Instant.parse(ctx.request.startDate);\n Instant end = Instant.parse(ctx.request.endDate);\n long diffMillis = Duration.between(start, end).toMillis();\n ctx.request.durationMinutes = diffMillis / 60000.0;\n }\n ",
"ignore_failure": true,
"on_failure": [
{
"set": {
"field": "timerange_calculation_error",
"value": "Timerange calculation failed"
}
}
]
}
},
{
"append": {
"field": "test",
"value": "kuldeep"
}
}
],
"on_failure": [
{
"set": {
"field": "pipeline_failure.processor_type",
"value": "{{ _ingest.on_failure_processor_type }}"
}
},
{
"set": {
"field": "pipeline_failure.processor_tag",
"value": "{{ _ingest.on_failure_processor_tag }}"
}
},
{
"set": {
"field": "pipeline_failure.message",
"value": "{{ _ingest.on_failure_message }}"
}
}
]
}
```
My goal is to parse time range of query in minutes. and in my nginx record(provided above) i have startTime and endTime field.
I tried testing for the given document and got this output:
```auto
{
"docs": [
{
"processor_results": [
{
"processor_type": "grok",
"status": "error",
"tag": "grok_extract_raw_body",
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "field [event] not present as part of path [event.original]"
}
],
"type": "illegal_argument_exception",
"reason": "field [event] not present as part of path [event.original]"
}
},
{
"processor_type": "set",
"status": "success",
"doc": {
"_index": "processor_test",
"_id": "HVlqX5cBY6REVzo7CdMj",
"_version": "-3",
"_source": {
"event.original": "172.16.102.98 - - [11/Jun/2025:19:05:43 +0530] \"POST /api/elasticsearchData HTTP/1.1\" 200 397 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36 Edg/137.0.0.0\" \"{\\x22query\\x22:\\x22/home/overview/trafficOverview.json\\x22,\\x22querySource\\x22:\\x22file\\x22,\\x22startDate\\x22:\\x222025-06-11T12:41:30.760Z\\x22,\\x22endDate\\x22:\\x222025-06-11T12:56:30.760Z\\x22,\\x22customUserFilter\\x22:false,\\x22tabify\\x22:false,\\x22debug\\x22:false}",
"pipeline_failure": {
"processor_type": "grok"
}
},
"_ingest": {
"pipeline": "_simulate_pipeline",
"on_failure_message": "field [event] not present as part of path [event.original]",
"on_failure_processor_tag": "grok_extract_raw_body",
"on_failure_pipeline": "_simulate_pipeline",
"timestamp": "2025-06-11T14:39:41.801Z",
"on_failure_processor_type": "grok"
}
}
},
{
"processor_type": "set",
"status": "success",
"doc": {
"_index": "processor_test",
"_id": "HVlqX5cBY6REVzo7CdMj",
"_version": "-3",
"_source": {
"event.original": "172.16.102.98 - - [11/Jun/2025:19:05:43 +0530] \"POST /api/elasticsearchData HTTP/1.1\" 200 397 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36 Edg/137.0.0.0\" \"{\\x22query\\x22:\\x22/home/overview/trafficOverview.json\\x22,\\x22querySource\\x22:\\x22file\\x22,\\x22startDate\\x22:\\x222025-06-11T12:41:30.760Z\\x22,\\x22endDate\\x22:\\x222025-06-11T12:56:30.760Z\\x22,\\x22customUserFilter\\x22:false,\\x22tabify\\x22:false,\\x22debug\\x22:false}",
"pipeline_failure": {
"processor_type": "grok",
"processor_tag": "grok_extract_raw_body"
}
},
"_ingest": {
"pipeline": "_simulate_pipeline",
"on_failure_message": "field [event] not present as part of path [event.original]",
"on_failure_processor_tag": "grok_extract_raw_body",
"on_failure_pipeline": "_simulate_pipeline",
"timestamp": "2025-06-11T14:39:41.801Z",
"on_failure_processor_type": "grok"
}
}
},
{
"processor_type": "set",
"status": "success",
"doc": {
"_index": "processor_test",
"_id": "HVlqX5cBY6REVzo7CdMj",
"_version": "-3",
"_source": {
"event.original": "172.16.102.98 - - [11/Jun/2025:19:05:43 +0530] \"POST /api/elasticsearchData HTTP/1.1\" 200 397 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36 Edg/137.0.0.0\" \"{\\x22query\\x22:\\x22/home/overview/trafficOverview.json\\x22,\\x22querySource\\x22:\\x22file\\x22,\\x22startDate\\x22:\\x222025-06-11T12:41:30.760Z\\x22,\\x22endDate\\x22:\\x222025-06-11T12:56:30.760Z\\x22,\\x22customUserFilter\\x22:false,\\x22tabify\\x22:false,\\x22debug\\x22:false}",
"pipeline_failure": {
"processor_type": "grok",
"processor_tag": "grok_extract_raw_body",
"message": "Grok failed: field [event] not present as part of path [event.original]"
}
},
"_ingest": {
"pipeline": "_simulate_pipeline",
"on_failure_message": "field [event] not present as part of path [event.original]",
"on_failure_processor_tag": "grok_extract_raw_body",
"on_failure_pipeline": "_simulate_pipeline",
"timestamp": "2025-06-11T14:39:41.801Z",
"on_failure_processor_type": "grok"
}
}
},
{
"processor_type": "script",
"status": "success",
"tag": "data_sanitization",
"doc": {
"_index": "processor_test",
"_id": "HVlqX5cBY6REVzo7CdMj",
"_version": "-3",
"_source": {
"event.original": "172.16.102.98 - - [11/Jun/2025:19:05:43 +0530] \"POST /api/elasticsearchData HTTP/1.1\" 200 397 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36 Edg/137.0.0.0\" \"{\\x22query\\x22:\\x22/home/overview/trafficOverview.json\\x22,\\x22querySource\\x22:\\x22file\\x22,\\x22startDate\\x22:\\x222025-06-11T12:41:30.760Z\\x22,\\x22endDate\\x22:\\x222025-06-11T12:56:30.760Z\\x22,\\x22customUserFilter\\x22:false,\\x22tabify\\x22:false,\\x22debug\\x22:false}",
"pipeline_failure": {
"processor_type": "grok",
"processor_tag": "grok_extract_raw_body",
"message": "Grok failed: field [event] not present as part of path [event.original]"
}
},
"_ingest": {
"pipeline": "_simulate_pipeline",
"timestamp": "2025-06-11T14:39:41.801Z"
}
}
},
{
"processor_type": "json",
"status": "skipped",
"tag": "json_parsing",
"if": {
"condition": "ctx.request_body != null && (ctx.request_body.startsWith('{') || ctx.request_body.startsWith('['))",
"result": false
}
},
{
"processor_type": "rename",
"status": "error_ignored",
"tag": "renaming_parsed_request_body.query",
"ignored_error": {
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "field [parsed_request_body.query] doesn't exist"
}
],
"type": "illegal_argument_exception",
"reason": "field [parsed_request_body.query] doesn't exist"
}
},
"doc": {
"_index": "processor_test",
"_id": "HVlqX5cBY6REVzo7CdMj",
"_version": "-3",
"_source": {
"event.original": "172.16.102.98 - - [11/Jun/2025:19:05:43 +0530] \"POST /api/elasticsearchData HTTP/1.1\" 200 397 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36 Edg/137.0.0.0\" \"{\\x22query\\x22:\\x22/home/overview/trafficOverview.json\\x22,\\x22querySource\\x22:\\x22file\\x22,\\x22startDate\\x22:\\x222025-06-11T12:41:30.760Z\\x22,\\x22endDate\\x22:\\x222025-06-11T12:56:30.760Z\\x22,\\x22customUserFilter\\x22:false,\\x22tabify\\x22:false,\\x22debug\\x22:false}",
"pipeline_failure": {
"processor_type": "grok",
"processor_tag": "grok_extract_raw_body",
"message": "Grok failed: field [event] not present as part of path [event.original]"
}
},
"_ingest": {
"pipeline": "_simulate_pipeline",
"timestamp": "2025-06-11T14:39:41.801Z"
}
}
},
{
"processor_type": "rename",
"status": "error_ignored",
"tag": "renaming_parsed_request_body.startDate",
"ignored_error": {
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "field [parsed_request_body.startDate] doesn't exist"
}
],
"type": "illegal_argument_exception",
"reason": "field [parsed_request_body.startDate] doesn't exist"
}
},
"doc": {
"_index": "processor_test",
"_id": "HVlqX5cBY6REVzo7CdMj",
"_version": "-3",
"_source": {
"event.original": "172.16.102.98 - - [11/Jun/2025:19:05:43 +0530] \"POST /api/elasticsearchData HTTP/1.1\" 200 397 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36 Edg/137.0.0.0\" \"{\\x22query\\x22:\\x22/home/overview/trafficOverview.json\\x22,\\x22querySource\\x22:\\x22file\\x22,\\x22startDate\\x22:\\x222025-06-11T12:41:30.760Z\\x22,\\x22endDate\\x22:\\x222025-06-11T12:56:30.760Z\\x22,\\x22customUserFilter\\x22:false,\\x22tabify\\x22:false,\\x22debug\\x22:false}",
"pipeline_failure": {
"processor_type": "grok",
"processor_tag": "grok_extract_raw_body",
"message": "Grok failed: field [event] not present as part of path [event.original]"
}
},
"_ingest": {
"pipeline": "_simulate_pipeline",
"timestamp": "2025-06-11T14:39:41.801Z"
}
}
},
{
"processor_type": "rename",
"status": "error_ignored",
"tag": "renaming_parsed_request_body.endDate",
"ignored_error": {
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "field [parsed_request_body.endDate] doesn't exist"
}
],
"type": "illegal_argument_exception",
"reason": "field [parsed_request_body.endDate] doesn't exist"
}
},
"doc": {
"_index": "processor_test",
"_id": "HVlqX5cBY6REVzo7CdMj",
"_version": "-3",
"_source": {
"event.original": "172.16.102.98 - - [11/Jun/2025:19:05:43 +0530] \"POST /api/elasticsearchData HTTP/1.1\" 200 397 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36 Edg/137.0.0.0\" \"{\\x22query\\x22:\\x22/home/overview/trafficOverview.json\\x22,\\x22querySource\\x22:\\x22file\\x22,\\x22startDate\\x22:\\x222025-06-11T12:41:30.760Z\\x22,\\x22endDate\\x22:\\x222025-06-11T12:56:30.760Z\\x22,\\x22customUserFilter\\x22:false,\\x22tabify\\x22:false,\\x22debug\\x22:false}",
"pipeline_failure": {
"processor_type": "grok",
"processor_tag": "grok_extract_raw_body",
"message": "Grok failed: field [event] not present as part of path [event.original]"
}
},
"_ingest": {
"pipeline": "_simulate_pipeline",
"timestamp": "2025-06-11T14:39:41.801Z"
}
}
},
{
"processor_type": "remove",
"status": "error_ignored",
"ignored_error": {
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "field [event] not present as part of path [event.original]"
}
],
"type": "illegal_argument_exception",
"reason": "field [event] not present as part of path [event.original]"
}
},
"doc": {
"_index": "processor_test",
"_id": "HVlqX5cBY6REVzo7CdMj",
"_version": "-3",
"_source": {
"event.original": "172.16.102.98 - - [11/Jun/2025:19:05:43 +0530] \"POST /api/elasticsearchData HTTP/1.1\" 200 397 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36 Edg/137.0.0.0\" \"{\\x22query\\x22:\\x22/home/overview/trafficOverview.json\\x22,\\x22querySource\\x22:\\x22file\\x22,\\x22startDate\\x22:\\x222025-06-11T12:41:30.760Z\\x22,\\x22endDate\\x22:\\x222025-06-11T12:56:30.760Z\\x22,\\x22customUserFilter\\x22:false,\\x22tabify\\x22:false,\\x22debug\\x22:false}",
"pipeline_failure": {
"processor_type": "grok",
"processor_tag": "grok_extract_raw_body",
"message": "Grok failed: field [event] not present as part of path [event.original]"
}
},
"_ingest": {
"pipeline": "_simulate_pipeline",
"timestamp": "2025-06-11T14:39:41.801Z"
}
}
},
{
"processor_type": "script",
"status": "success",
"tag": "timerange_calculation",
"doc": {
"_index": "processor_test",
"_id": "HVlqX5cBY6REVzo7CdMj",
"_version": "-3",
"_source": {
"event.original": "172.16.102.98 - - [11/Jun/2025:19:05:43 +0530] \"POST /api/elasticsearchData HTTP/1.1\" 200 397 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36 Edg/137.0.0.0\" \"{\\x22query\\x22:\\x22/home/overview/trafficOverview.json\\x22,\\x22querySource\\x22:\\x22file\\x22,\\x22startDate\\x22:\\x222025-06-11T12:41:30.760Z\\x22,\\x22endDate\\x22:\\x222025-06-11T12:56:30.760Z\\x22,\\x22customUserFilter\\x22:false,\\x22tabify\\x22:false,\\x22debug\\x22:false}",
"pipeline_failure": {
"processor_type": "grok",
"processor_tag": "grok_extract_raw_body",
"message": "Grok failed: field [event] not present as part of path [event.original]"
}
},
"_ingest": {
"pipeline": "_simulate_pipeline",
"timestamp": "2025-06-11T14:39:41.801Z"
}
}
},
{
"processor_type": "append",
"status": "success",
"doc": {
"_index": "processor_test",
"_id": "HVlqX5cBY6REVzo7CdMj",
"_version": "-3",
"_source": {
"event.original": "172.16.102.98 - - [11/Jun/2025:19:05:43 +0530] \"POST /api/elasticsearchData HTTP/1.1\" 200 397 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36 Edg/137.0.0.0\" \"{\\x22query\\x22:\\x22/home/overview/trafficOverview.json\\x22,\\x22querySource\\x22:\\x22file\\x22,\\x22startDate\\x22:\\x222025-06-11T12:41:30.760Z\\x22,\\x22endDate\\x22:\\x222025-06-11T12:56:30.760Z\\x22,\\x22customUserFilter\\x22:false,\\x22tabify\\x22:false,\\x22debug\\x22:false}",
"test": [
"kuldeep"
],
"pipeline_failure": {
"processor_type": "grok",
"processor_tag": "grok_extract_raw_body",
"message": "Grok failed: field [event] not present as part of path [event.original]"
}
},
"_ingest": {
"pipeline": "_simulate_pipeline",
"timestamp": "2025-06-11T14:39:41.801Z"
}
}
}
]
}
]
}
```
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [June 11, 2025, 3:28pm UTC](https://discuss.elastic.co/t/ingest-pipeline-is-not-working-for-given-document/379099/2 "2025-06-11T15:28:54Z")
Hi @kuldeep_gupta
> [@kuldeep\_gupta](#):
>
> `"event.original"`
That is a dotted field not supported in ingest pipeline that should look like an object in the `_source` document
`"event" : {"original" : ....}}`
Assuming that you're getting that event that original from an integration....
If you really getting dot Notation you need to expand it
You can about
> If your document contains flattened objects, use the dot\_expander processor to expand them first. Other ingest processors cannot access flattened objects.
> **[Elasticsearch ingest pipelines | Elastic Docs](https://www.elastic.co/docs/manage-data/ingest/transform-enrich/ingest-pipelines)**
>
> Elasticsearch ingest pipelines let you perform common transformations on your data before indexing. For example, you can use pipelines to remove fields,...
So get that figured out and try again
Are you using the `_simulate` API To test it was not clear. That's what I would use.
> **[Simulate a pipeline | Elasticsearch API documentation](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-simulate)**
>
> Elasticsearch provides REST APIs that are used by the UI components and can be called directly to configure and access Elasticsearch features.
> Documentation ...
---
### Author:  [@tazieviikz](https://discuss.elastic.co/u/tazieviikz)
#### Post date: [June 11, 2025, 3:28pm UTC](https://discuss.elastic.co/t/ingest-pipeline-is-not-working-for-given-document/379099/3 "2025-06-11T15:28:58Z")
You need to set event.original before like
{{\_source.message}}"
---
### Author:  [@kuldeep\_gupta](https://discuss.elastic.co/u/kuldeep_gupta)
#### Post date: [June 11, 2025, 3:43pm UTC](https://discuss.elastic.co/t/ingest-pipeline-is-not-working-for-given-document/379099/4 "2025-06-11T15:43:15Z")
```auto
PUT _ingest/pipeline/logs-nginx.access@custom
{
"processors": [
{
"dot_expander": {
"description": "Expand 'my-object-field.my-property'",
"field": "event.original"
}
},
{
"grok": {
"field": "event.original",
"tag": "grok_extract_raw_body",
"patterns": [
"(?:%{IPORHOST} %{HTTPDUSER} %{HTTPDUSER} \\[%{HTTPDATE}\\] \"(?:%{WORD} %{NOTSPACE}(?: HTTP/%{NUMBER})?|%{DATA})\" %{NUMBER} (?:%{NUMBER}|-) %{QS} %{QS}) %{GREEDYDATA:request_body_raw}"
],
"on_failure": [
{
"set": {
"field": "pipeline_failure.processor_type",
"value": "{{ _ingest.on_failure_processor_type }}"
}
},
{
"set": {
"field": "pipeline_failure.processor_tag",
"value": "{{ _ingest.on_failure_processor_tag }}"
}
},
{
"set": {
"field": "pipeline_failure.message",
"value": "Grok failed: {{ _ingest.on_failure_message }}"
}
}
]
}
},
{
"script": {
"lang": "painless",
"tag": "data_sanitization",
"source": """
if (ctx.containsKey('request_body_raw') && ctx.request_body_raw != null) {
String tempBody = ctx.request_body_raw;
// Remove outermost double quotes (if present) captured by Grok
if (tempBody.length() >= 2 && tempBody.startsWith('"') && tempBody.endsWith('"')) {
tempBody = tempBody.substring(1, tempBody.length() - 1);
}
// Correctly replace \\x22 with a single double quote "
ctx.request_body = tempBody.replace("\\\\x22", "\\\""); // Correct Painless escape for literal "
}
"""
}
},
{
"json": {
"field": "request_body",
"tag": "json_parsing",
"target_field": "parsed_request_body",
"if": "ctx.request_body != null && (ctx.request_body.startsWith('{') || ctx.request_body.startsWith('['))",
"ignore_failure": true,
"on_failure": [
{
"set": {
"field": "json_parsing_error",
"value": "JSON parsing failed for 'request_body': {{ _ingest.on_failure_message }}"
}
}
]
}
},
{
"rename": {
"tag": "renaming_parsed_request_body.query",
"field": "parsed_request_body.query",
"target_field": "request.query",
"ignore_failure": true
}
},
{
"rename": {
"tag": "renaming_parsed_request_body.startDate",
"field": "parsed_request_body.startDate",
"target_field": "request.startDate",
"ignore_failure": true
}
},
{
"rename": {
"tag": "renaming_parsed_request_body.endDate",
"field": "parsed_request_body.endDate",
"target_field": "request.endDate",
"ignore_failure": true
}
},
{
"remove": {
"field": "event.original",
"ignore_failure": true
}
},
{
"script": {
"tag": "timerange_calculation",
"lang": "painless",
"source": "\n if (ctx.containsKey('request') && ctx.request.startDate != null && ctx.request.endDate != null) {\n Instant start = Instant.parse(ctx.request.startDate);\n Instant end = Instant.parse(ctx.request.endDate);\n long diffMillis = Duration.between(start, end).toMillis();\n ctx.request.durationMinutes = diffMillis / 60000.0;\n }\n ",
"ignore_failure": true,
"on_failure": [
{
"set": {
"field": "timerange_calculation_error",
"value": "Timerange calculation failed"
}
}
]
}
},
{
"append": {
"field": "test",
"value": "kuldeep"
}
}
],
"on_failure": [
{
"set": {
"field": "pipeline_failure.processor_type",
"value": "{{ _ingest.on_failure_processor_type }}"
}
},
{
"set": {
"field": "pipeline_failure.processor_tag",
"value": "{{ _ingest.on_failure_processor_tag }}"
}
},
{
"set": {
"field": "pipeline_failure.message",
"value": "{{ _ingest.on_failure_message }}"
}
}
]
}
```
After you suggestion my pipeline looks like this.
grok is working now but not still not able to achieve my goal.
while parsing field **request\_body** in json if condition is giving false.
my ultimate goal is to parse request\_body into json .
Please help.
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [June 11, 2025, 3:50pm UTC](https://discuss.elastic.co/t/ingest-pipeline-is-not-working-for-given-document/379099/5 "2025-06-11T15:50:08Z")
It's just debugging now.....
I would incrementally add the processors
I would take out your script processor first....
See if the json works
Use the `_simulate` API with `verbose=true`
Etc
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [June 11, 2025, 4:19pm UTC](https://discuss.elastic.co/t/ingest-pipeline-is-not-working-for-given-document/379099/6 "2025-06-11T16:19:01Z")
Also I would read the docs on conditions and `null` safety
> **[Elasticsearch ingest pipelines | Elastic Docs](https://www.elastic.co/docs/manage-data/ingest/transform-enrich/ingest-pipelines#conditionally-run-processor)**
>
> Elasticsearch ingest pipelines let you perform common transformations on your data before indexing. For example, you can use pipelines to remove fields,...
---
### Author:  [@kuldeep\_gupta](https://discuss.elastic.co/u/kuldeep_gupta)
#### Post date: [June 11, 2025, 5:02pm UTC](https://discuss.elastic.co/t/ingest-pipeline-is-not-working-for-given-document/379099/7 "2025-06-11T17:02:45Z")
@stephenb Thank you.Now it is working as per requirement.
i had no idea about simulate api. Thanks again. 👍