# Problem elastic "document\_parsing\_exception" type field

**URL:** <https://discuss.elastic.co/t/problem-elastic-document-parsing-exception-type-field/356690>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-alerting\
**Created:** [April 3, 2024, 1:43pm UTC](https://discuss.elastic.co/t/problem-elastic-document-parsing-exception-type-field/356690 "2024-04-03T13:43:55Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![San9](https://avatars.discourse-cdn.com/v4/letter/s/a9adbd/32.png) [@San9](https://discuss.elastic.co/u/San9)\
**Post date:** [April 3, 2024, 1:43pm UTC](https://discuss.elastic.co/t/problem-elastic-document-parsing-exception-type-field/356690/1 "2024-04-03T13:43:55Z")

</div>

Hello, I have encountered a common problem that many Elastic users face, but I have not yet found an answer to it and I do not understand how to solve it so that I do not lose messages that are sent by Elastic.  
I use filebуat to connect to aws cloudtrail and receive logs.

And I replaced that Elasticsearch began to receive not all messages with logs; they were received selectively.  
I started analyzing logstash logs and saw the following error messages:

```auto
"status"=>400, "error"=>{"type"=>"document_parsing_exception", "reason"=>"[1:3774] failed to parse field [requestParameters.DescribeVpcEndpointsRequest] of type [text] in document with id '0Qe4oo4BOD3coVYthBSs'. Preview of field's value: '{MaxResults=1000}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:3756"}

```

or

```auto
"status"=>400, "error"=>{"type"=>"document_parsing_exception", "reason"=>"[1:3057] object mapping for [requestParameters.filter] tried to parse field [null] as object, but found a concrete value"}

```

As far as I understand, certain fields can be either in the form of text in one case, and in another case they can be an object.  
How can this problem be solved? Is it possible to specify a universal field type or not try to determine the field type for requestParameters.filter in Jason Data at all. and leave everything that it contains in its original form, if you don’t know how to dynamically determine the field type.

I will be glad for any help

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [April 3, 2024, 2:26pm UTC](https://discuss.elastic.co/t/problem-elastic-document-parsing-exception-type-field/356690/2 "2024-04-03T14:26:30Z")

</div>

You need to map the top-level field `requestParameters` as a `flattened` field, this way the entire json object of this field will be stored.

This is how elastic do that with the Cloudtrail integration.

They also copy the object field to another field to also keep the field as a string.

You can check how the ingest pipeline process the logs [here](https://github.com/elastic/integrations/blob/main/packages/aws/data_stream/cloudtrail/elasticsearch/ingest_pipeline/default.yml).

Are you using logstash? I have a filter for the 4 fields in cloudtrail logs that are dynamic, maybe you can adapt it for your use case.

---

<div class="post-metadata">

**Author:** ![San9](https://avatars.discourse-cdn.com/v4/letter/s/a9adbd/32.png) [@San9](https://discuss.elastic.co/u/San9)\
**Post date:** [April 3, 2024, 2:54pm UTC](https://discuss.elastic.co/t/problem-elastic-document-parsing-exception-type-field/356690/3 "2024-04-03T14:54:52Z")

</div>

Hi [leandrojmp](https://discuss.elastic.co/t/problem-elastic-document-parsing-exception-type-field/356690/2),

thanks for your reply.  
To connect to AWS, I use filebeat, it connects using the cloudtrail module and takes the logs I need, then I transfer the data to logstash for parsing json messages and then transfer it to elastic.

I will be glad if you share your ideas.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [April 3, 2024, 3:36pm UTC](https://discuss.elastic.co/t/problem-elastic-document-parsing-exception-type-field/356690/4 "2024-04-03T15:36:16Z")

</div>

> [@San9](#):
>
> To connect to AWS, I use filebeat, it connects using the cloudtrail module and takes the logs I need

Can you send it directly to Elasticsearch? If you do you will not need to worry with the parsing and would not have this issue.

In logstash I use the following filters to deal with the dynamic fields from cloudtrail, you may adapt it to your pipeline, my cloudtrail json has its fields nested under a field named `json`.

```auto
# requestParameters, responseElements, additionalEventData and serviceEventDetails can be dynamic and lead to mapping conflicts
#
# the following filter perform these stepes:
# - remove the field if it is empty
# - double check if the field exists and it is not empty
# - if true, mutate/add_field will create a new field with the json object as a string
# - if true, mutate/rename will rename the field as a nested field inside aws.cloudtrail.flattened, which needs to be mapped as flattened
#
filter {
    ruby {
        code => '
            event.remove("[json][requestParameters]") if event.get("[json][requestParameters]").nil?
            event.remove("[json][responseElements]") if event.get("[json][responseElements]").nil?
            event.remove("[json][additionalEventData]") if event.get("[json][additionalEventData]").nil?
            event.remove("[json][serviceEventDetails]") if event.get("[json][serviceEventDetails]").nil?
        '
    }
    if [json][requestParameters] and [json][requestParameters] != "" {
        mutate {
            add_field => {
                "[aws][cloudtrail][request_parameters]" => "%{[json][requestParameters]}"
            }
        }
        mutate {
            rename => {
                "[json][requestParameters]" => "[aws][cloudtrail][flattened][request_parameters]"
            }
        }

    }
    if [json][responseElements] and [json][responseElements] != "" {
        mutate {
            add_field => {
                "[aws][cloudtrail][response_elements]" => "%{[json][responseElements]}"
            }
        }
        mutate {
            rename => {
                "[json][responseElements]" => "[aws][cloudtrail][flattened][response_elements]"
            }
        }
    }
    if [json][additionalEventData] and [json][additionalEventData] != "" {
        mutate {
            add_field => {
                "[aws][cloudtrail][additional_eventdata]" => "%{[json][additionalEventData]}"
            }
        }
        mutate {
            rename => {
                "[json][additionalEventData]" => "[aws][cloudtrail][flattened][additional_eventdata]"
            }
        }
    }
    if [json][serviceEventDetails] and [json][serviceEventDetails] != "" {
        mutate {
            add_field => {
                "[aws][cloudtrail][service_event_details]" => "%{[json][serviceEventDetails]}"
            }
        }
        mutate {
            rename => {
                "[json][serviceEventDetails]" => "[aws][cloudtrail][flattened][service_event_details]"
            }
        }
    }
}
#

```

This basically replicate this [logic](https://github.com/elastic/integrations/blob/618de99e6a85b44be47bf47e1878bd3b9fa09c0e/packages/aws/data_stream/cloudtrail/elasticsearch/ingest_pipeline/default.yml#L181-L211) from the elasticsearch ingest pipeline.

---

<div class="post-metadata">

**Author:** ![San9](https://avatars.discourse-cdn.com/v4/letter/s/a9adbd/32.png) [@San9](https://discuss.elastic.co/u/San9)\
**Post date:** [April 4, 2024, 11:38am UTC](https://discuss.elastic.co/t/problem-elastic-document-parsing-exception-type-field/356690/5 "2024-04-04T11:38:12Z")

</div>

[Leandro Pereira](https://discuss.elastic.co/u/leandrojmp), thanks for the code!

I also thought about the possibility of sending data directly, but I’m not sure that in this case the message itself from the cloud trail will be parsed, since in the original it is stored there in the json format. one event contains a lot of information, and when I didn’t parse it, I just sent it through Logtash, then in this case the message had a pure format in JSON since it is stored on S3.  
Are you saying that filebeat itself parses the json message using the required fields? Or do you still need to use a separate module in filebeat?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [April 4, 2024, 11:48am UTC](https://discuss.elastic.co/t/problem-elastic-document-parsing-exception-type-field/356690/6 "2024-04-04T11:48:06Z")

</div>

> [@San9](#):
>
> Are you saying that filebeat itself parses the json message using the required fields? Or do you still need to use a separate module in filebeat?

Modules in Filebeat uses ingest pipelines in Elasticsearch to parse the data, so if you are using a module it is expected that the data will be parsed, but this will be done in Elasticsearch, not in Filebeat.

I do not use Filebeat anymore as I'm using the Elastic Agent and the Elastic Agent integrations will also use ingest pipelines in elasticsearch to parse your data.

You can check the integrations available [here](https://docs.elastic.co/integrations), there is one for AWS Cloudtrail and the instructions to configure.

If you want to check the ingest pipelines used you can check it [here](https://github.com/elastic/integrations/tree/main/packages).

---

<div class="post-metadata">

**Author:** ![San9](https://avatars.discourse-cdn.com/v4/letter/s/a9adbd/32.png) [@San9](https://discuss.elastic.co/u/San9)\
**Post date:** [April 4, 2024, 1:09pm UTC](https://discuss.elastic.co/t/problem-elastic-document-parsing-exception-type-field/356690/7 "2024-04-04T13:09:54Z")

</div>

[  
Leandro Pereira](https://discuss.elastic.co/u/leandrojmp), thanks

I checked, sent the logs directly to elastic, it works.  
And indeed aws.cloudtrail.request\_parameters is saved as an object. I remembered why I also redirected this data through logstash, using logstash I discarded messages containing /CloudTrail-Digest/ and I added information from the dictionary about the recipientAccountId. Now I don’t know how best to do this?

---

<div class="post-metadata">

**Author:** ![San9](https://avatars.discourse-cdn.com/v4/letter/s/a9adbd/32.png) [@San9](https://discuss.elastic.co/u/San9)\
**Post date:** [April 5, 2024, 3:27pm UTC](https://discuss.elastic.co/t/problem-elastic-document-parsing-exception-type-field/356690/8 "2024-04-05T15:27:21Z")

</div>

I tried to use your scheme, but I still couldn’t parse the fields dynamically ☹ or perhaps this scheme only works for data\_stream and not for regular index creation.

```auto
"status"=>400, "error"=>{"type"=>"document_parsing_exception", "reason"=>"[1:2091] object mapping for [aws.cloudtrail.flattened.request_parameters.DescribeVpcEndpointsRequest] tried to parse field [DescribeVpcEndpointsRequest] as object, but found a concrete value"}

```

or

```auto
 "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [aws.cloudtrail.flattened.request_parameters.DescribeTransitGatewaysRequest.Filter.Value.content] cannot be changed from type [long] to [text]"}

```

Apparently I'm still doing something wrong or haven't foreseen something...

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [April 5, 2024, 3:39pm UTC](https://discuss.elastic.co/t/problem-elastic-document-parsing-exception-type-field/356690/9 "2024-04-05T15:39:54Z")

</div>

Did you create the mapping in your template before sending the data?

What does your template looks like?

---

<div class="post-metadata">

**Author:** ![San9](https://avatars.discourse-cdn.com/v4/letter/s/a9adbd/32.png) [@San9](https://discuss.elastic.co/u/San9)\
**Post date:** [April 8, 2024, 7:37am UTC](https://discuss.elastic.co/t/problem-elastic-document-parsing-exception-type-field/356690/10 "2024-04-08T07:37:40Z")

</div>

/usr/share/filebeat/module/aws/cloudtrail/ingest/pipeline.yml

```auto
  - rename:
      field: "json.errorMessage"
      target_field: "aws.cloudtrail.error_message"
      ignore_failure: true
  - script:
      lang: painless
      source: |
        if (ctx.aws.cloudtrail?.flattened == null) {
            Map map = new HashMap();
            ctx.aws.cloudtrail.put("flattened", map);
          }
        if (ctx.json?.requestParameters != null) {
          ctx.aws.cloudtrail.request_parameters = ctx.json.requestParameters.toString();
          if (ctx.aws.cloudtrail.request_parameters.length() < 32766) {
            ctx.aws.cloudtrail.flattened.put("request_parameters", ctx.json.requestParameters);
          }
        }
        if (ctx.json?.responseElements != null) {
          ctx.aws.cloudtrail.response_elements = ctx.json.responseElements.toString();
          if (ctx.aws.cloudtrail.response_elements.length() < 32766) {
            ctx.aws.cloudtrail.flattened.put("response_elements", ctx.json.responseElements);
          }
        }
        if (ctx.json?.additionalEventData != null) {
          ctx.aws.cloudtrail.additional_eventdata = ctx.json.additionalEventData.toString();
          if (ctx.aws.cloudtrail.additional_eventdata.length() < 32766) {
            ctx.aws.cloudtrail.flattened.put("additional_eventdata", ctx.json.additionalEventData);
          }
        }
        if (ctx.json?.serviceEventDetails != null) {
          ctx.aws.cloudtrail.service_event_details = ctx.json.serviceEventDetails.toString();
          if (ctx.aws.cloudtrail.service_event_details.length() < 32766) {
            ctx.aws.cloudtrail.flattened.put("service_event_details", ctx.json.serviceEventDetails);
          }
        }
      ignore_failure: true
  - rename:
      field: "json.requestID"
      target_field: "aws.cloudtrail.request_id"
      ignore_failure: true

```

config logstash

```auto
json {
    source => "message"
        }
ruby {
        code => '
            event.remove("[requestParameters]") if event.get("[requestParameters]").nil?
            event.remove("[responseElements]") if event.get("[responseElements]").nil?
            event.remove("[additionalEventData]") if event.get("[additionalEventData]").nil?
            event.remove("[serviceEventDetails]") if event.get("[serviceEventDetails]").nil?
        '
    }
    if [requestParameters] and [requestParameters] != "" {
        mutate {
            add_field => {
                "[aws][cloudtrail][request_parameters]" => "%{[requestParameters]}"
            }
        }
        mutate {
            rename => {
                "[requestParameters]" => "[aws][cloudtrail][flattened][request_parameters]"
            }
        }

    }
    if [responseElements] and [responseElements] != "" {
        mutate {
            add_field => {
                "[aws][cloudtrail][response_elements]" => "%{[responseElements]}"
            }
        }
        mutate {
            rename => {
                "[responseElements]" => "[aws][cloudtrail][flattened][response_elements]"
            }
        }
    }
    if [additionalEventData] and [additionalEventData] != "" {
        mutate {
            add_field => {
                "[aws][cloudtrail][additional_eventdata]" => "%{[additionalEventData]}"
            }
        }
        mutate {
            rename => {
                "[additionalEventData]" => "[aws][cloudtrail][flattened][additional_eventdata]"
            }
        }
    }
    if [serviceEventDetails] and [serviceEventDetails] != "" {
        mutate {
            add_field => {
                "[aws][cloudtrail][service_event_details]" => "%{[serviceEventDetails]}"
            }
        }
        mutate {
            rename => {
                "[serviceEventDetails]" => "[aws][cloudtrail][flattened][service_event_details]"
            }
        }
    }

```

My log without embedded json.  
I'm wondering if I can write a config in Logstash that will not parse fields containing requestParameters.xxx responseElements.xxx into json { source =\> "message" }  
requestParameters.xxx  
responseElements.xxx  
additionalEventData.xxx  
serviceEventDetails.xxx

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [April 8, 2024, 11:50am UTC](https://discuss.elastic.co/t/problem-elastic-document-parsing-exception-type-field/356690/11 "2024-04-08T11:50:34Z")

</div>

Hello,

This does not answer the question, did you create the mapping or a template to apply your mapping?

The mapping is done in Elasticsearch, you didn´t share anything related to it, it is **required** to create the correct mapping for the flattened fields **before** infdexing any data.

---

<div class="post-metadata">

**Author:** ![San9](https://avatars.discourse-cdn.com/v4/letter/s/a9adbd/32.png) [@San9](https://discuss.elastic.co/u/San9)\
**Post date:** [April 8, 2024, 12:40pm UTC](https://discuss.elastic.co/t/problem-elastic-document-parsing-exception-type-field/356690/12 "2024-04-08T12:40:29Z")

</div>

[  
Leandro Pereira](https://discuss.elastic.co/u/leandrojmp), sorry  
At first I didn’t quite understand what you were talking about.  
I'm using this template, but it's an old version.

```auto
{
  "_routing": {
    "required": false
  },
  "numeric_detection": false,
  "dynamic_date_formats": [
    "strict_date_optional_time",
    "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
  ],
  "_source": {
    "excludes": [],
    "includes": [],
    "enabled": true
  },
  "dynamic": true,
  "dynamic_templates": [],
  "date_detection": true,
  "properties": {
    "requestParameters.filter": {
      "type": "text"
    },
    "aws.cloudtrail.flattened.request_parameters.DescribeTransitGatewaysRequest.Filter.Value.content": {
      "type": "text"
    },
    "apiVersion": {
      "eager_global_ordinals": false,
      "index_phrases": false,
      "fielddata": false,
      "norms": true,
      "index": true,
      "store": false,
      "type": "text",
      "index_options": "positions"
    },
    "requestParameters.maxResults": {
      "coerce": true,
      "index": true,
      "ignore_malformed": false,
      "store": false,
      "type": "long",
      "doc_values": true
    },
    "requestParameters.tagSpecificationSet.items.tags.value": {
      "eager_global_ordinals": false,
      "index_phrases": false,
      "fielddata": false,
      "norms": true,
      "index": true,
      "store": false,
      "type": "text",
      "index_options": "positions"
    },
    "requestParameters.DescribeEgressOnlyInternetGatewaysRequest": {
      "eager_global_ordinals": false,
      "index_phrases": false,
      "fielddata": false,
      "norms": true,
      "index": true,
      "store": false,
      "type": "text",
      "index_options": "positions"
    },
    "responseElements.credentials.sessionToken": {
      "eager_global_ordinals": false,
      "index_phrases": false,
      "fielddata": false,
      "norms": true,
      "index": true,
      "store": false,
      "type": "text",
      "index_options": "positions"
    },
    "message": {
      "type": "text"
    },
    "requestParameters.domainName": {
      "eager_global_ordinals": false,
      "index_phrases": false,
      "fielddata": false,
      "norms": true,
      "index": true,
      "store": false,
      "type": "text",
      "index_options": "positions"
    },
    "requestParameters.DescribeFlowLogsRequest": {
      "type": "text"
    },
    "requestParameters.maxItems": {
      "type": "text"
    },
    "tags": {
      "type": "text"
    }
  }
}

```

You suggest redefining new fields:  
[aws][cloudtrail][flattened][request\_parameters]  
[aws][cloudtrail][flattened][response\_elements]  
[aws][cloudtrail][flattened][additional\_eventdata]  
[aws][cloudtrail][flattened][service\_event\_details]

what type should I use? Text?

---

<div class="post-metadata">

**Author:** ![San9](https://avatars.discourse-cdn.com/v4/letter/s/a9adbd/32.png) [@San9](https://discuss.elastic.co/u/San9)\
**Post date:** [April 8, 2024, 3:00pm UTC](https://discuss.elastic.co/t/problem-elastic-document-parsing-exception-type-field/356690/13 "2024-04-08T15:00:54Z")

</div>

> [@leandrojmp](#):
>
> `flattened`

I create new template  
this is mappings field

```auto
{
  "_routing": {
    "required": false
  },
  "numeric_detection": false,
  "dynamic_date_formats": [
    "strict_date_optional_time",
    "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
  ],
  "_source": {
    "excludes": [],
    "includes": [],
    "enabled": true
  },
  "dynamic": true,
  "dynamic_templates": [],
  "date_detection": true,
  "properties": {
    "apiVersion": {
      "eager_global_ordinals": false,
      "index_phrases": false,
      "fielddata": false,
      "norms": true,
      "index": true,
      "store": false,
      "type": "text",
      "index_options": "positions"
    },
    "aws.cloudtrail.flattened.additional_eventdata": {
      "type": "flattened"
    },
    "aws.cloudtrail.flattened.service_event_details": {
      "type": "flattened"
    },
    "message": {
      "type": "text"
    },
    "aws.cloudtrail.flattened.response_elements": {
      "type": "flattened"
    },
    "tags": {
      "type": "text"
    },
    "aws.cloudtrail.flattened.request_parameters": {
      "type": "flattened"
    }
  }
}

```

is this what you mean?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [April 8, 2024, 3:15pm UTC](https://discuss.elastic.co/t/problem-elastic-document-parsing-exception-type-field/356690/14 "2024-04-08T15:15:00Z")

</div>

> [@San9](#):
>
> is this what you mean?

Yes, but your mappings are wrong.

It needs to be something like this:

```auto
        "aws": {
          "properties": {
            "cloudtrail": {
              "properties": {
                "flattened": {
                  "properties": {
                    "additional_eventdata": { "type":"flattened" },
                    "request_parameters": { "type":"flattened" },
                    "response_elements": { "type":"flattened" },
                    "service_event_details": { "type":"flattened" }
                  }
                },
                "additional_eventdata": { 
                  "type": "keyword",
                  "fields":{
                    "text": {
                      "type": "text"
                    }
                  }
                },
                "request_parameters": { 
                  "type": "keyword",
                  "fields":{
                    "text": {
                      "type": "text"
                    }
                  }
                },
                "response_elements": { 
                  "type": "keyword",
                  "fields":{
                    "text": {
                      "type": "text"
                    }
                  }
                },
                "service_event_details": { 
                  "type": "keyword",
                  "fields":{
                    "text": {
                      "type": "text"
                    }
                  }
                }
              }
            }
          }
        }

```

---

<div class="post-metadata">

**Author:** ![San9](https://avatars.discourse-cdn.com/v4/letter/s/a9adbd/32.png) [@San9](https://discuss.elastic.co/u/San9)\
**Post date:** [April 9, 2024, 12:09pm UTC](https://discuss.elastic.co/t/problem-elastic-document-parsing-exception-type-field/356690/15 "2024-04-09T12:09:12Z")

</div>

> [@leandrojmp](#):
>
> It needs to be something like this:

[Leandro Pereira](https://discuss.elastic.co/u/leandrojmp)  
Thanks for the recommendation. I slightly redesigned my template, now I don’t see any problems. I'll do one more test, let's hope everything goes well.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 7, 2024, 12:09pm UTC](https://discuss.elastic.co/t/problem-elastic-document-parsing-exception-type-field/356690/16 "2024-05-07T12:09:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.