# Elastic Serverless Forwarder for AWS adding reserved \_id field when sending to logstash

**URL:** <https://discuss.elastic.co/t/elastic-serverless-forwarder-for-aws-adding-reserved-id-field-when-sending-to-logstash/340084>\
**Category:** Elasticsearch\
**Created:** [August 3, 2023, 9:50pm UTC](https://discuss.elastic.co/t/elastic-serverless-forwarder-for-aws-adding-reserved-id-field-when-sending-to-logstash/340084 "2023-08-03T21:50:37Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![stabbotco1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stabbotco1/32/124293_2.png) [@stabbotco1](https://discuss.elastic.co/u/stabbotco1)\
**Post date:** [August 3, 2023, 9:50pm UTC](https://discuss.elastic.co/t/elastic-serverless-forwarder-for-aws-adding-reserved-id-field-when-sending-to-logstash/340084/1 "2023-08-03T21:50:37Z")

</div>

Hi All!  
I am new to ES, so apologies in advance if I mis-state some things.

We are looking to use the ES Serverless Forwarder for AWS ([Elastic Serverless Forwarder for AWS | Elastic Serverless Forwarder Guide | Elastic](https://www.elastic.co/guide/en/esf/current/aws-elastic-serverless-forwarder.html)) to send data to logstash before sending it on to our self hosted ES cluster.

The initial simple setup is using an emitting lambda that sends a json log event every minute to the logs, which the forwarder gets subscribed to as an event from CW when the event is emitted into the lambdas CW logs.

sample event:

```auto
{
    "timestamp": "2023-08-03T21:39:07.179193",
    "random_field": "constant string value here",
    "aws_request_id": "98f1d8fb-3361-474f-b525-10c6513edb36"
}

```

There are no filters, the event is getting triggered and sent to logstash, which returns a 200.

The logstash logs are showing a 400 with inability to create the index because the log event is containing a reserved field \_id.

error log snippet:  
{"type":"mapper\_parsing\_exception","reason":"failed to parse field [\_id] of type [\_id] in document with id 'jJdVvIkBlVXM\_z6z4Rmr'. Preview of field's value: '1691081882600-77a39f7173d6b4b6455fd7ae9f2fb147afd919365019f91f9bce160b1db21b100f45cc91f0c04489020b2725e53bacad-000000000000'","caused\_by":{"type":"mapper\_parsing\_exception","reason":"Field [\_id] is a metadata field and cannot be added inside a document. Use the index API request parameters."}}}}}.

We are trying this with es and logstash version 7.16.2 - which should be supported per the docs.

I understand \_id is reserved, but am surprised the forwarder is sending it as part of the document to be indexed, especially since the \_id field is not part of the meta data of the original event

I have seen suggestions on filtering out the field from the AWS CW subscription, as well as removing or renaming the field in logstash, but I am surprised this would be needed at all since the field is not present in the source, and I would not expect the ES addon to include this field in a minimal implementation with no mapping.

Any thoughts appreciated, and thank you all for reading!

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 3, 2023, 10:32pm UTC](https://discuss.elastic.co/t/elastic-serverless-forwarder-for-aws-adding-reserved-id-field-when-sending-to-logstash/340084/2 "2023-08-03T22:32:34Z")

</div>

@jsoriano any thoughts?

@stabbotco1

Can you share your logstash configuration please?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [August 4, 2023, 3:19am UTC](https://discuss.elastic.co/t/elastic-serverless-forwarder-for-aws-adding-reserved-id-field-when-sending-to-logstash/340084/3 "2023-08-04T03:19:07Z")

</div>

> [@stabbotco1](#):
>
> Any thoughts appreciated, and thank you all for reading!

I do not use the ESF, but looking for it the `_id` field is indeed created by the forwarder as this [github issue](https://github.com/elastic/elastic-serverless-forwarder/issues/212) makes clear.

For what I understood it is used to avoid duplicates in Elasticsearch.

The documentation on how the Logstash pipeline for the ESF should look like is non-existent, but I think that if you add the following line in your `elasticsearch` output in your Logstash configuration it should work:

```auto
document_id => "%{_id}"

```

This will tell Elasticsearch to use the `_id` field as the id of the document and should avoid the mapping error that you are getting.

---

<div class="post-metadata">

**Author:** ![stabbotco1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stabbotco1/32/124293_2.png) [@stabbotco1](https://discuss.elastic.co/u/stabbotco1)\
**Post date:** [August 4, 2023, 12:55pm UTC](https://discuss.elastic.co/t/elastic-serverless-forwarder-for-aws-adding-reserved-id-field-when-sending-to-logstash/340084/4 "2023-08-04T12:55:27Z")

</div>

Thank you! All the information was spot on and very helpful!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 1, 2023, 12:55pm UTC](https://discuss.elastic.co/t/elastic-serverless-forwarder-for-aws-adding-reserved-id-field-when-sending-to-logstash/340084/5 "2023-09-01T12:55:57Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.