Hey guys; I'm processing records from DynamoDB via a function beat and it's working great, if I specify the source record as the "document_id" for the "decode_json_fields" process Elastic correctly uses this as the ID for the resulting document in ES. But what I notice is that when that same recor…

For anyone out there that has the same need: that being needing to do upserts on the index (by being able to control the document id). The official word is that this is not supported by Function Beat. In order to do this one needs to either use Logstash or write to ES directly.

Providing document_id in function beat for decode_json_fields results in effectively no updates to any records

Elastic Stack Beats

ryantomaselli (Ryan Tomaselli) February 6, 2021, 10:00pm 3

Hmmm looking like this is either a bug or just not supported. Seems like it should be a feature. What's the point of being able specify the id if it means the record can never recieve updates.

Topic		Replies	Views
Functionbeat does not update documents with existing _id Beats functionbeat	5	515	December 9, 2020
Create new version of existing document / enforce the id Beats beats-development	14	3084	February 20, 2017
Update existing document with fields from incoming document Logstash	1	725	August 8, 2017
Duplication in Filebeat to Elasticsearch data pushing Beats filebeat	5	702	December 28, 2017
Deduplicate data Beats filebeat	4	513	July 15, 2022

Providing document_id in function beat for decode_json_fields results in effectively no updates to any records

Related topics