Providing document_id in function beat for decode_json_fields results in effectively no updates to any records

Elastic Stack Beats
#1

Hey guys;

I'm processing records from DynamoDB via a function beat and it's working great, if I specify the source record as the "document_id" for the "decode_json_fields" process Elastic correctly uses this as the ID for the resulting document in ES.

But what I notice is that when that same record in the source database is updated and it passes through the function beat again ES doesn't seem to be doing an upsert. I don't get a dupe record which is good, but ES doesn't reflect the updated fields.

Any pointers greatly appreciated.

Thanks!

Reference:

#2

Found this old post which sounds relevant.

"No, this is not something that Beats were intended to accomplish. Enrichment like you describe is one of the things that Logstash can help with."

#3

Hmmm looking like this is either a bug or just not supported. Seems like it should be a feature. What's the point of being able specify the id if it means the record can never recieve updates.

#4

For anyone out there that has the same need: that being needing to do upserts on the index (by being able to control the document id).

The official word is that this is not supported by Function Beat. In order to do this one needs to either use Logstash or write to ES directly.