Filebeat only partly updates documents

Elastic Stack Beats
1

Hi all,

Using Elastic I want to create a single source of thruth. In an index I would want to store a single record of an entity and update that record when its status changes.

The status of a entity gets shared via a .ndjson file which is read by Filebeat. In Filebeat a processor is put in the configuration which sets op_type to 'index' in the document metadata to make sure documents can be updated, also the documents are sent to an index, not a datastream.

All entities are already in Elastic with the entity ID being the document _id, I want to update their status which is stored in two seperate fields: 'ready' and 'blockers'.

Now the following strange things happen:

  • Whenever I add a single line to the .ndjson file, the @timestamp of the file being inested is changed in Elastic, but the other two fields are not.
  • Whenever I add a second line, similar as the previous line, to the .ndjson file, the entire document suddenly does get updated.

The document _id is set in an ingest pipeline, which works because the document gets updated, it being partly or completely.

Thanks in advance,
Rick

2

Hi @Rick_V

You will need to show us more... but filebeat is line oriented so if you add a line, it will not read it until there is a newline after it.. i.e. the next line... perhaps that is what you are seeing...

Add a few lines or in your test make sure you add a newline / empty line... check and see

3

Hi Stephen,

Thanks for your quick reply, and solution... As I was manually adding new lines to the file, I didn't add an empty newline and thus the records didn't get updated on the first time, but always on the second.

I was thinking more in de area of filebeat settings than actually checking whether the file I submitted was valid.

Thanks for your help.

1 Like