Does filebeat support ordered pubsub?

Elastic Stack Beats
1

I am working on a project that sends events (two types - open and closed) to an ordered pubsub (verified that the subscription is ordered with an ordering key). I am then consuming this data from the pubsub into my elasticsearch database - where I have fingerprinting enabled to upsert data based on event type. So for instance, an open event would always be generated first, and be initially inserted and then a closed event would overwrite the open event once generated. However, if a closed event is consumed first, then the open would would overwrite the closed event and leave a dangling open event - which is what I am see in my database. So, I was wondering if filebeat supports this? Or should I look custom writer?

Thanks!

2

Are you actually overwriting the data in Elasticsearch?

3

Yes, I am and that is by design. I want to overwrite the open event when the closed event, and I am seeing that. In most cases, filebeat is consuming the open event first and then the closed event, as should always be the case. But sometimes that is not the case as I only see open events in my database. I checked to make sure that the closed event was coming in (via BigQuery) and it is coming in. So, the only thing I can think of is that filebeat might not guarantee ordered consumption (maybe it is consuming the closed event before the open event).

4

I don't believe Filebeat guarantees anything other than at-least-once delivery.

I would reconsider your approach though, updating is an expensive operation for Elasticsearch and you could be storing each event individually and just grabbing the latest event to show current state.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.