Is publishing events thread safe?

Elastic Stack Beats
1

The developer guide shows how to create an instance of publisher.Client, but I'm wondering if it's PublishEvent and PublishEvents methods are thread safe.

I'm currently working on a beat that receives HTTP requests and uses the request body to generate events. Since each request in handled in a separate go routine, is it safe to have these go routines call the PublishEvents method of the same instance of a publisher.Client at the same time? If not, are there any alternatives for publishing events in a thread safe way? I can always use a mutex if there is no thread safe method to publish events.

2

In 5.x they are thread-safe. In 6.x the publish interfaces are somewhat reworked. In 6.x they are still thread-safe (for now), but there is no actual guarantee. In general it is good practice to create a publisher.Client instance per worker. All clients publish events to a shared queue.

Some beats use another tactic. They have some 'publisher' workers instead of a mutex. The data collectors forward events to the publisher workers (using a go channel). The publisher workers might add some more data to events or do some additional filtering and finally Publish the events to libbeat. e.g. packetbeat uses this tactic in order to drop events if the publisher workers are blocked to due back-pressure in the outputs.

3

thanks for the detailed response @steffens, that was exactly what I was looking for!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.