Managing an arbitrary number of pipelines

I am working on a project with these requirements:

  1. Each event contains a field identifying the syslog server to send the output to.
  2. Transport failure to any server should not block output to other servers.

From requirement #2, it seems like I would need separate output pipelines. Considering that I am able to retrieve a list of possible syslog servers, I could generate a pipelines.yml file to have separate pipelines for each syslog server output.

However, this would also require a massive, dynamically generated conditional in the main pipeline to forward events to the correct output pipeline.

Is there a cleaner solution?

If you want an output to block if there is a transport failure then I think a dynamic configuration is the only option. If you are OK with dropping data that cannot be delivered then sending to syslog over UDP might be an option.

It is an option that comes with issues around message order, data integrity, data corruption, third party interception and other things, but I do not know if those are concerns for your use case.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.