Split pipeline, multiple pipiline filter in logstash management center

As we as knew, with logstash pipeline, we can use multiple pipeline for multiple filter logstash on event and logstash will be execute pipeline order by naming of pipline
For example :

  • 001-input-beat.conf
  • 002-input-winevent.conf
  • 100-filter-beat-1.conf
  • 101-filter-beat-2.conf
  • 120-filter-winevent-1.conf
  • 121-filter-winevent-2.conf
  • 122-filter-winevent-3.conf
  • 123-filter-winevent-2.conf
  • 900-output-beat.conf
  • 900-output-winevent.conf

So in Elastic Stack 7.X version, have feature manager pippeline on Kibana (store config pipeline in ES). That is awesome feature. But i have some question :
How to make multiple filter for one pipeline
I have some pipeline in Pipeline center management :
Pipie-winevent
Pipie-beat
Pipie-nginx

Here is content of ls-winevent-pipeline

input {}
filter { 
  ## contain 120-filter-winevent-1.conf
  ## contain 121-filter-winevent-2.conf
  ## contain 122-filter-winevent-3.conf
  ## contain 123-filter-winevent-4.conf
}
output {}

Have problem, pipeline have very long content. How to split filter to other filter pipeline that does not affeect the other pipline

I don't think this is currently possible (at least not stable solution).
I have been waiting for the pipeline-to-pipeline communication to reach GA status in order to convert my pipelines for that. Then I could achieve what you are also trying to do.
So every filter (which forms sensible workload in order to be a pipeline) would be a separate pipeline and the filter pipelines would take input and push output into virtual addresses (other pipelines).

1 Like

Yeah, thank you so much, That's all i need. Hope in 7.4 version, Elastic team release GA :smiley:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.