How to output filter results before applying next filter

Elastic Stack Logstash
1

Hi there, I would like to be able to debug my pipeline configuration. I apply a series of filters, and at the end I output to a json file. It works as it should, and a skeleton of the code would be this:

filter {
whatever1
}
filter {
whatever2
}
filter {
whatever3
}
output {
file { path=> output_file}
}

What Id like to do is, I would like to be able to output in between filters to different files, so that I can see what the event looks like after each filter. I would like to see the gradual effects of my filters so I can debug them separately. Something like:

filter {
whatever1
}
output {
file { path=> output_file_1}
}
filter {
whatever2
}
output {
file { path=> output_file_2}
}
filter {
whatever3
}
output {
file { path=> output_file_3}
}

But when I do this, all the outputs are executed after the filters have been applied. Is there a was to do this, or should I just gradually test the output of each filter, one by one?

2

There's not really a way to have an output at each stage as Logstash groups inputs->filters->outputs together. Your best bet is to test one chunch at a time.

3

You could use a clone filter to make a copy of events at certain points.

2 Likes
4

This was exactly what I wanted, thank you! This is the solution, if anyone comes to this thread.
Just make sure your filters and outputs use the correct types when you control the flow of the program, so that you dont apply filters to the created event clone types, and for those types you output to file.

Logstash will work as it should, grouping inputs, filters and outputs, but it works since the filters are not applied.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.