Hi, I swear to god I don't believe myself when I am writng this.
I have a logstash configuration. it takes a CSV file we have and throws it into logstash. I have some ugly patches there that take information from the filename and also from another file using the ruby plugin, but nothing more.
Then I needed to make some changes (add files from another directory) so I copied this configuration to another file, changed the output from elastic to a file so I can debug the changes.
At some point I started getting an array of two results into a field that should have had one result, I ended up creating another field and removing the old field. but although I completely removed the old field - I still see it in my output. to be sure I am not dreaming, I delete the file every time and I stop logstash and start it again.
Another thing that happened is that information from that configuration found it way into elastic although my only output is the file. it sounds too strange but it is as if logstash mixed the two configurations together somehow.
I double (and triple) checked that I haven't copied the output from one file to another somehow, or the file configuration. this is just too strange!
Is there anyone from the elastic team that would like to see it....?
Just to check the most obvious solution, although you have been working with Logstash for a while: Did you define multiple pipelines in pipelines.yml or just create another file? Logstash concanates all conf files in conf.d to one big main pipeline if you don't configure multiple pipelines.
Oh wow, I wasn't aware of that! so you mean it takes all the files and make a huge pipeline out of them?
Okay, that's probably it. I used to have logstash on Windows a year ago, then left it and now I created a logstash server on linux. I remembered that I needed to configure pipline yaml but it just worked, so I never go to do it. this is probably the answer!
Pipelines can consist of multiple files. If a pipeline has multiple files that each contain one input, one filter and one output, Logstash will concatenate them in alphabetical order and will act as if you had one conf file with two inputs, two filters and then two outputs. The path for the files for a pipeline can be defined with glob expressions and I think the default for the main pipeline is just conf.d/*.conf
Yes Indeed that's exactly what I just saw in the pipeline YAML, I am probably missing something but I am not sure why anyone would prefer having a single pipeline consisting of many outputs and inputs. it's really confusing....
Just as an example for the usage:
I've got multiple pipelines with one worker thread (for the aggregate filter) that each that do the same, but get their filebeat input from different ports. So I separated the input in one file. All the other files listed in pipelines.yml are the same for all the pipelines, just the input file varies.
The transformation of the log lines is complicated, so I've split it into multiple files that only contain filters, but represent different logical steps.
And then the results are different event types that need different final filters and outputs. So there are multiple files that each contain one filter block and one output block with a condition if [event_type] == 'X' {...}.
This way the multiple files help me a lot to keep an overview and not lose my mind. I never have multiple files with all three, input, filter and output, all in one file, but sometimes two of them. And I used to have one pipeline with multiple inputs because I had to query multiple databases with the same structure, so everything after the input was the same.
… There are use cases for everything
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
