Help with correct big config file characterization and output to multiple index's

Elastic Stack Logstash
1

Hi guys,

I want to know what is the best way to maintain a big configuration file.
My plan is to get Different artifacts data through one port,
apply filters to every artifact depending on what artifact it is and outputing it to a relevant index.
so my questions are as follow:

• can i output data to a certain index given that i use an if statement that identify the artifact i want to send?
• can i dismantle the big configuration file somehow (preferably by filter for each artifact or something like that) so that it would be easier to manage? if so, how, sources would be appreciated.

2

There is no need to use a big configuration file, you can use multiple configuration files instead and logstash will merge them all in one.

If you are using a standard package installation you can create multiple configuration files in /etc/logstash/conf.d.

3

If they(the config files) all use the same input method and same port do i need to specify input in each config file?

4

When you use multiple config files, on starting, logstash will merge them and so it will act like a unique config file.

So you only need one declaration for your input, the same for you output. It's basically just splitting your file.

What I usually do is:

  • One input.conf
  • One output.conf
  • Multiple filter files named XXX_myfiltergoal.conf (XXX being a number)

The only thing you really need to pay attention, is to properly filter before output (so it is a yes for your 1st question) and the same for grok filter. (To be honest, if your have a proper config file, splitting it won't be complicated)

1 Like
5

If it merges all of the config file how will it know which output to send a certain data ?

6

How are you currently doing it?

7

i haven't executed a multiple config operation yet, i run each time a single config file and output it to elastic,
so i haven't figure out yet how to do it.

8

(I meant in your single conf file)
Then how do you want to separate the output?
Given the input or given a certain field value/tags?

9

I want to output each type of data (that can include several fields) to an index that would contains only this type of data.
i think by tags would be a good way, unless you know of a better one.
the input source would be the same for eveything so that wont be possible to output by the input.

10

Ok then, you can do the following (the same when you split as it will be merged on starting up):

I will assume 2 grok filter, that will determine the output.

filter{
 
 # Usefull (even necessary) when splitting to have a if statement
 if "SUCCESS" not in [tags] {
  grok{
   match => {"message" => "%{Some Pattern 1}"}

   remove_tag => ["_grokparsefailure"]
   add_tag => ["SUCCESS", "MyTag1"]
  }
 }

 if "SUCCESS" not in [tags] {
  grok{
   match => {"message" => "%{Some Pattern 2}"}

   remove_tag => ["_grokparsefailure"]
   add_tag => ["SUCCESS", "MyTag2"]
  }
 }

output{
 if "_grokparsefailure" not in [tags] {

  if "MyTag1" in [tags] {
   elasticsearch {
    ...
   }
  } else if "MyTag1" in [tags]  {
   elasticsearch {
    ...
   }
  } else {
    # Drop or to a default index or whatever
  }

 }
}
1 Like
11

thanks you've been extreamely helpful!

12

one last question, does logstash merge the config files by some order?
because for my knowledge the filter section executed by order, how does it know what to execute first among the different filter files?
the only possible way i can think of is putting every filter in an if statement and then the order will be redundant but then it will be less efficient.

13

one last question, does logstash merge the config files by some order?

Yes, alphabetical order.

1 Like
14

That's why I said I use:

1 Like
15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.