Best practices configuring indexes to periodic logs

Elastic Stack Logstash
1

Not sure if my search on the forum is correct (many hits on searchin in Indexes, so I'll try to post my question here:

We have several servers /switches/appliances which I would like to pull the logging into ELK. This is fairly straightforward and currently working nicely.

But,

I would like to know the best way to configure which log from which server is going into one of the folllowing indexes/Indices:

weekly
monthly
yearly
indefinate

This will also be my cleanup schedule.

By default into weekly (debugging/testing purpases) And then if need be into one of the others.

I am under the impression, that this can be configured in multiple ways, but I cannot imagine I am the only one tryingh to do this, so is there any information on this?

I tried to add filebeat as part of that cahin, but I didn't get it to work and I also have switches and appliances which do not have this option, so it would mean different type of configurations, which I am trying to avoid.

Is this something better to be configured in the output only, or it it better to use a combination of the filter and output configurations.

My idea, is to use only the output where I can possibly decide by server/log to which index I want to store the logs, but I may get into a situation, where Apache log from server A is 1 week and from server B is 1 month, But the syslog from these 2 servers need to bee a year for example.

Hopefully this makes any sense.

2

Depends on your volume, but weekly seems logical.

No.

Never.

3

But what is the best way to configure this?

To avoid complexity, it would be nice if you can decide which logfile from which source is going to which index with if...else statements.
But I get the idea this is not possible. I cannot seem to find a good manual on this, or at least one, I can understand.

4

Depends on your volumes and requirements.

You can, using conditionals!

5

Hi thanks for the confirmation,

Would this be a combination of the filter plugin and output plugin,
or can you do this simply in the output plugin conf.?

Would this be a good startingpoint: https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html

This suggest using both the filter and the output.

6

You can do it in filters and output, yes.