Elactic & Logstash infrastructure for diferent retention requeriment

[cdra]

We have a computer (C1) with elacticsearch and Kibana. Other computer (C2) with logstash installed.
Network computers send syslog messages to logstash. Logstash send messages to C1.These logs have some información: User access, equipment failures. etc....We keep this information for 6 months.

We need keep some syslog messages (those with facility 4 or 10) during 5 years, so I was thinking deploy extra computer (C3) with elacticsearch and kibana. C2 would send syslog messages (those with facility 4 or 10 facility 4 or 10) to C3 also.

is it a good design? other alternative?

[Christian_Dahlqvist]

When you use time-based indices, retention is managed by index. You could therefore simply create two separate indices (one per retention period) and have Logstash write events with facility 4 or 10 to the one with longer retention period. When doing this you probably want to adjust the time period each index type covers and e.g. use monthly indices for the index with longer retention period and a daily or weekly index for the other. You can easily do this within a single cluster.

[cdra]

My logstash config file is like:

if [type] == "syslog"
{
elasticsearch
{
hosts => ["xxxxxxxx:9200"]
}
}

So, it uses default index template....(example....logstash-2018.06.11)

How could create others logstash indexes for syslog with facility 4 or 10 using default logstash template ??

[Christian_Dahlqvist]

Once you have parsed out the facility from the log message, create two separate Elasticsearch outputs and use conditionals to send events to the correct plugin based on the facility value.

[cdra]

I know create different outputs using conditionals...but i suppose that I must configure "index => " parameter...not ??? how should I configure it for créate index by week ??

[Christian_Dahlqvist]

Yes, you need to specify two different index names for short and longer retention, e.g. logstash-short-%{+YYYY.MM.dd} and logstash-long-%{+YYYY.MM}.

[cdra]

My logstash template is like:

"logstash":
{
"order": 0,
"index_patterns": ["logstash-*"],
"settings": {"index": {
"number_of_shards": "1"
}

I suppose that is créate a new index like "logstash-long-%{+YYYY.MM}" it will use that template because index is like "logstash-*", not ???