Is it possible to create two index for the same day using logstash templete

Elastic Stack Logstash
1

I am using logstash to send data elasticsearch on daily basis but the size of the indexes created is too large .that is why i want ot create two index for the same day.is it possible using logstash

2

This is typically handled by setting shards to a value that would get each shard to around 50G.

ILM is your future, it can rollover an index at size limit.

3

Can we do it like this for hourly logs
output{
elasticsearch{
index => index_name%{+YY.MM.dd.HH}
}
}

and if this is possible is there a way to do it for every 12 hours

4

Maybe it will be not ideal but you could do the following:
parse the date from @timestamp, and check if the hour in the timestamp is having a certain value.

grok {
#process the @timestamp
}

if [hour] >= 12 {
mutate {
add_field => {"index_rotator"=> "1"}
}
}
else {
mutate {
add_field => {"index_rotator"=> "2"}
}
}

so you can have a predictable:

output{
elasticsearch{
index => index_name%{+YY.MM.dd}-%{index_rotator}
}
}

Or just use the _ilm.
It is much better and this is for what it was designed to.

BTW. What is your usecase? To store some log data? To query that data you are loading?

5

yes it should work. I have tested with ww (weekly index) and it worked

6

Thanks @pastechecker it may be not ideal but for now it will serve the purpose.Meanwhile i will try to find if there is a better way

7

Hey @elasticforme Can you show me what index pattern you used for weekly index

8

in output section
output {
elasticsearch {
index => "resource-%{+YYYY.ww}"
}
}

or you can use xxxx.ww

Follow this discussion

https://discuss.elastic.co/t/solved-weekly-indexes-instead-of-daily/56871

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.