Hi all. We are using the following pipeline config file to ship syslogs to Logstash:
input {
tcp {
port => 1514
codec => "json"
type => "syslog"
}
udp {
port => 1514
codec => "json"
type => "syslog"
}
}
filter {
# This replaces the host field (UDP source) with the host that generated the message (sysloghost)
if [sysloghost] {
mutate {
replace => [ "host", "%{sysloghost}" ]
remove_field => "sysloghost" # prune the field after successfully replacing "host"
}
}
}
output {
elasticsearch {
hosts => ["{IP}:9200","{IP}:9200","{IP}:9200","{IP}:9200"]
index => "logstash-%{+yyyy.MM.dd}"
}
}
This works, but the index is not getting tagged correctly when we look at the documents in Kibana. If I change the index line in the output block to something else - syslog-yyyy.MM.dd, logstash-yyyy.MM.dda, etc. - then everything is fine and the indices are created with the current date as you'd expect. If I leave them as shown with the standard logstash-yyyy.MM.dd pattern, it is not respected. Right now it's just using an old date with -00001 appended to the end, which I'm guessing is a rollover index or something.
I guess we can just use a different index pattern like syslog like I mentioned above, but I really would prefer to know how to fix this.
I appreciate the reply - I think I've wandered down this path in the past but I must not understand completely.
When changing the Index Lifecycle Policy for the logs Managed Index Template (which is the closest thing I can find that might apply to my current predicament) from 30 days or 50GB to 1 day, the index remains logstash-2022.03.29-000001. And, if I understand right, this policy would have already rolled this index over a long time ago based on the 30 day policy. I don't understand how we've been using the same index for 6 months.
Ideally, today for example, each document sent would get tagged with index logstash-2022.09.26, and increment with the days. Is this not possible using the default logstash-yyyy.MM.dd format?
And yes, that was what I wanted originally, but I think I've decided to just let ILM manage all of these for me and use the rollover pattern naming. I think the only reason we wanted daily indices is to be able to reliably expire them after they're a year old, but my understanding is that ILM can handle that using document data so I think doing it this way is fine.
The ILM will use the date in the index name as the creation date, for example for the index logstash-2022.03.29-000001 it will use 2022.03.29 as the date to calculate when it will move between the phases in the ILM policy.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.