I am sending Linux system logs using filebeat to logstash.
my "logstash.conf" file is something like below (within double Line). I am trying to name index dynamically using "type" variable. But when I view it in Kibana, I see index name name is not changed as per the value of the "type" substitution variable but instead taking the format as a string.
My idea to see an index with name "log-Test' but instead its creating the index name as
log-%{type}-2020.12.10
not worked.
the index name became log-%type-2020.12.12.
I have used same code like you only added a tail of date. it seems "type" although ideally as per the beats plugin documentation beats plugin support Common configuration option "type" is supported, it is not actually getting picked up dynamically by elasticsearch output plugin. this says that " You cannot use dynamic variable substitution when ilm_enabled is true and when using ilm_rollover_alias"
Although it also says ...
"In order to minimize the number of open connections to Elasticsearch, maximize the bulk size and reduce the number of "small" bulk requests (which could easily fill up the queue), it is usually more efficient to have a single Elasticsearch output."...
This is very strange situation, my case, "index" in elasticsearch output plugine dynamically replaces the %{+YYYY.MM.dd} value but rejecting % type...
This is strange... what is the missing link? I am using logstash 7.9.3.
my filter section like below...
'''
filter {
mutate {
add_field => { "[@metadata][target_index]" => "testlog-%type-%{+YYYY.MM.dd}"}
}
}
'''
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.