Forming Index name for Elasticsearch plugin

Elastic Stack Logstash
#1

I am sending Linux system logs using filebeat to logstash.
my "logstash.conf" file is something like below (within double Line). I am trying to name index dynamically using "type" variable. But when I view it in Kibana, I see index name name is not changed as per the value of the "type" substitution variable but instead taking the format as a string.
My idea to see an index with name "log-Test' but instead its creating the index name as
log-%{type}-2020.12.10

image
image989×83 6.29 KB

=====================================================

input {
beats {
port => 5044
type => "Test"
}
}
output {
elasticsearch {
hosts => ["internal-elastic-elk-1817988673.ap-southeast-1.elb.amazonaws.com:9200"]
index => "log-%{type}"
}
}

#2

Hi ! i saw you tried to use syntax but you have to change # to (`)x3 on the forum

Try this

filter {
mutate{ add_field =>{"[@metadata][target_index]"=> "log-%{type}"}}
}
index => "%{[@metadata][target_index]}"
#3

not worked.
the index name became log-%type-2020.12.12.
I have used same code like you only added a tail of date. it seems "type" although ideally as per the beats plugin documentation beats plugin support Common configuration option "type" is supported, it is not actually getting picked up dynamically by elasticsearch output plugin. this says that " You cannot use dynamic variable substitution when ilm_enabled is true and when using ilm_rollover_alias"
Although it also says ...
"In order to minimize the number of open connections to Elasticsearch, maximize the bulk size and reduce the number of "small" bulk requests (which could easily fill up the queue), it is usually more efficient to have a single Elasticsearch output."...

This is very strange situation, my case, "index" in elasticsearch output plugine dynamically replaces the %{+YYYY.MM.dd} value but rejecting % type...

This is strange... what is the missing link? I am using logstash 7.9.3.
my filter section like below...
'''
filter {
mutate {
add_field => { "[@metadata][target_index]" => "testlog-%type-%{+YYYY.MM.dd}"}
}
}
'''

image
image1107×138 12.8 KB

#4

sprintf references require the {}. Change %type to %{type}.

#5

I did that too.... but no resolution... actually, after changing the string as

'''
filter {
mutate {
add_field => { "[@metadata][target_index]" => "testlog-%{type}-%{+YYYY.MM.dd}" }
}
}

'''
no new indexes are created for the day 13th Dec.
seems that's a wrong syntax then

image
image1098×170 14.9 KB

#6

Please show full config for further help i cannot understand how it's not working.

#7

Ok, lets have the entire thing together...

'''
input {
beats {
port => 5044
type => "Test"
}
}
filer {
mutate{ add_field =>{"[@metadata][target_index]"=> "log-%{type}"}}
}
output {
elasticsearch {
hosts => ["myeshost:9200"]
index => "%{[@metadata][target_index]}"
}
}
'''

#8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.