I have a requirement for pull the data from two separate sources. I'm using 'file' and 'jdbc' as input plugin in the logstash configuration.
In the output, I've segragated elasticsearch indices based on the types.
Below is my output config:
output {
if [type] == "abc_logs" {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => true
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
template => "/etc/logstash/conf.d/abc-template.json"
template_name => "custom-abc-template"
ilm_enabled => true
ilm_pattern => "000001"
}
}
if [type] == "def_transactions" {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => true
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
template => "/etc/logstash/conf.d/def-template.json"
template_name => "custom-def-template"
ilm_enabled => true
ilm_pattern => "500001"
}
}
}
Below are my two different index templates:
abc-template.json:
{
"index_patterns" : ["abclogstash-0*"],
"settings" : {
"number_of_shards" : 5,
"index.lifecycle.name":"abc-logstash-policy",
"index.lifecycle.rollover_alias":"abclogstash"
},
"aliases" : {
"abc-alias" : {}
}
}
def-template.json:
{
"index_patterns" : ["deflogstash-0*"],
"settings" : {
"number_of_shards" : 5,
"index.lifecycle.name":"def-logstash-policy",
"index.lifecycle.rollover_alias":"deflogstash"
},
"aliases" : {
"def-alias" : {}
}
}
Now the problem is the when I'm starting the ELK container, the template is being overriden by logstash and creating same index.lifecycle.rollover_alias for both the indices. This is causing problem in elasticsearch. Below are the logs.
==> /var/log/elasticsearch/elasticsearch.log <==
[2019-09-18T15:15:09,916][INFO ][o.e.c.m.MetaDataIndexTemplateService] [3Qz6On0] adding template [custom-abc-template] for index patterns [logstash-]
[2019-09-18T15:15:09,948][INFO ][o.e.c.m.MetaDataIndexTemplateService] [3Qz6On0] adding template [custom-def-template] for index patterns [logstash-]
[2019-09-18T15:15:10,068][INFO ][o.e.c.m.MetaDataCreateIndexService] [3Qz6On0] [logstash-000001] creating index, cause [api], templates [custom-abc-template, custom-def-template], shards [5]/[1], mappings [doc]
==> /var/log/logstash/logstash-plain.log <==
[2019-09-18T15:15:09,318][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://localhost:9200/]}}
[2019-09-18T15:15:09,539][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-09-18T15:15:09,613][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-09-18T15:15:09,616][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>6}
[2019-09-18T15:15:09,644][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-09-18T15:15:09,666][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://localhost:9200/]}}
[2019-09-18T15:15:09,694][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-09-18T15:15:09,707][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-09-18T15:15:09,707][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>6}
[2019-09-18T15:15:09,713][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-09-18T15:15:09,790][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>"/etc/logstash/conf.d/abc-template.json"}
[2019-09-18T15:15:09,795][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>"/etc/logstash/conf.d/def-template.json"}
[2019-09-18T15:15:09,817][INFO ][logstash.outputs.elasticsearch] Overwriting index patterns, as ILM is enabled.
[2019-09-18T15:15:09,824][INFO ][logstash.outputs.elasticsearch] Overwriting index lifecycle name and rollover alias as ILM is enabled.
[2019-09-18T15:15:09,827][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-", "settings"=>{"number_of_shards"=>5, "index.lifecycle.name"=>"logstash-policy", "index.lifecycle.rollover_alias"=>"logstash"}, "aliases"=>{"mse-alias"=>{}}}}
[2019-09-18T15:15:09,835][INFO ][logstash.outputs.elasticsearch] Overwriting index patterns, as ILM is enabled.
[2019-09-18T15:15:09,836][INFO ][logstash.outputs.elasticsearch] Overwriting index lifecycle name and rollover alias as ILM is enabled.
[2019-09-18T15:15:09,843][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-", "settings"=>{"number_of_shards"=>5, "index.lifecycle.name"=>"logstash-policy", "index.lifecycle.rollover_alias"=>"logstash"}, "aliases"=>{"atg-alias"=>{}}}}
[2019-09-18T15:15:09,860][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/custom-abc-template
[2019-09-18T15:15:09,867][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/custom-def-template
[2019-09-18T15:15:09,958][INFO ][logstash.outputs.elasticsearch] Using Index lifecycle management - this feature is currently in beta.
[2019-09-18T15:15:09,987][INFO ][logstash.outputs.elasticsearch] Using Index lifecycle management - this feature is currently in beta.
[2019-09-18T15:15:09,988][WARN ][logstash.outputs.elasticsearch] Overwriting supplied index name with rollover alias logstash
[2019-09-18T15:15:09,989][WARN ][logstash.outputs.elasticsearch] Overwriting supplied index name with rollover alias logstash
[2019-09-18T15:15:10,029][INFO ][logstash.outputs.elasticsearch] Creating rollover alias
[2019-09-18T15:15:10,031][INFO ][logstash.outputs.elasticsearch] Creating rollover alias
==> /var/log/elasticsearch/elasticsearch.log <==
[2019-09-18T15:15:10,300][WARN ][r.suppressed ] [3Qz6On0] path: /%3Clogstash-500001%3E, params: {index=}
java.lang.IllegalStateException: alias [logstash] has more than one write index [logstash-000001,logstash-500001]
at org.elasticsearch.cluster.metadata.AliasOrIndex$Alias.computeAndValidateWriteIndex(AliasOrIndex.java:173) ~[elasticsearch-6.6.1.jar:6.6.1]
at org.elasticsearch.cluster.metadata.MetaData$Builder.lambda$buildAliasAndIndexLookup$1(MetaData.java:1158) ~[elasticsearch-6.6.1.jar:6.6.1]