Elasticsearch still writing to same index even After index rollover

[chandukreddi]

Hello Experts,

I am writing data into elastic search using logstash(see below) and using index template to enable some settings and aliases.

Logstash output:

output {
elasticsearch {
hosts => "10.1.27.6:9200"
index => "swift-proxy-log-%{+YYYY.MM.dd}-000001"
template => "swift_proxy_log_sizing_2.json"
template_name => "swift_proxy_log"
template_overwrite => true
}
}

Template:

[root@localhost config]# cat swift_proxy_log_sizing_2.json
{
"template": "swift_proxy_log",
"index_patterns": ["swift-proxy-log-*"],
"settings": {
"index.refresh_interval": "5s",
"index.codec": "best_compression",
"number_of_shards": 5,
"number_of_replicas": 0
},
"aliases": {
"logs_write_s": {}
}
}

I have noticed one thing after index rollover I see still data writing into old index swift-proxy-log-%{+YYYY.MM.dd}-000001 but alias is pointing to new index swift-proxy-log-%{+YYYY.MM.dd}-000002, any idea why this happening?

[root@localhost tmp]# curl -XGET 'localhost:9200/_alias/logs_write_s?pretty'
{
"swift-proxy-log-2018.03.20-000002" : {
"aliases" : {
"logs_write_s" : { }
}
}
}

I really appreciate your quick help.

Thanks
Chandra

[Badger]

@chandukreddi

I see still data writing into old index swift-proxy-log-%{+YYYY.MM.dd}-000001 but alias is pointing to new index swift-proxy-log-%{+YYYY.MM.dd}-000002, any idea why this happening?

Your config shows you are writing to the old index, not to the alias.

[chandukreddi]

@Badger

not to the alias

@Badger How do I point to alias?

[Badger]

@chandukreddi

How do I point to alias?

In you elasticsearch output set the index name to the alias.

  index => "logs_write_s"

[chandukreddi]

@Badger
After changing that it was not creating alias and my template not been applied to new index

in addition to your recommendation I have changed my template index_pattern => "index_patterns": [
"log_write_s*"

I do see template create in ES
GET /_template/swift_proxy_log
{
"swift_proxy_log": {
"order": 0,
"index_patterns": [
"log_write_s*"
],
"settings": {
"index": {
"codec": "best_compression",
"refresh_interval": "5s",
"number_of_shards": "5",
"number_of_replicas": "0"
}
},
"mappings": {},
"aliases": {
"logs_write_s": {}
}
}
}

I don't see alias is created

get _alias/logs_write_s?pretty
{
"error": "alias [logs_write_s] missing",
"status": 404
}

Index has been created but not applied my template
get logs_write_s/_settings
{
"logs_write_s": {
"settings": {
"index": {
"creation_date": "1521572633717",
"number_of_shards": "5",
"number_of_replicas": "1",
"uuid": "2EUfZ8PhSvWSSzRDS3iN8Q",
"version": {
"created": "6010199"
},
"provided_name": "logs_write_s"
}
}
}
}

[Badger]

@chandukreddi

get _alias/logs_write_s?pretty
{
"error": "alias [logs_write_s] missing",
"status": 404
}

Previously you showed that was an alias for swift-proxy-log-2018.03.20-000002. Did you delete that? If you want to write using the alias then the underlying index that is aliased to has to exist. Rollover should have updated the alias when it rolled to the new index.

I believe the template should refer to the underlying index name, not the alias, but you might want to ask a separate question for just that.

[chandukreddi]

@Badger.. I really appreciate your time..

Now all working fine and index => "logs_write_s" working perfect way..

Yes, your right template should refer to the underling index, not the alias

Thanks
Chandra