Elastic 7 mantaining a single Index

[Merlin_Nunez (Merlin Nuñez)]

Hi All,

I updated our lower environment stack from version 6.3 to version 7, last week, I´m using docker images to deploy elasticsearch+kibana+logstash to a single debian box using docker-compose.
I did a complete wipe of all the data and finally got it all working. I kept my original filter for logstash intact.
Now I notice that Elasticsearch is keeping a single index with all the documents inside:

On the previous version, a new index was created every day and I used elasticsearch curator to clean old index.
As I discovered the new index policies I wanted to leverage that to manage my indexes lifecycle, so I configured the following:

{
    "policy": {
        "phases": {
            "hot": {
                "min_age": "0ms",
                "actions": {
                    "rollover": {
                        "max_age": "1d",
                        "max_size": "200mb"
                    },
                    "set_priority": {
                        "priority": 1
                    }
                }
            },
            "warm": {
                "min_age": "3d",
                "actions": {
                    "set_priority": {
                        "priority": 2
                    }
                }
            },
            "cold": {
                "min_age": "6d",
                "actions": {
                    "set_priority": {
                        "priority": 3
                    }
                }
            },
            "delete": {
                "min_age": "15d",
                "actions": {
                    "delete": {}
                }
            }
        }
    }
}

Several days have passed now, and I still have a single index.
My filter, has no configuration over index generation, as I was using the default pattern logstash-2019.05.29

Thanks in advance,

PS: IDK how to format code
Regards,

[thekm1 (Mathew Thekkekara)]

Hi @Merlin_Nunez
You can edit code with the quote inside your wysiwyg editor

Then the code should be shown in

So to the problem.
Can you discribe which steps you used to setup the ilm?
Is logstash writing to the write_alias of the index?
Is the policy added to the index_template of the logstash template?

Thank you

[Merlin_Nunez (Merlin Nuñez)]

I don´t know what ilm means.
Logstash is writing to the default alias, since I used this configuration:

output {
elasticsearch {
hosts => [ "${ELASTICSEARCH_HOST:192.168.10.125}" ]
}
}
Regarding the policy, I have only one and is showing in the index stats:

Thank you so much for your help and time!

[thekm1 (Mathew Thekkekara)]

Hi @Merlin_Nunez
ILM is Index Lifecycle Management. That is the feature generating the 6 digit number at the end of your index. This feature is handling of new index creation based on parameter(index size, number of docs in index) that you define in the index lifecycle policy.

For getting data from logstash to elasticsearch with the ILM you can follow this doc part

Basically you need to set the output plugin in logstash to push data into the index which is the alias for your logstash index "logstash-2019.05.29-000001"
I am not sure but, maybe try this:
add to the logstash output plugin inside elasticsearch index: "logstash"

output {
elasticsearch {
hosts => [ "${ELASTICSEARCH_HOST:192.168.10.125}" ]
index: "logstash"
}
}

ps. I am not familiar with "hosts" => ["${ELASTICSEARCH_HOST:192.168.10.125}"], but i assume that is working because you have data in you logstash index mentioned. Otherwise try hosts: "<elasticsearch_ip>:<http_port> for example hosts: "127.0.0.1:9200"
here is the doc.

Hope you get a step forward

[Merlin_Nunez (Merlin Nuñez)]

As you can see from my first post, I have ILM configured, but it doesnt seem to be activating because the index keeps growing and growing.
Yeah sorry, that is because the actual host is passed as a parameter, the correct configuration would be:

output {
elasticsearch {
hosts => [ "${ELASTICSEARCH_HOST}" ]
}
}
And then I pass the host as IP:port
IN theory for logstash the default value for the index parameter is logstash-{date}-{rollup}

[thekm1 (Mathew Thekkekara)]

@Merlin_Nunez
Yes I can see from your first post that the ILM is configured and also that you have a index alias called logstash but as you said the default in output plugin elasticsearch for index value is "logstash-%{+YYYY.MM.dd}" So I think that you should write from logstash to the alias, which is called logstash
Did you tried to set the index in elasticsearch output to index: "logstash"? Did it worked?

What make a bit curious as well is that your index is in action rollover. And this this means that a new index going to get created.
Please try this.

• Set the index in elasticsearch output to index: "logstash"? Did it worked?
• Run the ilm explain on that index. and post the response here