I've been working on rolling over an already established index that we have called uberAgent in our stack and I'm clearly misunderstanding the steps referenced in the Rollover Index docs. I've started out with a basic GET _aliases in the dev tools which shows that our uberagent index has no aliases. From what I understand in the docs "the rollover index API rolls an alias over to a new index...", the key word being alias, so I need to create an alias to start rolling over. How do I get the uberagent index to rollover using date math? Within the docs I see you can use PUT /<logs-{now/d}-1> with URI encoding: PUT /%3Clogs-%7Bnow%2Fd%7D-1%3E. This command just creates an entirely new index that I don't need and if I try to use a version of this to create an alias it just gives me a "Invalid alias name [<uberagent5.1-1]: must not contain the following characters [ , ", *, \, <, |, ,, >, /, ?]" error. Any help on showing me the correct order of operations for setting up the index so that it rolls over with date math would be appreciated.
So after working on this for a bit I'm realizing that there is no way to rollover an existing index if it doesn't end with - and a number, eg logs-1. I was under that impression that the index just needed an alias that was properly formated and could roll that over. Afer getting "reason": "index name [uberagent] does not match pattern '^.*-\d+$'" I finally figured out that it has nothing to do with my alias formatting but was looking for the properly formatted concrete index.
As you can see from my post above I'm trying to rollover an existing index every day with a date in the title. Our metricbeats are doing it as we get a new index every day that looks like this metricbeat-6.3.2-2018.10.12. This is what I've been trying to do with no success:
GET _aliases
//just to check the aliases of uber
//trying to rollover our uberagent index that's to large for our current environment. By changing the index to a daily format we can delete older indices as we see fit.
POST /_aliases
Blockquote {
"actions" : [
{ "add" : { "index" : "uberagent", "alias" : "%3Cuberagent-v5-%7Bnow%2Fd%7D-1%3E" } }
]
}
//
POST %3Cuberagent-v5-%7Bnow%2Fd%7D-1%3E/_rollover
Blockquote {
"conditions": {
"max_age" : "1d"
},
"settings": {
"number_of_shards" : 3,
"number_of_replicas" : 2
}
}
//this is the reply I get if I try this route
{
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "source alias does not exist"
}
],
"type": "illegal_argument_exception",
"reason": "source alias does not exist"
},
"status": 400
}
//I've tried commands like:
POST uberagent/_rollover/%3Cuberagent-v5-%7Bnow%2Fd%7D-1%3E
{
"conditions": {
"max_age" : "1d"
},
"settings": {
"number_of_shards" : 3,
"number_of_replicas" : 2
}
}
//This of course gets me an error message. From what I understand this is due to the fact that only an alias can be rolled over and not the original index.
{
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "source alias is a concrete index"
}
],
"type": "illegal_argument_exception",
"reason": "source alias is a concrete index"
},
"status": 400
}
I've added some other screenshots so that you can see the way uberagent is set up. It should be pointing to uberagent, which it is since we are gathering data from test computers. When I can get an uberagent alias to rollover and create a new index no data goes into the newly created index from uberagent which defeats the point of rolling over.
The default index for uberAgent according to the conf file is just uberagent, you can see it in one of my screenshots, if I change the default index to %3Cuberagent-v5-%7Bnow%2Fd%7D-1%3E inthe uberagent.conf file and do in kibana I might be able to roll over the index:
PUT /%3Cuberagent-v5-%7Bnow%2Fd%7D-1%3E
Blockquote
{
"aliases": {
"uberagent_today": { }
}
}
This seems like it would direct the information to a index that it properly formatted and once I get documents in the index I could roll it over.
Read over - https://www.elastic.co/blog/managing-time-based-indices-efficiently
I think you are close - but read about the _template concept, you can use the alias - and rollover the alias -
But with the template it will create the underlying uberagent-0001 index, and keep incrementing that. But you can use the alias uberAgent name in kibana.
Also check out curator - https://www.elastic.co/guide/en/elasticsearch/client/curator/current/rollover.html
Nice. I will take a look at things shortly. Appreciate your time and pointing me towards some new info.
Also can you remove the screen shots - and put the underlining text with formatting - makes it easier to read.
POST Here_is_some code/_rollover
Blockquote something else important
thanks @Ryan_Downey
Curator error message. (Couldnt get this to post well with preformatted or block)
-Preparing Action ID: 1, "rollover"
-GET http://10.115.59.188:9200/ [status:200 request:0.005s]
-Trying Action ID: 1, "rollover": Rollover the index associated with alias 'uberagent-v5'. We are specifying the new index name using date math in the new_index line.
-"<uberagent-v5-{now/d}-1>" is using Elasticsearch date math.
-GET http://10.115.59.162:9200/_alias/uberagent-v5 [status:200 request:0.004s]
-ERROR Failed to complete action: rollover. <class 'ValueError'>: Unable to perform index rollover with alias "uberagent-v5". See previous logs for more details.
Does this mean that the original index, uberagent, is not properly formatted ie uberagent-1 and thus it can't be rolled over or is there something I'm missing in my files.
actions:
1:
action: rollover
description: >-
Rollover the index associated with alias 'uberagent-v5'. We are specifying the new index name using date math in the new_index line.
options:
name: uberagent-v5
new_index: "<uberagent-v5-{now/d}-1>"
conditions:
max_age: 1d
extra_settings:
index.number_of_shards: 3
index.number_of_replicas: 2
disable_action: False
GET aliases
"uberagent": {
"aliases": {
"uberagent-v5": {}Sorry. Been on vacation.
What index or indices does uberagent-v5 point to?
No worries man I've been working on a lot of other things within the Elastic environment. I hope you enjoyed your vacation. The alias uberagent-v5 currently points to the index uberagent.
"uberagent": { "aliases": { "uberagent-v5": {} } }
The index uberagent is not a rollover index, as it is not terminated by a dash followed by a number.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.