uberAgent Index Rollover - Steps


(Ryan Downey) #1

I've been working on rolling over an already established index that we have called uberAgent in our stack and I'm clearly misunderstanding the steps referenced in the Rollover Index docs. I've started out with a basic GET _aliases in the dev tools which shows that our uberagent index has no aliases. From what I understand in the docs "the rollover index API rolls an alias over to a new index...", the key word being alias, so I need to create an alias to start rolling over. How do I get the uberagent index to rollover using date math? Within the docs I see you can use PUT /<logs-{now/d}-1> with URI encoding: PUT /%3Clogs-%7Bnow%2Fd%7D-1%3E. This command just creates an entirely new index that I don't need and if I try to use a version of this to create an alias it just gives me a "Invalid alias name [<uberagent5.1-1]: must not contain the following characters [ , ", *, \, <, |, ,, >, /, ?]" error. Any help on showing me the correct order of operations for setting up the index so that it rolls over with date math would be appreciated.


(Ryan Downey) #2

So after working on this for a bit I'm realizing that there is no way to rollover an existing index if it doesn't end with - and a number, eg logs-1. I was under that impression that the index just needed an alias that was properly formated and could roll that over. Afer getting "reason": "index name [uberagent] does not match pattern '^.*-\d+$'" I finally figured out that it has nothing to do with my alias formatting but was looking for the properly formatted concrete index.


(Ryan Downey) #3

As you can see from my post above I'm trying to rollover an existing index every day with a date in the title. Our metricbeats are doing it as we get a new index every day that looks like this metricbeat-6.3.2-2018.10.12. This is what I've been trying to do with no success:
GET _aliases
//just to check the aliases of uber

//trying to rollover our uberagent index that's to large for our current environment. By changing the index to a daily format we can delete older indices as we see fit.
POST /_aliases

Blockquote {
"actions" : [
{ "add" : { "index" : "uberagent", "alias" : "%3Cuberagent-v5-%7Bnow%2Fd%7D-1%3E" } }
]
}

//
POST %3Cuberagent-v5-%7Bnow%2Fd%7D-1%3E/_rollover

Blockquote {
"conditions": {
"max_age" : "1d"
},
"settings": {
"number_of_shards" : 3,
"number_of_replicas" : 2
}
}

//this is the reply I get if I try this route

{
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "source alias does not exist"
}
],
"type": "illegal_argument_exception",
"reason": "source alias does not exist"
},
"status": 400
}

//I've tried commands like:
POST uberagent/_rollover/%3Cuberagent-v5-%7Bnow%2Fd%7D-1%3E

{
"conditions": {
"max_age" : "1d"
},
"settings": {
"number_of_shards" : 3,
"number_of_replicas" : 2
}
}

//This of course gets me an error message. From what I understand this is due to the fact that only an alias can be rolled over and not the original index.

{
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "source alias is a concrete index"
}
],
"type": "illegal_argument_exception",
"reason": "source alias is a concrete index"
},
"status": 400
}

I've added some other screenshots so that you can see the way uberagent is set up. It should be pointing to uberagent, which it is since we are gathering data from test computers. When I can get an uberagent alias to rollover and create a new index no data goes into the newly created index from uberagent which defeats the point of rolling over.


(Ryan Downey) #4

The default index for uberAgent according to the conf file is just uberagent, you can see it in one of my screenshots, if I change the default index to %3Cuberagent-v5-%7Bnow%2Fd%7D-1%3E inthe uberagent.conf file and do in kibana I might be able to roll over the index:
PUT /%3Cuberagent-v5-%7Bnow%2Fd%7D-1%3E

Blockquote
{
"aliases": {
"uberagent_today": { }
}
}

This seems like it would direct the information to a index that it properly formatted and once I get documents in the index I could roll it over.


(Matthew Isett) #5

@Ryan_Downey

Read over - https://www.elastic.co/blog/managing-time-based-indices-efficiently

I think you are close - but read about the _template concept, you can use the alias - and rollover the alias -
But with the template it will create the underlying uberagent-0001 index, and keep incrementing that. But you can use the alias uberAgent name in kibana.

Also check out curator - https://www.elastic.co/guide/en/elasticsearch/client/curator/current/rollover.html


(Ryan Downey) #6

Nice. I will take a look at things shortly. Appreciate your time and pointing me towards some new info.


(Matthew Isett) #7

Also can you remove the screen shots - and put the underlining text with formatting - makes it easier to read.

POST Here_is_some code/_rollover

Blockquote something else important

thanks @Ryan_Downey