Is it possible to make the:
curator.IndexList(es)
call do the filter at the same time?
We run multiple AWS lambdas, that use the python Curator and we are getting errors. I think it's because we run multiple lambdas at the same time (eg 1 per environment) and the IndexList will return ALL indices on the ES cluster, but while curator is paginating the GET requests, one of the other environments lambdas has deleted one or more of those indices which causes a 404 from that Lambda.
eg. (these times are made up and might not be realistic - but it shows what i suspect is happening)
01:00:00 Lambda A calls IndexList - gets index1, index2, index3
01:00:03 Lambda A calls GET for all the indices returned (as part of the IndexList call)
something like:
GET 'https://{ES_ENDPOINT}/index1,index2,index3'
01:00:04 Lambda A calls DeleteIndices for index3
01:00:05 index3 is deleted
01:00:00 Lambda B calls IndexList - gets index1, index2, index3
01:00:06 Lambda B calls GET for all the indices returned (as part of the IndexList call)
something like:
GET 'https://{ES_ENDPOINT}/index1,index2,index3'
01:00:07 Lambda B fails due to 404 response as index3 has been deleted already by Lambda A
After we run the IndexList call, we filter by regex so I think I'm looking for a way to have IndexList only bring back the ones that it would find in the Regex filter.
Is that possible? Or do you always need to return the whole list of indices before filtering them?
Alternatively, is there a way to make that IndexList call ignore any "missing" indices and not return a 404 for them?
Thanks!