Registering S3 Repo with AWS ElasticSearch Service


(Ninjada) #1

Not sure this is the right place, but i've migrated the elasticsearch portion of my existing ELK stack from individual EC2 nodes, onto AWS ElasticSearch Service.

My previous Elasticsearch cluster had automated backups with indices management to an s3 bucket, which im trying but failing to replicate, as I can't register the repo on the AWS ElasticSearch Service...

Due to the auth requirements on AWS, its unable to be done simply by curl and so im following the guide here:

https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-snapshots.html

however, the python "register" script example code in those instructions used to register the repo keeps failing.. First of all, i get errors

"TypeError: request() got an unexpected keyword argument 'json'"

unless i change the code from json=payload to data=payload... but after that change i get

401
{"Message":"Your request: '/_snapshot/elasticsearch-backups' is not allowed due to invalid input parameters."}

Not sure why the official AWS documentation is broken, nor can i find any info on the required parameters. Im thinking this documentation probably worked for es version 5 and something may have changed with elasticsearch 6 in terms of params required? nor can i find details for the es params for repo type s3 in the es version 6 docs.

Ive tried a couple other python scripts also, but they seem further out of date and i just get other code errors.

Any ideas or recommendations? surely many people have managed this successfully

Im running the latest versions for everything:
AWS Elasticsearch version 6.2
On my kibana node where im runnign the script; Python 2.7.x etc


(David Pilato) #2

I think you should ask on AWS forum.

BTW did you look at https://www.elastic.co/cloud and https://aws.amazon.com/marketplace/pp/B01N6YCISK ?

Cloud by elastic is the only way to have access to X-Pack. Think about what is there yet like Security, Monitoring, Reporting and what is coming like Canvas, SQL...


(Ninjada) #3

yeh no problems, ill await a response there. its very slow.

i have looked at those offerings, to be honest this is a very specific elk implementation fed by lambda. other more important stuff will probably go that route.


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.