Is it possible to create a customized S3 client to connect via the repository-s3 plugin?

Elastic Stack Elasticsearch
1

The docs would lead you to believe that it's possible to create your own clients as part of the repository-s3 plugin.

PUT _snapshot/my_s3_repository
{
  "type": "s3",
  "settings": {
    "bucket": "my_bucket",
    "client": "my_alternate_client"
  }
}

Also this bit:

There are a number of storage systems that provide an S3-compatible API, and the repository-s3 plugin allows you to use these systems in place of AWS S3. To do so, you should set the s3.client.CLIENT_NAME.endpoint setting to the system’s endpoint. This setting accepts IP addresses and hostnames and may include a port. For example, the endpoint may be 172.17.0.2 or 172.17.0.2:9000 . You may also need to set s3.client.CLIENT_NAME.protocol to http if the endpoint does not support HTTPS.

How does one go about doing this?

References

2

I'd say that in 99% of the time you don't need this. But if you want to be able to restore from a region X and snapshot to a region Y, then you need to have 2 clients.

Is that what you are looking for? What exactly is your question?

3

We're looking into this option b/c we use Okta manage access to AWS through assumed roles. When we authenticate with Okta we're given a session token + temp. AWS tokens to access that are good for ~12 hrs. We'd like to create a wrapper script and introduce it as our own client to facilitate the loading of our temp creds into ES and then use them.

4

Is the ES server in AWS(ec2) or on prem and pushing to s3 via the internet?

If it is on prem, what do you currently do for all the other automation/code that uses AWS services? You wrap all of that in custom okta wrapper?

5

The ES cluster is on prem. For other automations we wrap it or use the AWS credentials file that Okta generates.

For example when you log into Okta using their CLI tool provided (Java)

$ cat ~/.aws/credentails
[someprofile_103123456789]
aws_access_key_id = ASIARQLRPLYSQERFVF12
aws_secret_access_key = 5i8tGlH3tpUFxX79wlg9gvCt47123RfeDwSs/+du
region = us-east-1
aws_session_token = FQoGZXIvYXdzED8aDGsIhKjmpLCmm6jtwyKYAiihhBH/AC/W5L7yDfCUu8YFK611toMPvxPJoUtDK9wyDpla4sq3K5sTMEkUg8ptWZvtNWo+AVI6p2UtoP04A9UcSvtvq8KqHDtYkP+o/u68axsfWwaF3+475mfMsxdFpW2V0uL58CJIlbnsWUObsW35wu9cYNJiciVAWz4MZq6ZLVRB68rHmLBJPOOwCvUACqRmyaB34MMaO2zF6KGG5VsBfx4IxG1IekihliLQe917e+fvg4TdD728CGRlDvVf3HsuHH6KSZ2cUNXolAfbJWWcfJsl8AkpQ8xo6T98fAuGapLBllXzBS+18URacDZfSXHpqas+bHSLhfEASrznWS2qZuCITse0RtDOtG81iHJyTBMQxoh2A8gotfHy4wU=

These aren't real creds, just examples.

6

The repository plugin is just meant for snapshot/restore. I don't see how you'd use it for your use case.

7

We're attempting to back up our .kibana* indices to S3 is the idea with this.

8

Why can you not put these credentials directly into Elasticsearch? Elasticsearch reads the credentials (access key, secret key and session token) from the keystore, and you can reload credentials at runtime.

I can think of other more complicated solutions (e.g. use Minio in S3 Gateway mode, or emulate EC2's own instance metadata service) but I'm trying to understand why the tools that Elasticsearch already provides aren't sufficient.

9

We can absolutely put the creds into ES. One of our issues is that the creds expire after 12 hrs and so new ones would need to get created prior to the 12 hr expiry window. How would you propose these get "reloaded"?

Part of my assumption with the keystore is that this would also need to get updated across all the instances of the ES cluster.

From ES docs:

When changing multiple reloadable secure settings, modify all of them, on each cluster node, and then issue a reload_secure_settings call, instead of reloading after each modification.

So in our scenario, I auth to Okta using their CLI tool, I get 3 things back (AWS tokens + session token), I need to add this to all my cluster nodes keystores and hit the API. 12hrs later I need to do the same thing.

With the pieces ES has provided me how would I accomplish this?

It feels like I have to do this outside of the cluster, update all the node's individual keystores, and then hit the API to trigger the reloading.

Am I missing something here?

BTW for reference this is the CLI for Okta - GitHub - oktadev/okta-aws-cli-assume-role: Okta AWS CLI Assume Role Tool.

10

No, that sounds right, and pretty simple too. Yet you seem reluctant to do this. Can you explain why?

11

Well for starters, it would be preferable if we could update the secret material via the API. what you're suggesting is that I would have to write something that every ~12hrs will SSH to all our ES nodes and run the keystore cmds to update the secrets on said nodes, and then reload them.

That feels very dirty and not really like a good way to do this.

1 Like
12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.