Snapshot to S3 via corporate proxy

Elastic Stack Elasticsearch
1

I would like to use an S3 repository for snapshots, but access is via a corporate HTTP proxy that requires authentication.
Currently I'm using a shared file system repository and then using proxychains with S3cmd to send the snapshot to AWS.
Is there some way I can configure Elasticsearch to use the proxy so I can send directly?
I see there is a proxy uri option, but I don't see any option for authentication.

2

You can try something similar to https://www.elastic.co/guide/en/elasticsearch/plugins/current/_other_command_line_parameters.html#_proxy_settings

You can try setting http.proxyUser and http.proxyPassword.

But I'm unsure if S3 SDK will use that.

Can you test this?

Also, can you open an issue in elasticsearch project with your findings?

So we can either:

  • fix it by code if S3 SDK has this option
  • document it if it's only a parameter fix

Thanks!

3

Thanks, I tried http.proxyUser and http.proxyPassword to try a plugin install but this returned 'unknown' for http.proxyUser.
So the command line was 'sudo bin/plugin install mobz/elasticsearch-head -DproxyHost=host_name -DproxyPort=port_number http.proxyUser=username http.proxyPassword=password'
Also tried -http.proxyUser and -Dhttp.proxyUser
Currently using ES 1.44
I'll open an issue in elasticsearch project as you recommended.

4

You need to pass that within JAVA_OPTS IIRC.

5

As the cluster takes around 40 mins to rebalance after restarting elasticsearch can I just confirm the following settings?

Editing /bin/elasticsearch.in.sh, I tried the following but still could not install a plugin, failed to connect (/bin/plugin install mobz/elasticsearch-head -DproxyHost=192.168.0.1 -DproxyPort=8080)

Set Proxy

JAVA_OPTS="$JAVA_OPTS -Dhttp.proxyUser=username"
JAVA_OPTS="$JAVA_OPTS -Dhttp.proxyPassword=password"

6

I think you should add also http.proxyHost and http.proxyPort to JAVA_OPTS

Can you test that on a test cluster so it will be harmless?

7

Plugins install OK now with your recommendations.

Unfortunately a PUT S3 repository request returns the following error:

Service: Amazon S3; Status Code: 407; Error Code: 407 Proxy Authentication Required.

8

Can you open an issue and link to this thread?

1 Like