Failed to install S3 repository

Elastic Stack Elasticsearch
1

Hi team, recently we upgraded our cluster to be close to latest release.
We are using OSS version of stack, with current version 7.9.1:

rpm -qa | grep elasticsearch-oss
elasticsearch-oss-7.9.1-1.x86_64

However, we want to use S3 repo for backup of our indices, and issue we are facing is that ES is contacting wrong link for download of package:

./elasticsearch-plugin install repository-s3 -v
-> Installing repository-s3
Checking if url exists: https://artifacts.elastic.co/downloads/elasticsearch-plugins/repository-s3/repository-s3-linux-x86_64-7.9.1.zip
-> Failed installing repository-s3
-> Rolling back repository-s3
-> Rolled back repository-s3
Exception in thread "main" java.net.ConnectException: Connection timed out
        at java.base/sun.nio.ch.Net.connect0(Native Method)
        at java.base/sun.nio.ch.Net.connect(Net.java:503)
        at java.base/sun.nio.ch.Net.connect(Net.java:492)
        at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:588)
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:333)
        at java.base/java.net.Socket.connect(Socket.java:648)
        at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:290)
        at java.base/sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)
        at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:182)
        at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474)
        at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569)
        at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:265)
        at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:372)
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:177)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1194)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1082)
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:163)
        at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:142)
        at org.elasticsearch.plugins.InstallPluginCommand.urlExists(InstallPluginCommand.java:418)
        at org.elasticsearch.plugins.InstallPluginCommand.getElasticUrl(InstallPluginCommand.java:374)
        at org.elasticsearch.plugins.InstallPluginCommand.download(InstallPluginCommand.java:305)
        at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:251)
        at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:224)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
        at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:91)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
        at org.elasticsearch.cli.Command.main(Command.java:90)
        at org.elasticsearch.plugins.PluginCli.main(PluginCli.java:47)

Problem is that link https://artifacts.elastic.co/downloads/elasticsearch-plugins/repository-s3/repository-s3-linux-x86_64-7.9.1.zip does not exist, but S3 plugin exist on other link:
https://artifacts.elastic.co/downloads/elasticsearch-plugins/repository-s3/repository-s3-7.9.1.zip

I know that plugin can also be installed manually, but it would be easier for maintenance and scripting to use built in command instead of pulling from a hardcoded link.
It would be nice if this could be fixed and prevented in future releases, because I saw same issue in previous release 7.4.2 where I was also unable to install S3 with build in command.

Thanks,
Jovan

2

It looks like a bug to me.
Could you try with 7.9.3 or 7.10.0?

3

Hi David,

Honestly, we cannot upgrade anymore at this point.
We are using Opendistro for security purposes, and this version is latest that is supported according to their compatibility matrix:
https://opendistro.github.io/for-elasticsearch-docs/version-history/

I am sure that it was not working on 7.4.2. version, I just don't know if the issue is same on future releases.
Maybe a temporary fix would be to enable both links to work from ES perspective, and ensure that all future releases are pointing to same working links.

4

Ha! So it's not the official standard or oss distribution.

But this looks like our build?

elasticsearch-oss-7.9.1-1.x86_64

Not sure how this package is different from the official one and if this can be reproduced with a standard build.

It looks strange to me that no one complained about this though.

5

Not sure is it different, we use this repo for pulling OSS packages:

[elasticsearch-7.x]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/oss-7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
6

That's strange.

I started a Docker image with:

docker run -p 9200:9200 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch-oss:7.9.1

Then I ssh'ed into it.

Here is what I can see:

[root@1201200fcadc elasticsearch]# bin/elasticsearch-plugin install repository-s3 -v
-> Installing repository-s3
Checking if url exists: https://artifacts.elastic.co/downloads/elasticsearch-plugins/repository-s3/repository-s3-linux-x86_64-7.9.1.zip
-> Downloading repository-s3 from elastic
Retrieving zip from https://artifacts.elastic.co/downloads/elasticsearch-plugins/repository-s3/repository-s3-7.9.1.zip
[=================================================] 100%?? 
- Plugin information:
Name: repository-s3
Description: The S3 repository plugin adds S3 repositories
Version: 7.9.1
Elasticsearch Version: 7.9.1
Java Version: 1.8
Native Controller: false
Extended Plugins: []
...
7

Hi David,

I need to report additional issue with this. Just to mention that our infrastructure resides behind proxy which whitelisted .elastic.co domain.

While targeting link directly, I am unable to install plugin:

   https_proxy='<PROXY_URL>:<PORT>' http_proxy='<PROXY_URL>:<PORT>' ./elasticsearch-plugin install https://artifacts.elastic.co/downloads/elasticsearch-plugins/repository-s3/repository-s3-7.9.1.zip -v
   -> Installing https://artifacts.elastic.co/downloads/elasticsearch-plugins/repository-s3/repository-s3-7.9.1.zip
   -> Downloading https://artifacts.elastic.co/downloads/elasticsearch-plugins/repository-s3/repository-s3-7.9.1.zip
   Retrieving zip from https://artifacts.elastic.co/downloads/elasticsearch-plugins/repository-s3/repository-s3-7.9.1.zip
   -> Failed installing https://artifacts.elastic.co/downloads/elasticsearch-plugins/repository-s3/repository-s3-7.9.1.zip
   -> Rolling back https://artifacts.elastic.co/downloads/elasticsearch-plugins/repository-s3/repository-s3-7.9.1.zip
   -> Rolled back https://artifacts.elastic.co/downloads/elasticsearch-plugins/repository-s3/repository-s3-7.9.1.zip
   Exception in thread "main" java.net.ConnectException: Connection timed out
           at java.base/sun.nio.ch.Net.connect0(Native Method)
           at java.base/sun.nio.ch.Net.connect(Net.java:503)
           at java.base/sun.nio.ch.Net.connect(Net.java:492)
           at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:588)
           at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:333)
           at java.base/java.net.Socket.connect(Socket.java:648)
           at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:290)
           at java.base/sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)
           at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:182)
           at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474)
           at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569)
           at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:265)
           at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:372)
           at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:177)
           at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1194)
           at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1082)
           at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:163)
           at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1595)
           at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1523)
           at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:224)
           at org.elasticsearch.plugins.InstallPluginCommand.downloadZip(InstallPluginCommand.java:448)
           at org.elasticsearch.plugins.InstallPluginCommand.download(InstallPluginCommand.java:329)
           at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:251)
           at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:224)
           at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
           at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
           at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:91)
           at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
           at org.elasticsearch.cli.Command.main(Command.java:90)
           at org.elasticsearch.plugins.PluginCli.main(PluginCli.java:47)

However, if I use proxy to download package, and install using archive it works:

   https_proxy='<PROXY_URL>:<PORT>' http_proxy='<PROXY_URL>:<PORT>' wget https://artifacts.elastic.co/downloads/elasticsearch-plugins/repository-s3/repository-s3-7.9.1.zip
   --2020-11-16 19:10:56--  https://artifacts.elastic.co/downloads/elasticsearch-plugins/repository-s3/repository-s3-7.9.1.zip
   Connecting to 10.111.142.50:3128... connected.
   Proxy request sent, awaiting response... 200 OK
   Length: 4751033 (4.5M) [application/zip]
   Saving to: ‘repository-s3-7.9.1.zip’
   
   100%[=====================================================================================================================>] 4,751,033   3.00MB/s   in 1.5s
   
   2020-11-16 19:10:57 (3.00 MB/s) - ‘repository-s3-7.9.1.zip’ saved [4751033/4751033]

   ./elasticsearch-plugin install file:/usr/share/elasticsearch/bin/repository-s3-7.9.1.zip
   -> Installing file:/usr/share/elasticsearch/bin/repository-s3-7.9.1.zip
   -> Downloading file:/usr/share/elasticsearch/bin/repository-s3-7.9.1.zip
   [=================================================] 100%  
   @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
   @     WARNING: plugin requires additional permissions     @
   @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
   * java.lang.RuntimePermission accessDeclaredMembers
   * java.lang.RuntimePermission getClassLoader
   * java.lang.reflect.ReflectPermission suppressAccessChecks
   * java.net.SocketPermission * connect,resolve
   * java.util.PropertyPermission es.allow_insecure_settings read,write
   See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
   for descriptions of what these permissions allow and the associated risks.
   
   Continue with installation? [y/N]y
   -> Installed repository-s3
8

That is actually strange. I don`t know what ES is doing in background, but if you try to open that link it does not exist. And it starts downloading from different link after that:

I can assume that there is possible issue with starting this command behind a proxy as I mentioned in this post before.

9

I looked at the code and I can see this:

        final String platformUrl = String.format(
            Locale.ROOT,
            "%s/%s-%s-%s.zip",
            baseUrl,
            pluginId,
            platform,
            Build.CURRENT.getQualifiedVersion()
        );
        if (urlExists(terminal, platformUrl)) {
            return platformUrl;
        }
        return String.format(Locale.ROOT, "%s/%s-%s.zip", baseUrl, pluginId, Build.CURRENT.getQualifiedVersion());

That's how repository-s3-linux-x86_64-7.9.1.zip is rewritten to repository-s3-7.9.1.zip.

Could you try with the exact command listed in the documentation?

sudo ES_JAVA_OPTS="-Dhttp.proxyHost=host_name -Dhttp.proxyPort=port_number -Dhttps.proxyHost=host_name -Dhttps.proxyPort=https_port_number" bin/elasticsearch-plugin install repository-s3
1 Like
10

Yap, that actually works:

ES_JAVA_OPTS="-Dhttp.proxyHost=<PROXY_URL> -Dhttp.proxyPort=<PROXY_PORT> -Dhttps.proxyHost=<PROXY_URL> -Dhttps.proxyPort=<PROXY_PORT>" ./elasticsearch-plugin install repository-s3 -v
-> Installing repository-s3
Checking if url exists: https://artifacts.elastic.co/downloads/elasticsearch-plugins/repository-s3/repository-s3-linux-x86_64-7.9.1.zip
-> Downloading repository-s3 from elastic
Retrieving zip from https://artifacts.elastic.co/downloads/elasticsearch-plugins/repository-s3/repository-s3-7.9.1.zip
[=================================================] 100%  
- Plugin information:
Name: repository-s3
Description: The S3 repository plugin adds S3 repositories
Version: 7.9.1
Elasticsearch Version: 7.9.1
Java Version: 1.8
Native Controller: false
Extended Plugins: []
 * Classname: org.elasticsearch.repositories.s3.S3RepositoryPlugin
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.RuntimePermission getClassLoader
* java.lang.reflect.ReflectPermission suppressAccessChecks
* java.net.SocketPermission * connect,resolve
* java.util.PropertyPermission es.allow_insecure_settings read,write
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
-> Installed repository-s3

I see the actual difference between parsing Proxy now.
Thanks David!

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.