Elasticsearch-plugin install PLUGIN hangs

mlrodrigo · July 4, 2021, 11:09am

Installing an Elasticsearch plugin hangs after a few tries due to (likely) a lack of entropy in /dev/random:

bin/elasticsearch-plugin install analysis-smartcn --verbose
-> Installing analysis-smartcn
Checking if url exists: https://artifacts.elastic.co/downloads/elasticsearch-plugins/analysis-smartcn/analysis-smartcn-linux-aarch64-7.13.2.zip
-> Downloading analysis-smartcn from elastic
Retrieving zip from https://artifacts.elastic.co/downloads/elasticsearch-plugins/analysis-smartcn/analysis-smartcn-7.13.2.zip
[=================================================] 100%

Eventually after several minutes/hours, the plugin installs.

Stack trace when process is stuck:

"main" #1 prio=5 os_prio=0 cpu=1950.48ms elapsed=221.78s tid=0x0000ffff980225a0 nid=0x899f8 runnable  [0x0000ffff9ca05000]
   java.lang.Thread.State: RUNNABLE
	at java.io.FileInputStream.readBytes(java.base@16/Native Method)
	at java.io.FileInputStream.read(java.base@16/FileInputStream.java:271)
	at java.io.FilterInputStream.read(java.base@16/FilterInputStream.java:132)
	at sun.security.provider.NativePRNG$RandomIO.readFully(java.base@16/NativePRNG.java:424)
	at sun.security.provider.NativePRNG$RandomIO.implGenerateSeed(java.base@16/NativePRNG.java:441)
	- locked <0x00000000fd9573c0> (a java.lang.Object)
	at sun.security.provider.NativePRNG.engineGenerateSeed(java.base@16/NativePRNG.java:226)
	at java.security.SecureRandom.generateSeed(java.base@16/SecureRandom.java:857)
	at org.bouncycastle.crypto.util.BasicEntropySourceProvider$1.getEntropy(Unknown Source)
	at org.bouncycastle.crypto.fips.ContinuousTestingEntropySource.getEntropy(Unknown Source)
	- locked <0x00000000fc858128> (a org.bouncycastle.crypto.fips.ContinuousTestingEntropySource)
	at org.bouncycastle.crypto.fips.HashSP800DRBG.getEntropy(Unknown Source)
	at org.bouncycastle.crypto.fips.HashSP800DRBG.init(Unknown Source)
	at org.bouncycastle.crypto.fips.HashSP800DRBG.<init>(Unknown Source)
	at org.bouncycastle.crypto.fips.FipsDRBG$HashDRBGProvider.get(Unknown Source)
	at org.bouncycastle.crypto.fips.DRBGPseudoRandom.lazyInitDRBG(Unknown Source)
	at org.bouncycastle.crypto.fips.DRBGPseudoRandom.generate(Unknown Source)
	- locked <0x00000000fc853248> (a org.bouncycastle.crypto.fips.DRBGPseudoRandom)
	at org.bouncycastle.crypto.fips.FipsSecureRandom$RandomSpi.engineNextBytes(Unknown Source)
	- locked <0x00000000fc853248> (a org.bouncycastle.crypto.fips.DRBGPseudoRandom)
	at java.security.SecureRandom.nextBytes(java.base@16/SecureRandom.java:754)
	- locked <0x00000000fc8503a8> (a org.bouncycastle.crypto.fips.FipsSecureRandom)
	at java.math.BigInteger.randomBits(java.base@16/BigInteger.java:702)
	at java.math.BigInteger.<init>(java.base@16/BigInteger.java:691)
	at org.bouncycastle.util.BigIntegers.createRandomInRange(Unknown Source)
	at org.bouncycastle.math.internal.Primes.enhancedMRProbablePrimeTest(Unknown Source)
	at org.bouncycastle.crypto.asymmetric.KeyUtils.validatedModulus(Unknown Source)
	at org.bouncycastle.crypto.asymmetric.KeyUtils.validated(Unknown Source)
	at org.bouncycastle.crypto.asymmetric.AsymmetricRSAPublicKey.<init>(Unknown Source)
	at org.bouncycastle.jcajce.provider.ProvRSAPublicKey.<init>(Unknown Source)
	at org.bouncycastle.jcajce.provider.ProvRSA$RSAKeyFactory.engineGeneratePublic(Unknown Source)
	at java.security.KeyFactory.generatePublic(java.base@16/KeyFactory.java:346)
	at org.bouncycastle.openpgp.operator.jcajce.JcaPGPKeyConverter.implGeneratePublic(Unknown Source)
	at org.bouncycastle.openpgp.operator.jcajce.JcaPGPKeyConverter.getPublicKey(Unknown Source)
	at org.bouncycastle.openpgp.operator.jcajce.JcaPGPContentVerifierBuilderProvider$JcaPGPContentVerifierBuilder.build(Unknown Source)
	at org.bouncycastle.openpgp.PGPSignature.init(Unknown Source)
	at org.elasticsearch.plugins.InstallPluginCommand.verifySignature(InstallPluginCommand.java:640)
	at org.elasticsearch.plugins.InstallPluginCommand.downloadAndValidate(InstallPluginCommand.java:602)
	at org.elasticsearch.plugins.InstallPluginCommand.download(InstallPluginCommand.java:299)
	at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:242)
	at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:215)
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
	at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:80)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
	at org.elasticsearch.cli.Command.main(Command.java:79)
	at org.elasticsearch.plugins.PluginCli.main(PluginCli.java:36)

github.com

elastic/elasticsearch/blob/7.13/distribution/tools/plugin-cli/src/main/java/org/elasticsearch/plugins/InstallPluginCommand.java#L640

    
      
          
          
// validate the signature has key ID matching our public key ID
          final String keyId = Long.toHexString(signature.getKeyID()).toUpperCase(Locale.ROOT);
          if (getPublicKeyId().equals(keyId) == false) {
              throw new IllegalStateException("key id [" + keyId + "] does not match expected key id [" + getPublicKeyId() + "]");
          }
          
          
// compute the signature of the downloaded plugin zip
          final PGPPublicKeyRingCollection collection = new PGPPublicKeyRingCollection(ain, new JcaKeyFingerprintCalculator());
          final PGPPublicKey key = collection.getPublicKey(signature.getKeyID());
          signature.init(new JcaPGPContentVerifierBuilderProvider().setProvider(new BouncyCastleFipsProvider()), key);
          final byte[] buffer = new byte[1024];
          int read;
          while ((read = fin.read(buffer)) != -1) {
              signature.update(buffer, 0, read);
          }
          
          
// finally we verify the signature of the downloaded plugin zip matches the expected signature
          if (signature.verify() == false) {
              throw new IllegalStateException("signature verification for [" + urlString + "] failed");
          }

github.com

bcgit/bc-java/blob/master/core/src/main/java/org/bouncycastle/crypto/prng/BasicEntropySourceProvider.java#L53

    
      
                      {
                          // is the RNG regarded as useful for seeding?
                          if (_sr instanceof SP800SecureRandom || _sr instanceof X931SecureRandom)
                          {
                              byte[] rv = new byte[(bitsRequired + 7) / 8];
          
          
                    _sr.nextBytes(rv);
          
          
                    return rv;
                          }
                          return _sr.generateSeed((bitsRequired + 7) / 8);
                      }
          
          
            public int entropySize()
                      {
                          return bitsRequired;
                      }
                  };
              }
          }

Server specs:

Elasticsearch 7.13.2
Test node with very little traffic: AWS Graviton EC2 instance: r6g.xlarge behind a NAT
Ubuntu 20.04

Workarounds:

Wait (possibly hours) for the plugins to install
Increase entropy with tools such as rng-tools or haveged (untested)
?

Has anyone experienced this issue? Does verifying the download signature "$url.acm" need to read from /dev/random? Can this be switched to /dev/urandom so the install doesn't hang? Or perhaps I'm misunderstanding what the problem is.

system · August 1, 2021, 11:09am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.