Esrally configure does not find jdk on AWS linux (centos based)

Elastic Stack Elasticsearch
1

When I run "esrally configure" it does not detect JDK8, which i have installed.

yum list installed | grep jdk

java-1.7.0-openjdk.x86_64 1:1.7.0.111-2.6.7.2.68.amzn1 installed
java-1.8.0-openjdk.x86_64 1:1.8.0.111-1.b15.25.amzn1 @amzn-updates
java-1.8.0-openjdk-devel.x86_64 1:1.8.0.111-1.b15.25.amzn1 @amzn-updates
java-1.8.0-openjdk-headless.x86_64 1:1.8.0.111-1.b15.25.amzn1 @amzn-updates

java -version

openjdk version "1.8.0_111"
OpenJDK Runtime Environment (build 1.8.0_111-b15)
OpenJDK 64-Bit Server VM (build 25.111-b15, mixed mode)

Looking at this file, it looks for /etc/alternatives/java_sdk_1.%d.0

But on this os (AWS linux) /etc/alternatives/java_sdk_1.8.0 points to /usr/lib/jvm/java-1.8.0-openjdk.x86_64
ls -ltrh /etc/alternatives/java_sdk_1.8.0
lrwxrwxrwx 1 root root 38 Nov 3 20:24 /etc/alternatives/java_sdk_1.8.0 -> /usr/lib/jvm/java-1.8.0-openjdk.x86_64

and then /usr/lib/jvm/java-1.8.0-openjdk.x86_64 to java-1.8.0-openjdk-1.8.0.111-1.b15.25.amzn1.x86_64
ls -ltrh /usr/lib/jvm/java-1.8.0-openjdk.x86_64
lrwxrwxrwx 1 root root 50 Nov 3 19:48 /usr/lib/jvm/java-1.8.0-openjdk.x86_64 -> java-1.8.0-openjdk-1.8.0.111-1.b15.25.amzn1.x86_64

And that double pointing seem trip it up (i think).
Nonetheless, I enter "/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-1.b15.25.amzn1.x86_64" as the java home but when I run "esrally --distribution-version=5.0.0"
I get a huge error:

ERROR: Cannot race
Reason: This race ended early with a fatal crash. For details please see the logs.
And in the logs i see:
2016-11-03 20:34:27,464 root ERROR Cannot run subcommand [race].
Traceback (most recent call last):
File "/usr/local/lib/python3.4/site-packages/esrally/config.py", line 109, in opts
return self._opts[self._k(scope, section, key)]
KeyError: 'source::local.src.dir'

Any ideas how to get this to work?

1 Like
2

Hi @Alexander_Gray_II,

Yes, this is definitely the cause of the problem. Could you please open a ticket on Github?

Yes, you are missing a configuration value but it seems to me we can reconstruct it together. Can you please run:

cat ~/.rally/rally.ini

and paste the output? Please also check that you scrub the values for datastore.user and datastore.password before posting in case they are set.

Daniel

3

Here is my rally.ini:

# cat /root/.rally/rally.ini
[meta]
config.version = 6

[system]
root.dir = /root/.rally/benchmarks
log.root.dir = logs
env.name = local

[provisioning]
local.install.dir = install

[runtime]
java8.home = /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-1.b15.25.amzn1.x86_64

[benchmarks]
local.dataset.cache = ${system:root.dir}/data

[reporting]
datastore.type = in-memory
datastore.host =
datastore.port =
datastore.secure =
datastore.user =
datastore.password =

[tracks]
default.url = https://github.com/elastic/rally-tracks

[defaults]
preserve_benchmark_candidate = False

Here is the output of the esrally configure command:

# esrally configure

    ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/

Running simple configuration. You can run the advanced configuration with:

  esrally configure --advanced-config

[✓] Autodetecting available third-party software
which: no gradle in (/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/opt/aws/bin)
  git    : [✓]
  gradle : [✕]
  JDK 8  : [✕] (You cannot benchmark Elasticsearch 5.x without a JDK 8 installation)

**********************************************************************************
You don't have the necessary software to benchmark source builds of Elasticsearch.

You can still benchmark binary distributions with e.g.:

  esrally --pipeline=from-distribution --distribution-version=5.0.0-alpha5

See https://esrally.readthedocs.io/en/latest/pipelines.html#from-distribution
**********************************************************************************

[✓] Setting up benchmark data directory in [/root/.rally/benchmarks] (needs several GB).

Enter the JDK 8 root directory:: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-1.b15.25.amzn1.x86_64

[✓] Configuration successfully written to [/root/.rally/rally.ini]. Happy benchmarking!

To benchmark Elasticsearch 5.0.0-alpha2 with the default benchmark run:

  esrally --pipeline=from-distribution --distribution-version=5.0.0-alpha2

For help, type esrally --help or see the user documentation at https://esrally.readthedocs.io

So the command that I was running that got the error is:

But the command that I SHOULD be running (by looking at the configure output) is:

And when I run THAT command it gets much much further. I get this error:

Ah, so I can't be root, so I switch back to ec2-user and re-run the command (I run esrally configure for ec2-user and the rally.ini file looks the same as roots)

Now I get this error when running:

2016-11-04 11:12:04,711 rally.launcher INFO ES launch: ['bin/elasticsearch', '-Enode.name=rally-node0', '-Epath.logs=/home/ec2-user/.rally/benchmarks/races/2016-11-04-11-11-51/local/logs/geonames/append-no-conflicts/server']
2016-11-04 11:12:04,961 rally.launcher INFO rally-node0: starts elasticsearch
2016-11-04 11:12:04,961 rally.launcher INFO rally-node0:
2016-11-04 11:12:04,999 rally.launcher INFO rally-node0: Option             Description
2016-11-04 11:12:04,999 rally.launcher INFO rally-node0: ------             -----------
2016-11-04 11:12:04,999 rally.launcher INFO rally-node0: -E <KeyValuePair>  Configure an Elasticsearch setting
2016-11-04 11:12:04,999 rally.launcher INFO rally-node0: -V, --version      Prints elasticsearch version
2016-11-04 11:12:04,999 rally.launcher INFO rally-node0:                      information and exits
2016-11-04 11:12:05,0 rally.launcher INFO rally-node0: -d, --daemonize    Starts Elasticsearch in the background
2016-11-04 11:12:05,0 rally.launcher INFO rally-node0: -h, --help         show help
2016-11-04 11:12:05,0 rally.launcher INFO rally-node0: -p, --pidfile      Creates a pid file in the specified
2016-11-04 11:12:05,0 rally.launcher INFO rally-node0:                      path on start
2016-11-04 11:12:05,0 rally.launcher INFO rally-node0: -s, --silent       show minimal output
2016-11-04 11:12:05,0 rally.launcher INFO rally-node0: -v, --verbose      show verbose output
2016-11-04 11:12:05,0 rally.launcher INFO rally-node0: ERROR: Elasticsearch settings must be prefixed with [es.] but was [node.name]
2016-11-04 11:12:24,714 rally.launcher ERROR Could not start node 'rally-node0' within timeout period of 20.0 seconds.

I feel I'm so close. Any help will be appreciated.

4

Hi @Alexander_Gray_II,

Based on the output you've pasted here I guess you are on Rally 0.3.3 (current is 0.4.3); you can check this with esrally --version.

Can you please upgrade it by running pip3 install --upgrade esrally? You don't have to reconfigure Rally. Just upgrade the package and with pip and you should be fine.

Support for the Elasticsearch alpha releases has been removed in more recent versions of Rally, so you should now run:

esrally --pipeline=from-distribution --distribution-version=5.0.0

Daniel

5

I've now created:

due to your feedback. The next release (0.4.4) is right around the corner so I scheduled them for 0.4.5.

Daniel

6

Oh wow, thanks Daniel! You're really on top of this!
I verified that I do indeed have esrally:

$ esrally --version
esrally 0.4.3

I'm installing it via ansible too, and I tear down the ec2 instance after each run.
This did the trick:

esrally --pipeline=from-distribution --distribution-version=5.0.0

Thank you so much (I still got an error OOM, but that is my own fault, since I'm running this on a tiny t2.small instance, which doesn't have enough memory)

Sincerely,
@Alexander_Gray_II

7

Hi @Alexander_Gray_II,

I'm glad to hear that it works now! I also fixed both issues that I've mentioned above in the meantime so they'll be included in Rally 0.4.4. that I'll release next week.

Daniel

8

Hi @danielmitterdorfer,

I'm getting the same exact issue, with the latest esrally version (0.4.4) running on an AWS instance. I have also arrived at the OOM conclusion (t2.micro causes), but this is with 0.4.4 and the issue fix for following multiple symlinks.

9

Hi @khou,

0.4.4. indeed fixed the multi-symlink issue so I wonder about the cause. Let me grab an AWS Linux EC2 instance later and figure out what's going on.

Daniel

10

Hi @khou,

I could reproduce the problem now. I don't know (yet) why it cannot determine the JDK directory but I will eventually release a fix.

However, note that this is a convenience feature and I do not expect to autodetect the JDK directory on every platform. It works perfectly fine if you enter the path to the JDK 8 directory.

Daniel

1 Like
11

@danielmitterdorfer

Understand! Thank you very much for the swift replies! I will manually enter the path for now :)!

12

Hi @khou,

great! Thanks for your report though. It's fixed on master now and will be released with Rally 0.4.5. The problem was that the JDK detection code thought it was running on Debian.

Daniel

1 Like