Can't understand the problem from the log

[Mikhail_Khludnev]

I'm trying to run

$ esrally --track-path=track.json --preserve-install true --car="4gheap" --client-options="timeout:6000" --distribution-version=7.0.0-rc2

Here's what I have in log https://paste.apache.org/0wRo
Something happen, but I can't grasp what.
Elastic log is absent. Then when I'me trying to launch elastic I've got a culprit.

:~/.rally/benchmarks/races/2019-04-09-12-22-24/rally-node-0/install/elasticsearch-7.0.0-rc2/bin$ ./elasticsearch --verbose
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
ERROR: Temporary file directory [${ES_TMPDIR}] does not exist or is not accessible

Here's my java

:~/.rally/benchmarks/races/2019-04-09-12-22-24/rally-node-0/install/elasticsearch-7.0.0-rc2/bin$ $JAVA_HOME/bin/java --version
openjdk 12 2019-03-19
OpenJDK Runtime Environment AdoptOpenJDK (build 12+33)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 12+33, mixed mode, sharing)

Checking tmp dir write access.

:~/.rally/benchmarks/races/2019-04-09-12-22-24/rally-node-0/install/elasticsearch-7.0.0-rc2/bin$ cat > ${ES_TMPDIR}/o
sss

:~/.rally/benchmarks/races/2019-04-09-12-22-24/rally-node-0/install/elasticsearch-7.0.0-rc2/bin$ file ${ES_TMPDIR}/o
/tmp/o: ASCII text, with no line terminators

How I can understand what's going on from rally logs? Do you know how to deal with that ES_TMPDIR failure?

[dliappis (Dimitrios Liappis)]

Hi Mikhail,

Unfortunately you are hitting: https://github.com/elastic/elasticsearch/issues/40812

Rally picks up the configuration option from the rally-teams repo (7 branch) which contains the right content for -Djava.io.tmpdir (without double quotes) for the latest 7.0.0. The 7.0.0-rc2 artifact, that Rally downloads, requires double quotes for this setting.

Since you are having a local setup of Rally and Elasticsearch you could also switch to the from-sources-complete pipeline and pick one of the commits of the Elasticsearch 7.0 branch using --revision= until 7.0.0 is GA. This way Rally will build Elasticsearch locally (just make sure you have the right JAVAx_HOME env vars set see also here -- for Elasticsearch 7 you'll need JAVA12_HOME to build and if you want to use a different jvm to run Elasticsearch with, set JAVAx_HOME accordingly and specify the major version using --runtime-jdk).

Regards,
Dimitris

[Mikhail_Khludnev]

Ok. Thanks, @dliappis. It clarifies the issue. But how we can make it easier understandable from rally.log? Can we make it log stderr from elastic?

[dliappis (Dimitrios Liappis)]

This should be present in rally.log. We reproduced this using:

esrally --distribution-version=7.0.0-rc2 --test-mode --challenge=append-no-conflicts-index-only

and could see the following in rally.log

2019-04-10 10:03:51,772 ActorAddr-(T|:52550)/PID:41926 esrally.mechanic.launcher INFO Allowed JDK versions are [12, 11, 8].
2019-04-10 10:03:51,949 ActorAddr-(T|:52550)/PID:41926 esrally.mechanic.launcher INFO Detected JDK with major version [12] in [/Library/Java/JavaVirtualMachines/jdk-12.jdk/Contents/Home].
2019-04-10 10:03:51,949 ActorAddr-(T|:52550)/PID:41926 esrally.mechanic.launcher INFO Starting node [rally-node-0] based on car [defaults].
2019-04-10 10:03:52,3 ActorAddr-(T|:52550)/PID:41926 esrally.utils.process INFO b'openjdk version "12-ea" 2019-03-19\n'
2019-04-10 10:03:52,4 ActorAddr-(T|:52550)/PID:41926 esrally.utils.process INFO b'OpenJDK Runtime Environment (build 12-ea+24)\n'
2019-04-10 10:03:52,4 ActorAddr-(T|:52550)/PID:41926 esrally.utils.process INFO b'OpenJDK 64-Bit Server VM (build 12-ea+24, mixed mode, sharing)\n'
2019-04-10 10:03:52,8 ActorAddr-(T|:52550)/PID:41926 esrally.mechanic.launcher INFO Setting [-XX:+ExitOnOutOfMemoryError] to detect out of memory errors during the benchmark.
2019-04-10 10:03:52,409 ActorAddr-(T|:52550)/PID:41926 esrally.mechanic.launcher INFO rally-node-0 (stdout): OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
2019-04-10 10:03:53,488 ActorAddr-(T|:52550)/PID:41926 esrally.mechanic.launcher INFO rally-node-0 (stdout): ERROR: Temporary file directory [${ES_TMPDIR}] does not exist or is not accessible
2019-04-10 10:03:53,570 ActorAddr-(T|:52550)/PID:41926 esrally.mechanic.launcher INFO rally-node-0 (stdout): No more output. Process has likely terminated.
2019-04-10 10:03:53,675 ActorAddr-(T|:52550)/PID:41926 esrally.mechanic.launcher ERROR Node [rally-node-0] has terminated with exit code [78].
2019-04-10 10:03:53,697 ActorAddr-(T|:52524)/PID:41787 esrally.actor INFO Received a benchmark failure from [ActorAddr-(T|:52537)] and will forward it now.