Run two versions of ES on machine

[mrodent (Mike)]

This is on a W10 box. ES (7.10.2) is currently running on localhost:9200. I need to upgrade. When I attempted to upgrade to 7.16.3 some time ago a regression occurred, reported by me and acknowledged by Elasticsearch HQ, which hopefully will have been corrected in the current version (8.6.2). But I absolutely need to do some testing to ensure that my stuff won't break, hence two versions, one on another localhost port.

I have just downloaded the latest stable release, 8.6.2 and unzipped the .zip.

I want 8.6.2 to run as a W10 service, for example at localhost:9500.

I don't need any warnings about "the danger" of doing this. All this is on a single PC, and when I'm happy that 8.6.2 doesn't break any tests I'll switch to that.

[mrodent (Mike)]

I found this recent answer on the same subject (*nix OS).

So (having unzipped 8.6.2) I changed all those settings in elasticsearch.yml.

I ran elasticsearch.bat.

Error:

[2023-03-04T15:59:45,029][ERROR][o.e.b.Elasticsearch      ] [M17A] fatal exception while booting Elasticsearch
java.lang.RuntimeException: org.elasticsearch.cli.UserException: will not overwrite keystore at [D:\apps\ElasticSearch\elasticsearch-7.10.2\config\elasticsearch.keystore], because this incurs changing the file owner
	at org.elasticsearch.bootstrap.Elasticsearch.initPhase2(Elasticsearch.java:150) ~[elasticsearch-8.6.2.jar:?]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:66) ~[elasticsearch-8.6.2.jar:?]
Caused by: org.elasticsearch.cli.UserException: will not overwrite keystore at [D:\apps\ElasticSearch\elasticsearch-7.10.2\config\elasticsearch.keystore], because this incurs changing the file owner
	at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:480) ~[elasticsearch-8.6.2.jar:?]
	at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:412) ~[elasticsearch-8.6.2.jar:?]
	at org.elasticsearch.common.settings.KeyStoreWrapper.upgrade(KeyStoreWrapper.java:291) ~[elasticsearch-8.6.2.jar:?]
	at org.elasticsearch.common.settings.KeyStoreWrapper.bootstrap(KeyStoreWrapper.java:221) ~[elasticsearch-8.6.2.jar:?]
	at org.elasticsearch.bootstrap.Elasticsearch.initPhase2(Elasticsearch.java:148) ~[elasticsearch-8.6.2.jar:?]
	... 1 more

... right: I don't want it to "overwrite keystore at ... \elasticsearch-7.10.2 ...". Why is it using a "key store" from another ES instance? How can I persuade it to make a separate "key store"?

[Christian_Dahlqvist]

You can not have two instances share data directories, if that is what you are trying to do. If you want to run two nodes on a single host they need to be completely separate.

It might help if you share the configuration files of the two nodes.

[mrodent (Mike)]

Thanks. I confirm that my aim is to have two entirely separate instances of ES running. I want no "sharing of data", indeed no sharing of anything whatsoever. Separate, distinct, apart, unrelated.

The file \config\elasticsearch.yml for 7.10.2 is completely commented-out.

The same file for 8.6.2 has the following uncommented lines:

cluster.name: my-862application
...
node.name: node862-1
...
path.data: D:/temp/es862_data
...
path.logs: D:/temp/es862_logs
...
http.port: 9500

(NB I also tried the above paths with Windows backslashes in the paths: same error).

... are there other "configuration files" which I need to modify for the 8.6.2 setup?

From the error stack trace above, the culprit line would appear to be KeyStoreWrapper.java, line 480: at this point, for some reason, the 8.6.2. jar file decides it wants to use a path with "7.10.2" in it.

So I checked the source code involved here:

// check that replace doesn't change the owner
if (Files.exists(keystoreFile, LinkOption.NOFOLLOW_LINKS)
	&& false == Files.getOwner(keystoreTempFile, LinkOption.NOFOLLOW_LINKS)
		.equals(Files.getOwner(keystoreFile, LinkOption.NOFOLLOW_LINKS))) {
	String message = String.format(
		Locale.ROOT,
		"will not overwrite keystore at [%s], because this incurs changing the file owner",
		keystoreFile
	);
	throw new UserException(ExitCodes.CONFIG, message);
}

... somehow the variable "keystoreFile" seems to have been set "wrongly".

To me this suggests (possibly) that in the course of "registering" ES something has maybe previously happened like maybe a key has been configured in the W10 registry (or something). But I haven't got sufficient knowledge of this level of gory details.

I did a search in the registry on "7.10.2". The most likely key was this:

Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Elastic\Elasticsearch\7.10.2: several keys, including "CONFIGDIRECTORY: D:\apps\Elasticsearch\elasticsearch-7.10.2\config"

So I created this key:
Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Elastic\Elasticsearch\8.6.2: "CONFIGDIRECTORY: D:\apps\Elasticsearch\elasticsearch-8.6.2\config"

... same error. Needs an expert.

[mrodent (Mike)]

OK, some progress but still not there yet.

After consulting the docs (as usual, not really directed at Windoze users), I did this:

>set ES_PATH_CONF=D:/apps/ElasticSearch/elasticsearch-8.6.2/config
>elasticsearch.bat

... a torrent of output ensued, but it looked like configuring was happening.

When I attempted to browse to localhost:9500, however, I got this output in the console:

[2023-03-05T09:01:48,356][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [node862-1] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9500, remoteAddress=/127.0.0.1:9690}
[2023-03-05T09:01:48,356][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [node862-1] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9500, remoteAddress=/127.0.0.1:9691}
[2023-03-05T09:01:48,356][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [node862-1] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9500, remoteAddress=/127.0.0.1:9692}
[2023-03-05T09:01:48,356][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [node862-1] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9500, remoteAddress=/127.0.0.1:9693}
[2023-03-05T09:01:48,356][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [node862-1] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9500, remoteAddress=/127.0.0.1:9694}
[2023-03-05T09:01:48,371][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [node862-1] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9500, remoteAddress=/127.0.0.1:9695}
[2023-03-05T09:01:48,371][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [node862-1] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9500, remoteAddress=/127.0.0.1:9696}
[2023-03-05T09:01:48,371][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [node862-1] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9500, remoteAddress=/127.0.0.1:9697}
[2023-03-05T09:01:48,371][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [node862-1] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9500, remoteAddress=/127.0.0.1:9698}
[2023-03-05T09:01:48,371][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [node862-1] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9500, remoteAddress=/127.0.0.1:9699}

... and in the browser page I get a page with an error message: "Connection was reset". No "you know, for search" or other connection details output...

However, I do see that one of the last log messages in the console screen before I tried to get to "localhost:9500" was this:

[2023-03-05T09:01:30,685][INFO ][o.e.c.r.a.AllocationService] [node862-1] current.health="GREEN" message="Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.geoip_databases][0], [.security-7][0]]])." previous.health="RED" reason="shards started [[.geoip_databases][0], [.security-7][0]]"

which looks hopeful. Seem to be close.

... any diagnosis, suggestions??

later: This is the right answer! ... but then (if using a non-standard port at least) there are security issues which follow. See solution here.

[mrodent (Mike)]

It now seems I am getting into an area which has baffled others:

When I put "https://localhost:9500" in the browser I am asked for a "username" and "password".

It appears that ES 8 more or less requires you to have strong security from the start. Unfortunately the docs here appear to be far from ideal about what to do next. Especially if

• you're on Windoze
• you don't know what Docker is, how to install it or use, or what its purpose might be
• you don't know what Kibana is, how to install it or use it, or what its purpose might be
• you're not interested in installing Docker or Kibana
• you're told to set up xpack security, but in fact that all appears to be configured correctly (i.e. to use it) in the yml file.
• the supposed default pairing username and password "elastic" and "changeme" doesn't get you in
• there is no listing in the logs showing an initial username and password as suggested might be the case, for example, here.

... however it appears (hopefully) that I have achieved the objective of this question. "http://localhost:7200" still produces the expected output, and my app using 7.10.2 appears to be unaffected by the parallel ES 8.6.2 instance.