Unable to start Elasticsearch - java.lang.IllegalStateException: Duplicate key org.elasticsearch.plugins.PluginsService$Bundle@e3623fe

subhas · March 5, 2019, 8:57am

After server reboot, elasticsearch fails to restart. I use only 2 plugins namely ingest-geoip and ingest-user-agent. Same issue is coming when i try to uninstall and install them again. Below is the error log -

[2019-03-05T08:46:07,168][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [gMG0tHu] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Duplicate key org.elasticsearch.plugins.PluginsService$Bundle@e3623fe
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.6.1.jar:6.6.1]...

Please help! This is centos server with javac 1.8.0_181. Thanks!

dadoonet · March 5, 2019, 9:15am

Could you ls the plugins dir?
Also what are the other logs?

subhas · March 5, 2019, 9:31am

At present, there is no plugin installed.

[root@gaboli7 gaboli]# cd /usr/share/elasticsearch/
[root@gaboli7 elasticsearch]# bin/elasticsearch-plugin list
[root@gaboli7 elasticsearch]# cd plugins/
[root@gaboli7 plugins]# ls
[root@gaboli7 plugins]#

Here is the error that I am getting when I try to install any plugin.

[root@gaboli7 elasticsearch]# bin/elasticsearch-plugin install ingest-user-agent
-> Downloading ingest-user-agent from elastic
[=================================================] 100%  
Exception in thread "main" java.lang.IllegalStateException: Duplicate key org.elasticsearch.plugins.PluginsService$Bundle@e3623fe
        at java.util.stream.Collectors.lambda$throwingMerger$0(Collectors.java:133)
        at java.util.HashMap.merge(HashMap.java:1254)
        at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1320)
        at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
        at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1553)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
        at org.elasticsearch.plugins.PluginsService.sortBundles(PluginsService.java:414)
        at org.elasticsearch.plugins.InstallPluginCommand.jarHellCheck(InstallPluginCommand.java:759)
        at org.elasticsearch.plugins.InstallPluginCommand.loadPluginInfo(InstallPluginCommand.java:728)
        at org.elasticsearch.plugins.InstallPluginCommand.installPlugin(InstallPluginCommand.java:793)
        at org.elasticsearch.plugins.InstallPluginCommand.install(InstallPluginCommand.java:776)
        at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:231)
        at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:216)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
        at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:77)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
        at org.elasticsearch.cli.Command.main(Command.java:90)
        at org.elasticsearch.plugins.PluginCli.main(PluginCli.java:47)
[root@gaboli7 elasticsearch]#

dadoonet · March 5, 2019, 9:48am

Could you ls -a the plugins dir?
May be remove the plugins dir and recreate it?

subhas · March 5, 2019, 9:52am

Thanks for the quick response. I get the following issue.

[root@gaboli7 elasticsearch]# cd /usr/share/elasticsearch/plugins/
[root@gaboli7 plugins]# ls -a
.  ..
[root@gaboli7 plugins]# cd ../
[root@gaboli7 elasticsearch]# rm -rf plugins/
[root@gaboli7 elasticsearch]# bin/elasticsearch-plugin install ingest-user-agent
-> Downloading ingest-user-agent from elastic
[=================================================] 100%  
Exception in thread "main" java.nio.file.NoSuchFileException: /usr/share/elasticsearch/plugins/.installing-9200493113704111332
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
        at java.nio.file.Files.createDirectory(Files.java:674)
        at java.nio.file.TempFileHelper.create(TempFileHelper.java:136)
        at java.nio.file.TempFileHelper.createTempDirectory(TempFileHelper.java:173)
        at java.nio.file.Files.createTempDirectory(Files.java:950)
        at org.elasticsearch.plugins.InstallPluginCommand.stagingDirectory(InstallPluginCommand.java:669)
        at org.elasticsearch.plugins.InstallPluginCommand.unzip(InstallPluginCommand.java:621)
        at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:230)
        at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:216)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
        at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:77)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
        at org.elasticsearch.cli.Command.main(Command.java:90)
        at org.elasticsearch.plugins.PluginCli.main(PluginCli.java:47)
[root@gaboli7 elasticsearch]#

Afte I recreate the plugins directory, the previous issue is back.

dadoonet · March 5, 2019, 5:24pm

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

There's a live preview panel for exactly this reasons.

Some thoughts:

You should not be connected as root but as elasticsearch user.
The documentation says to run:

sudo bin/elasticsearch-plugin install ingest-user-agent

May be you did not follow from the start how to properly install elasticsearch? Which package did you install?

subhas · March 6, 2019, 6:13am

Thanks for the tips. I have formatted the code in the previous replies.

It is not only in installing plugins. The problem is there when i try to start elasticsearch.

service elasticsearch restart

 vi /var/log/elasticsearch/elasticsearch.log

[2019-03-06T06:11:53,916][ERROR][o.e.b.Bootstrap          ] [gMG0tHu] Exception
java.lang.IllegalStateException: Duplicate key org.elasticsearch.plugins.PluginsService$Bundle@e3623fe
        at java.util.stream.Collectors.lambda$throwingMerger$0(Collectors.java:133) ~[?:1.8.0_181]

dadoonet · March 6, 2019, 6:33am

What is your elasticsearch.yml file?

subhas · March 6, 2019, 6:39am

my elasticsearch.yml file -

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0

dadoonet · March 6, 2019, 7:06am

Can you remove the plugins dir, recreate it and start elasticsearch again (without trying to install anything).

subhas · March 6, 2019, 7:12am

Thanks. I just did. Same error is coming - Duplicate key. What are the other options. Do i need to reinstall elasticsearch or downgrade to lower version?

dadoonet · March 6, 2019, 7:35am

If you can yeah reinstalling by following the installation guide would help.

How did you install it ?

subhas · March 6, 2019, 7:42am

I installed using rpm by following the below link.

https://www.elastic.co/guide/en/elasticsearch/reference/6.2/rpm.html

I think the problem is with this auto upgrade manner of elasticsearch. It was working fine before the server reboot. The reboot process has cause the elasticsearch to upgrade to latest version automatically, and hence the issue. How do we turn off auto upgrade?

By the way, I want to keep my existing data in the reistallation process. How to do this?

dadoonet · March 6, 2019, 2:31pm

Ah! Auto upgrades...

How do we turn off auto upgrade?

I don't know anything about that but IMHO that should be disabled on servers... (If possible)

Was your elasticsearch.yml that you shared previously the complete file?

I wonder about:

Can you remove the plugins dir, recreate it and start elasticsearch again (without trying to install anything).

You said that you did that and that the same error happened.
Just to make sure we are looking in the right direction, could you run:

sudo rm -r /usr/share/elasticsearch/plugins
sudo mkdir /usr/share/elasticsearch/plugins
sudo chown elasticsearch /usr/share/elasticsearch/plugins
# If using SysV init
sudo -i service elasticsearch start
# If using systemd
sudo systemctl start elasticsearch.service

And share the whole commands and their output and the whole elasticsearch logs here, please?

subhas · March 6, 2019, 6:07pm

Please see the same steps. I am getting the same output.

[root@gaboli7 gaboli]# sudo rm -r /usr/share/elasticsearch/plugins
[root@gaboli7 gaboli]# sudo mkdir /usr/share/elasticsearch/plugins
[root@gaboli7 gaboli]# sudo chown elasticsearch /usr/share/elasticsearch/plugins
[root@gaboli7 gaboli]# sudo -i service elasticsearch start
Starting elasticsearch (via systemctl):                    [  OK  ]

[root@gaboli7 gaboli]# service elasticsearch status
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vend                                                                                        or preset: disabled)
   Active: failed (Result: exit-code) since Wed 2019-03-06 17:59:12 UTC; 11s ago
     Docs: http://www.elastic.co
  Process: 9577 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DI                                                                                        R}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 9577 (code=exited, status=1/FAILURE)
Mar 06 17:59:06 gaboli7 systemd[1]: Started Elasticsearch.
Mar 06 17:59:12 gaboli7 systemd[1]: elasticsearch.service: main process exi...RE
Mar 06 17:59:12 gaboli7 systemd[1]: Unit elasticsearch.service entered fail...e.
Mar 06 17:59:12 gaboli7 systemd[1]: elasticsearch.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

On running vi /var/log/elasticsearch/elasticsearch.log I get the same error log.

[2019-03-06T06:11:53,937][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [gMG0tHu] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Duplicate key org.elasticsearch.plugins.PluginsService$Bundle@e3623fe
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.6.1.jar:6.6.1]

dadoonet · March 6, 2019, 9:46pm

Can you share the full logs please ?

subhas · March 6, 2019, 10:03pm

Here it is.

[2019-03-06T17:59:11,901][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [gMG0tHu] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Duplicate key org.elasticsearch.plugins.PluginsService$Bundle@e3623fe
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.6.1.jar:6.6.1]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.6.1.jar:6.6.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.6.1.jar:6.6.1]
Caused by: java.lang.IllegalStateException: Duplicate key org.elasticsearch.plugins.PluginsService$Bundle@e3623fe
        at java.util.stream.Collectors.lambda$throwingMerger$0(Collectors.java:133) ~[?:1.8.0_181]
        at java.util.HashMap.merge(HashMap.java:1254) ~[?:1.8.0_181]
        at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1320) ~[?:1.8.0_181]
        at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) ~[?:1.8.0_181]
        at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_181]
        at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:1.8.0_181]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_181]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_181]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_181]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_181]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_181]
        at org.elasticsearch.plugins.PluginsService.sortBundles(PluginsService.java:414) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:460) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:157) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.node.Node.<init>(Node.java:337) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.6.1.jar:6.6.1]
        ... 6 more

dadoonet · March 6, 2019, 11:45pm

Does the log really start with this line? Nothing before and after ?

subhas · March 7, 2019, 5:44am

This is the complete log that I got while restarting elasticsearch.

[2019-03-06T17:59:10,851][INFO ][o.e.e.NodeEnvironment    ] [gMG0tHu] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [37.6gb], net total_space [49.9gb], types [rootfs]
[2019-03-06T17:59:10,854][INFO ][o.e.e.NodeEnvironment    ] [gMG0tHu] heap size [1015.6mb], compressed ordinary object pointers [true]
[2019-03-06T17:59:11,819][INFO ][o.e.n.Node               ] [gMG0tHu] node name derived from node ID [gMG0tHunSL6sOqBZ_lWibw]; set [node.name] to override
[2019-03-06T17:59:11,820][INFO ][o.e.n.Node               ] [gMG0tHu] version[6.6.1], pid[9577], build[default/rpm/1fd8f69/2019-02-13T17:10:04.160291Z], OS[Linux/3.10.0-957.5.1.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_181/25.181-b13]
[2019-03-06T17:59:11,894][ERROR][o.e.b.Bootstrap          ] [gMG0tHu] Exception
java.lang.IllegalStateException: Duplicate key org.elasticsearch.plugins.PluginsService$Bundle@e3623fe
        at java.util.stream.Collectors.lambda$throwingMerger$0(Collectors.java:133) ~[?:1.8.0_181]
        at java.util.HashMap.merge(HashMap.java:1254) ~[?:1.8.0_181]
        at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1320) ~[?:1.8.0_181]
        at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) ~[?:1.8.0_181]
        at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_181]
        at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:1.8.0_181]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_181]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_181]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.6.1.jar:6.6.1]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.6.1.jar:6.6.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.6.1.jar:6.6.1]
[2019-03-06T17:59:11,901][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [gMG0tHu] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Duplicate key org.elasticsearch.plugins.PluginsService$Bundle@e3623fe
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.6.1.jar:6.6.1]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.6.1.jar:6.6.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.6.1.jar:6.6.1]
Caused by: java.lang.IllegalStateException: Duplicate key org.elasticsearch.plugins.PluginsService$Bundle@e3623fe
        at java.util.stream.Collectors.lambda$throwingMerger$0(Collectors.java:133) ~[?:1.8.0_181]
        at java.util.HashMap.merge(HashMap.java:1254) ~[?:1.8.0_181]
        at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1320) ~[?:1.8.0_181]
        at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) ~[?:1.8.0_181]
        at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_181]
        at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:1.8.0_181]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_181]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.6.1.jar:6.6.1]
        ... 6 more

dadoonet · March 8, 2019, 11:53am

Can you run:

ls -a /usr/share/elasticsearch/plugins

And:

cat /etc/elasticsearch/elasticsearch.yml