Unable to start Elasticsearch - java.lang.IllegalStateException: Duplicate key org.elasticsearch.plugins.PluginsService$Bundle@e3623fe

[subhas (Subhas Takhellambam)]

After server reboot, elasticsearch fails to restart. I use only 2 plugins namely ingest-geoip and ingest-user-agent. Same issue is coming when i try to uninstall and install them again. Below is the error log -

[2019-03-05T08:46:07,168][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [gMG0tHu] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Duplicate key org.elasticsearch.plugins.PluginsService$Bundle@e3623fe
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.6.1.jar:6.6.1]...

Please help! This is centos server with javac 1.8.0_181. Thanks!

[dadoonet (David Pilato)]

Could you ls the plugins dir?
Also what are the other logs?

[subhas (Subhas Takhellambam)]

At present, there is no plugin installed.

[root@gaboli7 gaboli]# cd /usr/share/elasticsearch/
[root@gaboli7 elasticsearch]# bin/elasticsearch-plugin list
[root@gaboli7 elasticsearch]# cd plugins/
[root@gaboli7 plugins]# ls
[root@gaboli7 plugins]#

Here is the error that I am getting when I try to install any plugin.

[root@gaboli7 elasticsearch]# bin/elasticsearch-plugin install ingest-user-agent
-> Downloading ingest-user-agent from elastic
[=================================================] 100%  
Exception in thread "main" java.lang.IllegalStateException: Duplicate key org.elasticsearch.plugins.PluginsService$Bundle@e3623fe
        at java.util.stream.Collectors.lambda$throwingMerger$0(Collectors.java:133)
        at java.util.HashMap.merge(HashMap.java:1254)
        at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1320)
        at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
        at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1553)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
        at org.elasticsearch.plugins.PluginsService.sortBundles(PluginsService.java:414)
        at org.elasticsearch.plugins.InstallPluginCommand.jarHellCheck(InstallPluginCommand.java:759)
        at org.elasticsearch.plugins.InstallPluginCommand.loadPluginInfo(InstallPluginCommand.java:728)
        at org.elasticsearch.plugins.InstallPluginCommand.installPlugin(InstallPluginCommand.java:793)
        at org.elasticsearch.plugins.InstallPluginCommand.install(InstallPluginCommand.java:776)
        at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:231)
        at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:216)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
        at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:77)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
        at org.elasticsearch.cli.Command.main(Command.java:90)
        at org.elasticsearch.plugins.PluginCli.main(PluginCli.java:47)
[root@gaboli7 elasticsearch]#

[dadoonet (David Pilato)]

Could you ls -a the plugins dir?
May be remove the plugins dir and recreate it?

[subhas (Subhas Takhellambam)]

Thanks for the quick response. I get the following issue.

[root@gaboli7 elasticsearch]# cd /usr/share/elasticsearch/plugins/
[root@gaboli7 plugins]# ls -a
.  ..
[root@gaboli7 plugins]# cd ../
[root@gaboli7 elasticsearch]# rm -rf plugins/
[root@gaboli7 elasticsearch]# bin/elasticsearch-plugin install ingest-user-agent
-> Downloading ingest-user-agent from elastic
[=================================================] 100%  
Exception in thread "main" java.nio.file.NoSuchFileException: /usr/share/elasticsearch/plugins/.installing-9200493113704111332
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
        at java.nio.file.Files.createDirectory(Files.java:674)
        at java.nio.file.TempFileHelper.create(TempFileHelper.java:136)
        at java.nio.file.TempFileHelper.createTempDirectory(TempFileHelper.java:173)
        at java.nio.file.Files.createTempDirectory(Files.java:950)
        at org.elasticsearch.plugins.InstallPluginCommand.stagingDirectory(InstallPluginCommand.java:669)
        at org.elasticsearch.plugins.InstallPluginCommand.unzip(InstallPluginCommand.java:621)
        at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:230)
        at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:216)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
        at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:77)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
        at org.elasticsearch.cli.Command.main(Command.java:90)
        at org.elasticsearch.plugins.PluginCli.main(PluginCli.java:47)
[root@gaboli7 elasticsearch]#

Afte I recreate the plugins directory, the previous issue is back.

[dadoonet (David Pilato)]

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

There's a live preview panel for exactly this reasons.

Some thoughts:

• You should not be connected as root but as elasticsearch user.
• The documentation says to run:

sudo bin/elasticsearch-plugin install ingest-user-agent

May be you did not follow from the start how to properly install elasticsearch? Which package did you install?

[subhas (Subhas Takhellambam)]

Thanks for the tips. I have formatted the code in the previous replies.

It is not only in installing plugins. The problem is there when i try to start elasticsearch.

service elasticsearch restart

 vi /var/log/elasticsearch/elasticsearch.log

[2019-03-06T06:11:53,916][ERROR][o.e.b.Bootstrap          ] [gMG0tHu] Exception
java.lang.IllegalStateException: Duplicate key org.elasticsearch.plugins.PluginsService$Bundle@e3623fe
        at java.util.stream.Collectors.lambda$throwingMerger$0(Collectors.java:133) ~[?:1.8.0_181]

[dadoonet (David Pilato)]

What is your elasticsearch.yml file?

[subhas (Subhas Takhellambam)]

my elasticsearch.yml file -

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0

[dadoonet (David Pilato)]

Can you remove the plugins dir, recreate it and start elasticsearch again (without trying to install anything).

[subhas (Subhas Takhellambam)]

Thanks. I just did. Same error is coming - Duplicate key. What are the other options. Do i need to reinstall elasticsearch or downgrade to lower version?

[dadoonet (David Pilato)]

If you can yeah reinstalling by following the installation guide would help.

How did you install it ?

[subhas (Subhas Takhellambam)]

I installed using rpm by following the below link.

https://www.elastic.co/guide/en/elasticsearch/reference/6.2/rpm.html

I think the problem is with this auto upgrade manner of elasticsearch. It was working fine before the server reboot. The reboot process has cause the elasticsearch to upgrade to latest version automatically, and hence the issue. How do we turn off auto upgrade?

By the way, I want to keep my existing data in the reistallation process. How to do this?

[dadoonet (David Pilato)]

Ah! Auto upgrades...

How do we turn off auto upgrade?

I don't know anything about that but IMHO that should be disabled on servers... (If possible)

Was your elasticsearch.yml that you shared previously the complete file?

I wonder about:

Can you remove the plugins dir, recreate it and start elasticsearch again (without trying to install anything).

You said that you did that and that the same error happened.
Just to make sure we are looking in the right direction, could you run:

sudo rm -r /usr/share/elasticsearch/plugins
sudo mkdir /usr/share/elasticsearch/plugins
sudo chown elasticsearch /usr/share/elasticsearch/plugins
# If using SysV init
sudo -i service elasticsearch start
# If using systemd
sudo systemctl start elasticsearch.service

And share the whole commands and their output and the whole elasticsearch logs here, please?

[subhas (Subhas Takhellambam)]

Please see the same steps. I am getting the same output.

[root@gaboli7 gaboli]# sudo rm -r /usr/share/elasticsearch/plugins
[root@gaboli7 gaboli]# sudo mkdir /usr/share/elasticsearch/plugins
[root@gaboli7 gaboli]# sudo chown elasticsearch /usr/share/elasticsearch/plugins
[root@gaboli7 gaboli]# sudo -i service elasticsearch start
Starting elasticsearch (via systemctl):                    [  OK  ]

[root@gaboli7 gaboli]# service elasticsearch status
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vend                                                                                        or preset: disabled)
   Active: failed (Result: exit-code) since Wed 2019-03-06 17:59:12 UTC; 11s ago
     Docs: http://www.elastic.co
  Process: 9577 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DI                                                                                        R}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 9577 (code=exited, status=1/FAILURE)
Mar 06 17:59:06 gaboli7 systemd[1]: Started Elasticsearch.
Mar 06 17:59:12 gaboli7 systemd[1]: elasticsearch.service: main process exi...RE
Mar 06 17:59:12 gaboli7 systemd[1]: Unit elasticsearch.service entered fail...e.
Mar 06 17:59:12 gaboli7 systemd[1]: elasticsearch.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

On running vi /var/log/elasticsearch/elasticsearch.log I get the same error log.

[2019-03-06T06:11:53,937][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [gMG0tHu] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Duplicate key org.elasticsearch.plugins.PluginsService$Bundle@e3623fe
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.6.1.jar:6.6.1]

[dadoonet (David Pilato)]

Can you share the full logs please ?

[subhas (Subhas Takhellambam)]

Here it is.

[2019-03-06T17:59:11,901][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [gMG0tHu] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Duplicate key org.elasticsearch.plugins.PluginsService$Bundle@e3623fe
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.6.1.jar:6.6.1]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.6.1.jar:6.6.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.6.1.jar:6.6.1]
Caused by: java.lang.IllegalStateException: Duplicate key org.elasticsearch.plugins.PluginsService$Bundle@e3623fe
        at java.util.stream.Collectors.lambda$throwingMerger$0(Collectors.java:133) ~[?:1.8.0_181]
        at java.util.HashMap.merge(HashMap.java:1254) ~[?:1.8.0_181]
        at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1320) ~[?:1.8.0_181]
        at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) ~[?:1.8.0_181]
        at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_181]
        at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:1.8.0_181]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_181]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_181]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_181]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_181]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_181]
        at org.elasticsearch.plugins.PluginsService.sortBundles(PluginsService.java:414) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:460) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:157) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.node.Node.<init>(Node.java:337) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.6.1.jar:6.6.1]
        ... 6 more

[dadoonet (David Pilato)]

Does the log really start with this line? Nothing before and after ?

[subhas (Subhas Takhellambam)]

This is the complete log that I got while restarting elasticsearch.

[2019-03-06T17:59:10,851][INFO ][o.e.e.NodeEnvironment    ] [gMG0tHu] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [37.6gb], net total_space [49.9gb], types [rootfs]
[2019-03-06T17:59:10,854][INFO ][o.e.e.NodeEnvironment    ] [gMG0tHu] heap size [1015.6mb], compressed ordinary object pointers [true]
[2019-03-06T17:59:11,819][INFO ][o.e.n.Node               ] [gMG0tHu] node name derived from node ID [gMG0tHunSL6sOqBZ_lWibw]; set [node.name] to override
[2019-03-06T17:59:11,820][INFO ][o.e.n.Node               ] [gMG0tHu] version[6.6.1], pid[9577], build[default/rpm/1fd8f69/2019-02-13T17:10:04.160291Z], OS[Linux/3.10.0-957.5.1.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_181/25.181-b13]
[2019-03-06T17:59:11,894][ERROR][o.e.b.Bootstrap          ] [gMG0tHu] Exception
java.lang.IllegalStateException: Duplicate key org.elasticsearch.plugins.PluginsService$Bundle@e3623fe
        at java.util.stream.Collectors.lambda$throwingMerger$0(Collectors.java:133) ~[?:1.8.0_181]
        at java.util.HashMap.merge(HashMap.java:1254) ~[?:1.8.0_181]
        at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1320) ~[?:1.8.0_181]
        at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) ~[?:1.8.0_181]
        at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_181]
        at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:1.8.0_181]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_181]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_181]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.6.1.jar:6.6.1]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.6.1.jar:6.6.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.6.1.jar:6.6.1]
[2019-03-06T17:59:11,901][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [gMG0tHu] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Duplicate key org.elasticsearch.plugins.PluginsService$Bundle@e3623fe
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.6.1.jar:6.6.1]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.6.1.jar:6.6.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.6.1.jar:6.6.1]
Caused by: java.lang.IllegalStateException: Duplicate key org.elasticsearch.plugins.PluginsService$Bundle@e3623fe
        at java.util.stream.Collectors.lambda$throwingMerger$0(Collectors.java:133) ~[?:1.8.0_181]
        at java.util.HashMap.merge(HashMap.java:1254) ~[?:1.8.0_181]
        at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1320) ~[?:1.8.0_181]
        at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) ~[?:1.8.0_181]
        at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_181]
        at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:1.8.0_181]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_181]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.6.1.jar:6.6.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.6.1.jar:6.6.1]
        ... 6 more

[dadoonet (David Pilato)]

Can you run:

ls -a /usr/share/elasticsearch/plugins

And:

cat /etc/elasticsearch/elasticsearch.yml