Elasticsearch has stopped and no longer works

4y555 · January 19, 2020, 10:52pm

Hi guys
I was using Elasticsearch (with the nextcloud), but the elastic just stopped after a while.
Try to install other versions or reinstall the same version, but it doesn't work.
Can you help me? Thank you very much!

CentOS Linux release 7.7.1908 (Core)
Php 7.3.13
Mysql 5.7.29
Nextcloud 17.0.2

elasticsearch-6.8.6.rpm
readonlyrest-1.18.10_es6.8.6.zip

[root@nextcloud ~]# sudo -u apache /usr/bin/php /var/www/html/occ fulltextsearch:check
Full text search 1.3.5

- Search Platform:
Elasticsearch 1.4.0
{
    "elastic_host": [
        "http://nextcloud:********@localhost:9200"
    ],
    "elastic_index": "nextcloud",
    "fields_limit": "10000",
    "es_ver_below66": "0",
    "analyzer_tokenizer": "standard"
}

- Content Providers:
Files 1.3.6
{
    "files_local": "1",
    "files_external": "2",
    "files_group_folders": "1",
    "files_encrypted": "0",
    "files_federated": "0",
    "files_size": "20",
    "files_pdf": "1",
    "files_office": "1",
    "files_image": "0",
    "files_audio": "0"
}
[root@nextcloud ~]#

FILE /etc/elasticsearch/readonlyrest.yml
readonlyrest:
access_controle_rules:
name: "Require HTTP Basic Auth"
type: allow
auth_key: nextcloud:nextcloud
indices: ["nextcloud"]

ADD /etc/elasticsearch/elasticsearch.yml
xpack.security.enabled: false

Elasticsearch service FAILED after 10 seconds.

[root@nextcloud ~]# systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Qui 2020-01-16 09:26:24 -03; 27s ago
     Docs: http://www.elastic.co
  Process: 1305 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 1305 (code=exited, status=1/FAILURE)

Jan 16 09:28:01 nextcloud.local systemd[1]: Started Elasticsearch.
Jan 16 09:26:24 nextcloud.local systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Jan 16 09:26:24 nextcloud.local systemd[1]: Unit elasticsearch.service entered failed state.
Jan 16 09:26:24 nextcloud.local systemd[1]: elasticsearch.service failed.
[root@nextcloud ~]#

I've tested several versions, updated everything, searched several blogs, but I can't solve the problem. It is really frustrating.

Can any kindness help me?

Christian_Dahlqvist · January 20, 2020, 6:17am

Please share what is in the Elasticsearch logs.

4y555 · January 20, 2020, 9:27am

[2020-01-20T06:24:52,153][INFO ][o.e.e.NodeEnvironment    ] [FItIT-1] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [440gb], net total_space [469.7gb], types [rootfs]
[2020-01-20T06:24:52,159][INFO ][o.e.e.NodeEnvironment    ] [FItIT-1] heap size [990.7mb], compressed ordinary object pointers [true]
[2020-01-20T06:24:52,162][INFO ][o.e.n.Node               ] [FItIT-1] node name derived from node ID [FItIT-1JRt-j8Bcnv5kRwQ]; set [node.name] to override
[2020-01-20T06:24:52,162][INFO ][o.e.n.Node               ] [FItIT-1] version[6.8.6], pid[23041], build[default/rpm/3d9f765/2019-12-13T17:11:52.013738Z], OS[Linux/3.10.0-1062.9.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_232/25.232-b09]
[2020-01-20T06:24:52,162][INFO ][o.e.n.Node               ] [FItIT-1] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-2452565900209370013, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm]
[2020-01-20T06:24:58,641][INFO ][t.b.r.c.RorSsl$          ] [FItIT-1] Cannot find SSL configuration is elasticsearch.yml, trying: /etc/elasticsearch/readonlyrest.yml
[2020-01-20T06:24:58,809][ERROR][o.e.b.Bootstrap          ] [FItIT-1] Exception
java.lang.IllegalStateException: failed to load plugin class [tech.beshu.ror.es.ReadonlyRestPlugin]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:614) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.node.Node.<init>(Node.java:339) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.8.6.jar:6.8.6]
	at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.8.6.jar:6.8.6]
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_232]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-6.8.6.jar:6.8.6]
	... 15 more
Caused by: org.elasticsearch.ElasticsearchException: Cannot parse file /etc/elasticsearch/readonlyrest.yml content. Cause: mapping values are not allowed here
 in 'reader', line 6, column 16:
           auth_key: nextcloud:nextcloud
                   ^

	at tech.beshu.ror.es.ReadonlyRestPlugin.lambda$null$1(ReadonlyRestPlugin.java:119) ~[?:?]
	at tech.beshu.ror.utils.ScalaJavaHelper$.$anonfun$getOrElse$1(ScalaJavaHelper.scala:35) ~[?:?]
	at scala.util.Either.fold(Either.scala:192) ~[?:?]
	at tech.beshu.ror.utils.ScalaJavaHelper$.getOrElse(ScalaJavaHelper.scala:36) ~[?:?]
	at tech.beshu.ror.es.ReadonlyRestPlugin.lambda$new$2(ReadonlyRestPlugin.java:119) ~[?:?]
	at monix.eval.Task$Map.apply(Task.scala:4514) ~[?:?]
	at monix.eval.Task$Map.apply(Task.scala:4510) ~[?:?]
	at monix.eval.internal.TaskRunSyncUnsafe$.apply(TaskRunSyncUnsafe.scala:107) ~[?:?]
	at monix.eval.Task.runSyncUnsafeOpt(Task.scala:1096) ~[?:?]
	at monix.eval.Task.runSyncUnsafe(Task.scala:1063) ~[?:?]
	at tech.beshu.ror.es.ReadonlyRestPlugin.<init>(ReadonlyRestPlugin.java:120) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_232]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-6.8.6.jar:6.8.6]
	... 15 more
[2020-01-20T06:24:58,826][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [FItIT-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [tech.beshu.ror.es.ReadonlyRestPlugin]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.8.6.jar:6.8.6]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.6.jar:6.8.6]
Caused by: java.lang.IllegalStateException: failed to load plugin class [tech.beshu.ror.es.ReadonlyRestPlugin]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:614) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.node.Node.<init>(Node.java:339) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.6.jar:6.8.6]
	... 6 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_232]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.node.Node.<init>(Node.java:339) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.6.jar:6.8.6]
	... 6 more
Caused by: org.elasticsearch.ElasticsearchException: Cannot parse file /etc/elasticsearch/readonlyrest.yml content. Cause: mapping values are not allowed here
 in 'reader', line 6, column 16:
           auth_key: nextcloud:nextcloud
                   ^

	at tech.beshu.ror.es.ReadonlyRestPlugin.lambda$null$1(ReadonlyRestPlugin.java:119) ~[?:?]
	at tech.beshu.ror.utils.ScalaJavaHelper$.$anonfun$getOrElse$1(ScalaJavaHelper.scala:35) ~[?:?]
	at scala.util.Either.fold(Either.scala:192) ~[?:?]
	at tech.beshu.ror.utils.ScalaJavaHelper$.getOrElse(ScalaJavaHelper.scala:36) ~[?:?]
	at tech.beshu.ror.es.ReadonlyRestPlugin.lambda$new$2(ReadonlyRestPlugin.java:119) ~[?:?]
	at monix.eval.Task$Map.apply(Task.scala:4514) ~[?:?]
	at monix.eval.Task$Map.apply(Task.scala:4510) ~[?:?]
	at monix.eval.internal.TaskRunSyncUnsafe$.apply(TaskRunSyncUnsafe.scala:107) ~[?:?]
	at monix.eval.Task.runSyncUnsafeOpt(Task.scala:1096) ~[?:?]
	at monix.eval.Task.runSyncUnsafe(Task.scala:1063) ~[?:?]
	at tech.beshu.ror.es.ReadonlyRestPlugin.<init>(ReadonlyRestPlugin.java:120) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_232]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.node.Node.<init>(Node.java:339) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.6.jar:6.8.6]
	... 6 more

dadoonet · January 20, 2020, 9:41am

I think the error message is pretty clear, no?

Cannot parse file /etc/elasticsearch/readonlyrest.yml content. Cause: mapping values are not allowed here
in 'reader', line 6, column 16:
          auth_key: nextcloud:nextcloud
                  ^

You are using here a third party plugin ReadonlyRestPlugin which is not supported. You should either use the official package and its security features or ask the authors of the plugin you are using for help.

My 2 cents.

May be you should try though:

auth_key: "nextcloud:nextcloud"

4y555 · January 20, 2020, 10:51am

Hi David, how are you? Hope so.

I published the logs and then realized where the error was: there were two "spaces" in readonlyrest.yml, just where you pointed out.

Now it's running.

Can you help me with one more thing?

After indexing everything, it presented error 82/82 files.
How can I list all of these files?

dadoonet · January 20, 2020, 11:10am

I have no idea of what it means...

system · February 17, 2020, 11:10am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.