Server disk broken and recover, but failed to start

Hello.

My server disk was broken and recover now.

But the elasticsearch service isn`t started.

Here is my logs.

[2020-07-22T15:54:22,153][INFO ][o.e.e.NodeEnvironment    ] [apm-data-node4] using [1] data paths, mounts [[/data/1 (/dev/mapper/vg_data-lv_data1)]], net usable_space [934.6gb], net total_space [1.7tb], types [xfs]
[2020-07-22T15:54:22,155][INFO ][o.e.e.NodeEnvironment    ] [apm-data-node4] heap size [3.8gb], compressed ordinary object pointers [true]
[2020-07-22T15:54:22,699][INFO ][o.e.n.Node               ] [apm-data-node4] node name [apm-data-node4], node ID [B9s7SzjMRE65YF4wrULl7Q]
[2020-07-22T15:54:22,700][INFO ][o.e.n.Node               ] [apm-data-node4] version[6.5.4], pid[25116], build[default/rpm/d2ef93d/2018-12-17T21:17:40.758843Z], OS[Linux/3.10.0-693.21.1.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_201/25.201-b09]
[2020-07-22T15:54:22,700][INFO ][o.e.n.Node               ] [apm-data-node4] JVM arguments [-ea, -Xms20g, -Xmx20g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Des.cgroups.hierarchy.override=/, -Xms4g, -Xmx4g, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.YJ1pgI3u, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm]
[2020-07-22T15:54:23,860][INFO ][o.e.p.PluginsService     ] [apm-data-node4] loaded module [aggs-matrix-stats]
[2020-07-22T15:54:23,861][INFO ][o.e.p.PluginsService     ] [apm-data-node4] loaded module [analysis-common]
[2020-07-22T15:54:23,861][INFO ][o.e.p.PluginsService     ] [apm-data-node4] loaded module [ingest-common]
[2020-07-22T15:54:23,861][INFO ][o.e.p.PluginsService     ] [apm-data-node4] loaded module [lang-expression]
[2020-07-22T15:54:23,861][INFO ][o.e.p.PluginsService     ] [apm-data-node4] loaded module [lang-mustache]
[2020-07-22T15:54:23,861][INFO ][o.e.p.PluginsService     ] [apm-data-node4] loaded module [lang-painless]
[2020-07-22T15:54:23,861][INFO ][o.e.p.PluginsService     ] [apm-data-node4] loaded module [mapper-extras]
[2020-07-22T15:54:23,861][INFO ][o.e.p.PluginsService     ] [apm-data-node4] loaded module [parent-join]
[2020-07-22T15:54:23,861][INFO ][o.e.p.PluginsService     ] [apm-data-node4] loaded module [percolator]
[2020-07-22T15:54:23,861][INFO ][o.e.p.PluginsService     ] [apm-data-node4] loaded module [rank-eval]
[2020-07-22T15:54:23,861][INFO ][o.e.p.PluginsService     ] [apm-data-node4] loaded module [reindex]
[2020-07-22T15:54:23,861][INFO ][o.e.p.PluginsService     ] [apm-data-node4] loaded module [repository-url]
[2020-07-22T15:54:23,861][INFO ][o.e.p.PluginsService     ] [apm-data-node4] loaded module [transport-netty4]
[2020-07-22T15:54:23,861][INFO ][o.e.p.PluginsService     ] [apm-data-node4] loaded module [tribe]
[2020-07-22T15:54:23,861][INFO ][o.e.p.PluginsService     ] [apm-data-node4] loaded module [x-pack-ccr]
[2020-07-22T15:54:23,861][INFO ][o.e.p.PluginsService     ] [apm-data-node4] loaded module [x-pack-core]
[2020-07-22T15:54:23,861][INFO ][o.e.p.PluginsService     ] [apm-data-node4] loaded module [x-pack-deprecation]
[2020-07-22T15:54:23,861][INFO ][o.e.p.PluginsService     ] [apm-data-node4] loaded module [x-pack-graph]
[2020-07-22T15:54:23,861][INFO ][o.e.p.PluginsService     ] [apm-data-node4] loaded module [x-pack-logstash]
[2020-07-22T15:54:23,861][INFO ][o.e.p.PluginsService     ] [apm-data-node4] loaded module [x-pack-ml]
[2020-07-22T15:54:23,861][INFO ][o.e.p.PluginsService     ] [apm-data-node4] loaded module [x-pack-monitoring]
[2020-07-22T15:54:23,861][INFO ][o.e.p.PluginsService     ] [apm-data-node4] loaded module [x-pack-rollup]
[2020-07-22T15:54:23,861][INFO ][o.e.p.PluginsService     ] [apm-data-node4] loaded module [x-pack-security]
[2020-07-22T15:54:23,861][INFO ][o.e.p.PluginsService     ] [apm-data-node4] loaded module [x-pack-sql]
[2020-07-22T15:54:23,861][INFO ][o.e.p.PluginsService     ] [apm-data-node4] loaded module [x-pack-upgrade]
[2020-07-22T15:54:23,861][INFO ][o.e.p.PluginsService     ] [apm-data-node4] loaded module [x-pack-watcher]
[2020-07-22T15:54:23,862][INFO ][o.e.p.PluginsService     ] [apm-data-node4] no plugins loaded
[2020-07-22T15:54:23,880][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [apm-data-node4] fatal error in thread [main], exiting
java.lang.AssertionError: null
        at org.elasticsearch.monitor.os.OsProbe.getCgroup(OsProbe.java:487) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.monitor.os.OsProbe.osStats(OsProbe.java:531) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.xpack.ml.MachineLearning.additionalSettings(MachineLearning.java:327) ~[?:?]
        at org.elasticsearch.plugins.PluginsService.updatedSettings(PluginsService.java:203) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.node.Node.<init>(Node.java:339) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.5.4.jar:6.5.4]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.5.4.jar:6.5.4]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.5.4.jar:6.5.4]

How can I fix this?

Thank you

Is that all that is in the logs?

Hi~ I edit the logs. Thank you

Thanks!
Is there anything after that last entry?

Yes.. There are no more logs after last entry..

Why is -Xms, -Xmx on the command line twice? Both for 20G and 4G?

This in docker, as I see OsProbe.getCgroup as last area the exception happened, which is odd - any other strange settings as it's still probing for OS and related settings.

Looks like a CentOS VM but maybe some weird cgroup settings, like cgroup is returning null, which I suspect it should not do (I think Linux has a default cgroup).

Seems this happens sometimes, see these and fixed in V7.4


Also some workarounds here:

Thank you

It had problems for mount.

I mounted cgroups, then the elasticsearch was started.

Thank you for your help :slight_smile:

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.