The following is the log. Here are my questions.
- What's the cause of the crash.
- After crash, I restart the server, it can't use the historical data, instead, it only use the new disk. Can it recover from history data?
- Is there any method to check the search history?
Thanks
[2018-10-26T00:04:25,952][INFO ][o.e.m.j.JvmGcMonitorService] [phxriskdna016] [gc][10346442] overhead, spent [265ms] collecting in the last [1s]
[2018-10-26T00:53:17,075][ERROR][o.e.t.n.Netty4Utils ] fatal error on the network layer
at org.elasticsearch.transport.netty4.Netty4Utils.maybeDie(Netty4Utils.java:182)
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.exceptionCaught(Netty4HttpRequestHandler.java:176)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285)
at io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:850)
....
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at java.lang.Thread.run(Thread.java:745)
[2018-10-26T00:53:17,079][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [phxriskdna016] fatal error in thread [Thread-1108904], exiting
java.lang.StackOverflowError: null
at org.apache.lucene.util.automaton.RegExp.parseCharClassExp(RegExp.java:1092) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
at org.apache.lucene.util.automaton.RegExp.parseComplExp(RegExp.java:1080) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
at org.apache.lucene.util.automaton.RegExp.parseRepeatExp(RegExp.java:1049) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1042) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1044) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1044) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1044) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1044) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1044) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1044) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1044) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
[2018-10-26T01:21:06,079][INFO ][o.e.n.Node ] [phxriskdna016] initializing ...
[2018-10-26T01:21:06,198][INFO ][o.e.e.NodeEnvironment ] [phxriskdna016] using [3] data paths, mounts [[/data05 (/dev/sdf1), /data06 (/dev/sdg1), /data04 (/dev/sde1)]], net usable_space [949.4gb], net tot
al_space [5.4tb], types [xfs]
[2018-10-26T01:21:06,198][INFO ][o.e.e.NodeEnvironment ] [phxriskdna016] heap size [23.8gb], compressed ordinary object pointers [true]
[2018-10-26T01:21:26,066][INFO ][o.e.n.Node ] [phxriskdna016] node name [phxriskdna016], node ID [AYPmZaAEQoy_B-20jLOVtQ]
[2018-10-26T01:21:26,066][INFO ][o.e.n.Node ] [phxriskdna016] version[6.3.0], pid[32279], build[default/tar/424e937/2018-06-11T23:38:03.357887Z], OS[Linux/3.10.0-514.el7.x86_64/amd64], JVM[Orac
le Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_73/25.73-b02]
[2018-10-26T01:21:26,067][INFO ][o.e.n.Node ] [phxriskdna016] JVM arguments [-Xms24g, -Xmx24g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly
, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.re
cycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.zlJPw4WQ, -XX:+HeapDumpOnOutOfMemoryError, -XX:+PrintGCDetails, -XX:+PrintGCDa
teStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/x/home/pp_risk_
rom_batch/elasticsearch-6.3.0, -Des.path.conf=/x/home/pp_risk_rom_batch/elasticsearch-6.3.0/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]
[2018-10-26T01:21:30,119][INFO ][o.e.p.PluginsService ] [phxriskdna016] loaded module [aggs-matrix-stats]
[2018-10-26T01:21:30,119][INFO ][o.e.p.PluginsService ] [phxriskdna016] loaded module [analysis-common]