Hi guys I have problem with elasticsearch Java its stopping, please see logs below

mabialakienda · March 11, 2018, 7:46am

org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:869) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:164) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:230) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:115) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4443) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4083) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:99) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
[2018-03-10T18:07:46,696][INFO ][o.e.n.Node ] [] initializing ...
[2018-03-10T18:07:46,818][INFO ][o.e.e.NodeEnvironment ] [9hb_Gdj] using [1] data paths, mounts [[/var (/dev/md126)]], net usable_space [1.6tb], net total_space [1.7tb], types [ext4]
[2018-03-10T18:07:46,819][INFO ][o.e.e.NodeEnvironment ] [9hb_Gdj] heap size [989.8mb], compressed ordinary object pointers [true]
[2018-03-10T18:07:46,948][INFO ][o.e.n.Node ] node name [9hb_Gdj] derived from node ID [9hb_GdjPTH-qN-aMrqivkg]; set [node.name] to override
[2018-03-10T18:07:46,948][INFO ][o.e.n.Node ] version[6.2.2], pid[32615], build[10b1edd/2018-02-16T19:01:30.685723Z], OS[Linux/4.9.0-6-amd64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_161/25.161-b12]
[2018-03-10T18:07:46,948][INFO ][o.e.n.Node ] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.DNj7es11, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch]
[2018-03-10T18:07:47,571][INFO ][o.e.p.PluginsService ] [9hb_Gdj] loaded module [aggs-matrix-stats]
[2018-03-10T18:07:47,571][INFO ][o.e.p.PluginsService ] [9hb_Gdj] loaded module [analysis-common]
[2018-03-10T18:07:47,571][INFO ][o.e.p.PluginsService ] [9hb_Gdj] loaded module [ingest-common]
[2018-03-10T18:07:47,571][INFO ][o.e.p.PluginsService ] [9hb_Gdj] loaded module [lang-expression]
[2018-03-10T18:07:47,571][INFO ][o.e.p.PluginsService ] [9hb_Gdj] loaded module [lang-mustache]
[2018-03-10T18:07:47,571][INFO ][o.e.p.PluginsService ] [9hb_Gdj] loaded module [lang-painless]
[2018-03-10T18:07:47,571][INFO ][o.e.p.PluginsService ] [9hb_Gdj] loaded module [mapper-extras]
[2018-03-10T18:07:47,571][INFO ][o.e.p.PluginsService ] [9hb_Gdj] loaded module [parent-join]
[2018-03-10T18:07:47,571][INFO ][o.e.p.PluginsService ] [9hb_Gdj] loaded module [percolator]
[2018-03-10T18:07:47,571][INFO ][o.e.p.PluginsService ] [9hb_Gdj] loaded module [rank-eval]
[2018-03-10T18:07:47,572][INFO ][o.e.p.PluginsService ] [9hb_Gdj] loaded module [reindex]
[2018-03-10T18:07:47,572][INFO ][o.e.p.PluginsService ] [9hb_Gdj] loaded module [repository-url]
[2018-03-10T18:07:47,572][INFO ][o.e.p.PluginsService ] [9hb_Gdj] loaded module [transport-netty4]
[2018-03-10T18:07:47,572][INFO ][o.e.p.PluginsService ] [9hb_Gdj] loaded module [tribe]
[2018-03-10T18:07:47,572][INFO ][o.e.p.PluginsService ] [9hb_Gdj] no plugins loaded
[2018-03-10T18:07:50,027][INFO ][o.e.d.DiscoveryModule ] [9hb_Gdj] using discovery type [zen]
[2018-03-10T18:07:50,521][INFO ][o.e.n.Node ] initialized
[2018-03-10T18:07:50,521][INFO ][o.e.n.Node ] [9hb_Gdj] starting ...
[2018-03-10T18:07:50,656][INFO ][o.e.t.TransportService ] [9hb_Gdj] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
[2018-03-10T18:07:53,827][INFO ][o.e.c.s.MasterService ] [9hb_Gdj] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {9hb_Gdj}{9hb_GdjPTH-qN-aMrqivkg}{A0pnHA65QCOxGr9sd_d3Mg}{localhost}{127.0.0.1:9300}
[2018-03-10T18:07:53,832][INFO ][o.e.c.s.ClusterApplierService] [9hb_Gdj] new_master {9hb_Gdj}{9hb_GdjPTH-qN-aMrqivkg}{A0pnHA65QCOxGr9sd_d3Mg}{localhost}{127.0.0.1:9300}, reason: apply cluster state (from master [master {9hb_Gdj}{9hb_GdjPTH-qN-aMrqivkg}{A0pnHA65QCOxGr9sd_d3Mg}{localhost}{127.0.0.1:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-03-10T18:07:53,850][INFO ][o.e.h.n.Netty4HttpServerTransport] [9hb_Gdj] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200}
[2018-03-10T18:07:53,850][INFO ][o.e.n.Node ] [9hb_Gdj] started
[2018-03-10T18:07:55,284][INFO ][o.e.g.GatewayService ] [9hb_Gdj] recovered [19] indices into cluster_state
[2018-03-10T18:08:15,566][INFO ][o.e.c.r.a.AllocationService] [9hb_Gdj] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[logstash-2015.05.20][3], [logstash-2015.05.20][2], [logstash-2015.05.20][4]] ...]).

dadoonet · March 11, 2018, 8:24am

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.

Also move your title content in the body instead and add a more explicit title.

Please update your post.

mabialakienda · March 11, 2018, 9:39am

Hi Dadoonet

Thanks for your replay, but im new in this and I dont know how to struture this because I dont know where the line started and where that stopped.

We have more then 23gb memery and 1.7tb disck, using Linux debian 8

[2018-03-10T18:07:50,656][INFO ][o.e.t.TransportService ] [9hb_Gdj] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}

[2018-03-10T18:07:46,948][INFO ][o.e.n.Node ] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.DNj7es11, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch]

dadoonet · March 11, 2018, 10:26am

Just use markdown format as I explained to make all your logs looking like code.

mabialakienda · March 11, 2018, 10:38am

Looking to this, its working but to work I have to disable filebeat index, my implementation was for Squid Transparent.

I need to upgrade memory in Java to feat my implementation.

[2018-03-11T04:57:23,783][WARN ][o.e.b.JNANatives ] Unable to lock JVM Memory: error=12, reason=Não foi possível alocar memória
[2018-03-11T04:57:23,789][WARN ][o.e.b.JNANatives ] This can result in part of the JVM being swapped out.
[2018-03-11T04:57:23,790][WARN ][o.e.b.JNANatives ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2018-03-11T04:57:23,790][WARN ][o.e.b.JNANatives ] These can be adjusted by modifying /etc/security/limits.conf, for example:
# allow user 'elasticsearch' mlockall
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited

[2018-03-11T04:57:23,790][WARN ][o.e.b.JNANatives ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2018-03-11T04:57:23,935][INFO ][o.e.n.Node ] [] initializing ...
[2018-03-11T04:57:24,073][INFO ][o.e.e.NodeEnvironment ] [9hb_Gdj] using [1] data paths, mounts [[/var (/dev/md126)]], net usable_space [1.6tb], net total_space [1.7tb], types [ext4]

[2018-03-11T04:57:24,073][INFO ][o.e.e.NodeEnvironment ] [9hb_Gdj] heap size [989.8mb], compressed ordinary object pointers [true]
[2018-03-11T04:57:24,194][INFO ][o.e.n.Node ] node name [9hb_Gdj] derived from node ID [9hb_GdjPTH-qN-aMrqivkg]; set [node.name] to override

[2018-03-11T04:57:24,194][INFO ][o.e.n.Node ] version[6.2.2], pid[3009], build[10b1edd/2018-02-16T19:01:30.685723Z], OS[Linux/4.9.0-6-amd64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_161/25.161-b12]

[2018-03-11T04:57:24,194][INFO ][o.e.n.Node ] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.yJqDcFtp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch]

[2018-03-11T04:57:24,782][INFO ][o.e.p.PluginsService [2018-03-11T04:57:24,783][INFO ][o.e.p.PluginsService [2018-03-11T04:57:24,783][INFO ][o.e.p.PluginsService [2018-03-11T04:57:24,783][INFO ][o.e.p.PluginsService [2018-03-11T04:57:24,783][INFO ][o.e.p.PluginsService [2018-03-11T04:57:24,783][INFO ][o.e.p.PluginsService [2018-03-11T04:57:24,783][INFO ][o.e.p.PluginsService [2018-03-11T04:57:24,783][INFO ][o.e.p.PluginsService [2018-03-11T04:57:24,783][INFO ][o.e.p.PluginsService [2018-03-11T04:57:24,783][INFO ][o.e.p.PluginsService [2018-03-11T04:57:24,783][INFO ][o.e.p.PluginsService [2018-03-11T04:57:24,783][INFO ][o.e.p.PluginsService [2018-03-11T04:57:24,783][INFO ][o.e.p.PluginsService [2018-03-11T04:57:24,783][INFO ][o.e.p.PluginsService [2018-03-11T04:57:24,784][INFO ][o.e.p.PluginsService [2018-03-11T04:57:27,236][INFO ][o.e.d.DiscoveryModule [2018-03-11T04:57:27,732][INFO ][o.e.n.Node [2018-03-11T04:57:27,732][INFO ][o.e.n.Node [2018-03-11T04:57:27,862][INFO ][o.e.t.TransportService [2018-03-11T04:57:27,906][WARN ][o.e.b.BootstrapChecks ] [9hb_Gdj] loaded module [aggs-matrix-stats]
] [9hb_Gdj] loaded module [analysis-common]
] [9hb_Gdj] loaded module [ingest-common]
] [9hb_Gdj] loaded module [lang-expression]
] [9hb_Gdj] loaded module [lang-mustache]
] [9hb_Gdj] loaded module [lang-painless]
] [9hb_Gdj] loaded module [mapper-extras]
] [9hb_Gdj] loaded module [parent-join]
] [9hb_Gdj] loaded module [percolator]
] [9hb_Gdj] loaded module [rank-eval]
] [9hb_Gdj] loaded module [reindex]
] [9hb_Gdj] loaded module [repository-url]
] [9hb_Gdj] loaded module [transport-netty4]
] [9hb_Gdj] loaded module [tribe]
] [9hb_Gdj] no plugins loaded
] [9hb_Gdj] using discovery type [zen]
] initialized
] [9hb_Gdj] starting ...
] [9hb_Gdj] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
] [9hb_Gdj] memory locking requested for elasticsearch process but memory is not locked

[2018-03-11T04:57:30,956][INFO ][o.e.c.s.MasterService ] [9hb_Gdj] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {9hb_Gdj}{9hb_GdjPTH-qN-aMrqivkg}{ZNNXw_2bSO6hTRwwWeIlEg}{localhost}{127.0.0.1:9300}

[2018-03-11T04:57:30,961][INFO ][o.e.c.s.ClusterApplierService] [9hb_Gdj] new_master {9hb_Gdj}{9hb_GdjPTH-qN-aMrqivkg}{ZNNXw_2bSO6hTRwwWeIlEg}{localhost}{127.0.0.1:9300}, reason: apply cluster state (from master [master {9hb_Gdj}{9hb_GdjPTH-qN-aMrqivkg}{ZNNXw_2bSO6hTRwwWeIlEg}{localhost}{127.0.0.1:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])

[2018-03-11T04:57:30,981][INFO ][o.e.h.n.Netty4HttpServerTransport] [9hb_Gdj] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200}

[2018-03-11T04:57:30,982][INFO ][o.e.n.Node ] [9hb_Gdj] started
[2018-03-11T04:57:32,507][INFO ][o.e.g.GatewayService ] [9hb_Gdj] recovered [19] indices into cluster_state
[2018-03-11T04:57:59,050][INFO ][o.e.c.r.a.AllocationService] [9hb_Gdj] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[logstash-2015.05.20][0]] ...]).

[2018-03-11T04:58:08,287][INFO ][o.e.c.m.MetaDataCreateIndexService] [9hb_Gdj] [logstash-2018.03.11] creating index, cause [auto(bulk api)], templates [logstash], shards [5]/[1], mappings [default]

[2018-03-11T04:58:09,568][INFO ][o.e.c.m.MetaDataMappingService] [9hb_Gdj] [logstash-2018.03.11/FmEPa6S9R12uUrb2Ke_h1g] create_mapping [doc]
[2018-03-11T04:59:37,168][INFO ][o.e.c.m.MetaDataMappingService] [9hb_Gdj] [logstash-2018.03.11/FmEPa6S9R12uUrb2Ke_h1g] update_mapping [doc]
[2018-03-11T04:59:37,404][INFO ][o.e.c.m.MetaDataMappingService] [9hb_Gdj] [logstash-2018.03.11/FmEPa6S9R12uUrb2Ke_h1g] update_mapping [doc]
[2018-03-11T04:59:37,495][INFO ][o.e.c.m.MetaDataMappingService] [9hb_Gdj] [logstash-2018.03.11/FmEPa6S9R12uUrb2Ke_h1g] update_mapping [doc]
[2018-03-11T07:23:37,898][INFO ][o.e.c.m.MetaDataMappingService] [9hb_Gdj] [logstash-2018.03.11/FmEPa6S9R12uUrb2Ke_h1g] update_mapping [doc]

dadoonet · March 11, 2018, 11:16am

Please format your logs.

mabialakienda · March 11, 2018, 2:07pm

Hi Dadoonet, as I said, I dont know how to format it Im new in this challenge, I guess this will not take to you long time, could you take one of my posted log to format and show me this first time?

I will appreciate

dadoonet · March 11, 2018, 3:10pm

What is unclear when I say to use:

```
Insert your logs here
```

?

mabialakienda · March 11, 2018, 3:23pm

Thanks for support

"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>16}


[2018-03-11T04:57:23,783][WARN ][o.e.b.JNANatives         ] Unable to lock JVM Memory: error=12, reason=Não foi possível alocar memória
[2018-03-11T04:57:23,789][WARN ][o.e.b.JNANatives         ] This can result in part of the JVM being swapped out.
[2018-03-11T04:57:23,790][WARN ][o.e.b.JNANatives         ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2018-03-11T04:57:23,790][WARN ][o.e.b.JNANatives         ] These can be adjusted by modifying /etc/security/limits.conf, for example: 
	# allow user 'elasticsearch' mlockall
			elasticsearch soft memlock unlimited
			elasticsearch hard memlock unlimited


[2018-03-11T04:57:23,790][WARN ][o.e.b.JNANatives         ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2018-03-11T04:57:23,935][INFO ][o.e.n.Node               ] [] initializing ...
[2018-03-11T04:57:24,073][INFO ][o.e.e.NodeEnvironment    ] [9hb_Gdj] using [1] data paths, mounts [[/var (/dev/md126)]], net usable_space [1.6tb], net total_space [1.7tb], types [ext4]

[2018-03-11T04:57:24,073][INFO ][o.e.e.NodeEnvironment    ] [9hb_Gdj] heap size [989.8mb], compressed ordinary object pointers [true]
[2018-03-11T04:57:24,194][INFO ][o.e.n.Node               ] node name [9hb_Gdj] derived from node ID [9hb_GdjPTH-qN-aMrqivkg]; set [node.name] to override

[2018-03-11T04:57:24,194][INFO ][o.e.n.Node               ] version[6.2.2], pid[3009], build[10b1edd/2018-02-16T19:01:30.685723Z], OS[Linux/4.9.0-6-amd64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_161/25.161-b12]

[2018-03-11T04:57:24,194][INFO ][o.e.n.Node               ] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.yJqDcFtp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch]

[2018-03-11T04:57:24,782][INFO ][o.e.p.PluginsService     ] [9hb_Gdj] loaded module [aggs-matrix-stats]
[2018-03-11T04:57:24,783][INFO ][o.e.p.PluginsService     ] [9hb_Gdj] loaded module [analysis-common]
[2018-03-11T04:57:24,783][INFO ][o.e.p.PluginsService     ] [9hb_Gdj] loaded module [ingest-common]
[2018-03-11T04:57:24,783][INFO ][o.e.p.PluginsService     ] [9hb_Gdj] loaded module [lang-expression]
[2018-03-11T04:57:24,783][INFO ][o.e.p.PluginsService     ] [9hb_Gdj] loaded module [lang-mustache]
[2018-03-11T04:57:24,783][INFO ][o.e.p.PluginsService     ] [9hb_Gdj] loaded module [lang-painless]
[2018-03-11T04:57:24,783][INFO ][o.e.p.PluginsService     ] [9hb_Gdj] loaded module [mapper-extras]
[2018-03-11T04:57:24,783][INFO ][o.e.p.PluginsService     ] [9hb_Gdj] loaded module [parent-join]
[2018-03-11T04:57:24,783][INFO ][o.e.p.PluginsService     ] [9hb_Gdj] loaded module [percolator]
[2018-03-11T04:57:24,783][INFO ][o.e.p.PluginsService     ] [9hb_Gdj] loaded module [rank-eval]
[2018-03-11T04:57:24,783][INFO ][o.e.p.PluginsService     ] [9hb_Gdj] loaded module [reindex]
[2018-03-11T04:57:24,783][INFO ][o.e.p.PluginsService     ] [9hb_Gdj] loaded module [repository-url]
[2018-03-11T04:57:24,783][INFO ][o.e.p.PluginsService     ] [9hb_Gdj] loaded module [transport-netty4]
[2018-03-11T04:57:24,783][INFO ][o.e.p.PluginsService     ] [9hb_Gdj] loaded module [tribe]
[2018-03-11T04:57:24,784][INFO ][o.e.p.PluginsService     ] [9hb_Gdj] no plugins loaded
[2018-03-11T04:57:27,236][INFO ][o.e.d.DiscoveryModule    ] [9hb_Gdj] using discovery type [zen]
[2018-03-11T04:57:27,732][INFO ][o.e.n.Node               ] initialized
[2018-03-11T04:57:27,732][INFO ][o.e.n.Node               ] [9hb_Gdj] starting ...
[2018-03-11T04:57:27,862][INFO ][o.e.t.TransportService   ] [9hb_Gdj] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
[2018-03-11T04:57:27,906][WARN ][o.e.b.BootstrapChecks    ] [9hb_Gdj] memory locking requested for elasticsearch process but memory is not locked

[2018-03-11T04:57:30,956][INFO ][o.e.c.s.MasterService    ] [9hb_Gdj] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {9hb_Gdj}{9hb_GdjPTH-qN-aMrqivkg}{ZNNXw_2bSO6hTRwwWeIlEg}{localhost}{127.0.0.1:9300}

[2018-03-11T04:57:30,961][INFO ][o.e.c.s.ClusterApplierService] [9hb_Gdj] new_master {9hb_Gdj}{9hb_GdjPTH-qN-aMrqivkg}{ZNNXw_2bSO6hTRwwWeIlEg}{localhost}{127.0.0.1:9300}, reason: apply cluster state (from master [master {9hb_Gdj}{9hb_GdjPTH-qN-aMrqivkg}{ZNNXw_2bSO6hTRwwWeIlEg}{localhost}{127.0.0.1:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])

[2018-03-11T04:57:30,981][INFO ][o.e.h.n.Netty4HttpServerTransport] [9hb_Gdj] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200}

[2018-03-11T04:57:30,982][INFO ][o.e.n.Node               ] [9hb_Gdj] started
[2018-03-11T04:57:32,507][INFO ][o.e.g.GatewayService     ] [9hb_Gdj] recovered [19] indices into cluster_state
[2018-03-11T04:57:59,050][INFO ][o.e.c.r.a.AllocationService] [9hb_Gdj] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[logstash-2015.05.20][0]] ...]).

[2018-03-11T04:58:08,287][INFO ][o.e.c.m.MetaDataCreateIndexService] [9hb_Gdj] [logstash-2018.03.11] creating index, cause [auto(bulk api)], templates [logstash], shards [5]/[1], mappings [_default_]

[2018-03-11T04:58:09,568][INFO ][o.e.c.m.MetaDataMappingService] [9hb_Gdj] [logstash-2018.03.11/FmEPa6S9R12uUrb2Ke_h1g] create_mapping [doc]
[2018-03-11T04:59:37,168][INFO ][o.e.c.m.MetaDataMappingService] [9hb_Gdj] [logstash-2018.03.11/FmEPa6S9R12uUrb2Ke_h1g] update_mapping [doc]
[2018-03-11T04:59:37,404][INFO ][o.e.c.m.MetaDataMappingService] [9hb_Gdj] [logstash-2018.03.11/FmEPa6S9R12uUrb2Ke_h1g] update_mapping [doc]
[2018-03-11T04:59:37,495][INFO ][o.e.c.m.MetaDataMappingService] [9hb_Gdj] [logstash-2018.03.11/FmEPa6S9R12uUrb2Ke_h1g] update_mapping [doc]
[2018-03-11T07:23:37,898][INFO ][o.e.c.m.MetaDataMappingService] [9hb_Gdj] [logstash-2018.03.11/FmEPa6S9R12uUrb2Ke_h1g] update_mapping [doc]

mabialakienda · March 11, 2018, 3:30pm

Its a JAVA memory error - java.lang.OutOfMemoryError: Java heap space

java.lang.OutOfMemoryError: Java heap space
	at org.apache.lucene.util.packed.PackedLongValues$Builder.<init>(PackedLongValues.java:185) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
	at org.apache.lucene.util.packed.DeltaPackedLongValues$Builder.<init>(DeltaPackedLongValues.java:59) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
	at org.apache.lucene.util.packed.PackedLongValues.deltaPackedBuilder(PackedLongValues.java:55) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
	at org.apache.lucene.util.packed.PackedLongValues.deltaPackedBuilder(PackedLongValues.java:60) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
	at org.apache.lucene.index.SortedSetDocValuesWriter.<init>(SortedSetDocValuesWriter.java:70) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
	at org.apache.lucene.index.DefaultIndexingChain.indexDocValue(DefaultIndexingChain.java:562) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
	at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:466) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:392) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:240) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:496) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1729) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1464) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
	at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:1070) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1012) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:878) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:738) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:707) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:673) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:548) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequest(TransportShardBulkAction.java:140) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:236) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:123) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:110) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:72) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1034) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1012) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:103) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:238) ~[elasticsearch-6.2.2.jar:6.2.2]
[2018-03-10T14:18:19,314][INFO ][o.e.n.Node               ] [] initializing ...
[2018-03-10T14:18:19,442][INFO ][o.e.e.NodeEnvironment    ] [9hb_Gdj] using [1] data paths, mounts [[/var (/dev/md126)]], net usable_space [1.6tb], net total_space [1.7tb], types [ext4]
[2018-03-10T14:18:19,442][INFO ][o.e.e.NodeEnvironment    ] [9hb_Gdj] heap size [989.8mb], compressed ordinary object pointers [true]
[2018-03-10T14:18:19,568][INFO ][o.e.n.Node               ] node name [9hb_Gdj] derived from node ID [9hb_GdjPTH-qN-aMrqivkg]; set [node.name] to override
[2018-03-10T14:18:19,568][INFO ][o.e.n.Node               ] version[6.2.2], pid[19121], build[10b1edd/2018-02-16T19:01:30.685723Z], OS[Linux/4.9.0-6-amd64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_161/25.161-b12]
[2018-03-10T14:18:19,568][INFO ][o.e.n.Node               ] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.pfVLbZJk, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch]

dadoonet · March 11, 2018, 5:47pm

Sounds like you are starting with only 1gb of heap: -Xms1g, -Xmx1g

May be give more memory?

How much data you have? Are you running a query when this happens?

mabialakienda · March 11, 2018, 7:08pm

The project I have is, that I have a squid in transparent mode, send logs to Logstash and elasticsearch receiving and sending to Kibana based in this project - https://reticent.net.nz/visualising-kibana-squid-logs/

I have 23gb and 15gb located to squid cache and 1.7tb for cache

I need to increase Elastic memory its stopping java or java is stopping elastic

dadoonet · March 11, 2018, 7:25pm

Not sure I'm fully understanding.

Anyway, you need to stop the node, change the jvm.options settings and start again the node.

mabialakienda · March 12, 2018, 6:32am

Hi Dadoonet

Please check this as I have adjust my memory to increase ELS but getting this error below, unable to lock JVM Memory, I have configured the limites.conf file to include both line SOFT and HARD, but I still getting this error.

dadoonet:

[2018-03-12T07:16:50,696][WARN ][o.e.b.JNANatives         ] Unable to lock JVM Memory: error=12, reason=Não foi possível alocar memória
[2018-03-12T07:16:50,707][WARN ][o.e.b.JNANatives         ] This can result in part of the JVM being swapped out.
[2018-03-12T07:16:50,707][WARN ][o.e.b.JNANatives         ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2018-03-12T07:16:50,708][WARN ][o.e.b.JNANatives         ] These can be adjusted by modifying /etc/security/limits.conf, for example: 
	# allow user 'elasticsearch' mlockall
	elasticsearch soft memlock unlimited
	elasticsearch hard memlock unlimited
[2018-03-12T07:16:50,708][WARN ][o.e.b.JNANatives         ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2018-03-12T07:16:50,979][INFO ][o.e.n.Node               ] [] initializing ...
[2018-03-12T07:16:51,179][INFO ][o.e.e.NodeEnvironment    ] [9hb_Gdj] using [1] data paths, mounts [[/var (/dev/md126)]], net usable_space [1.6tb], net total_space [1.7tb], types [ext4]
[2018-03-12T07:16:51,179][INFO ][o.e.e.NodeEnvironment    ] [9hb_Gdj] heap size [5.9gb], compressed ordinary object pointers [true]
[2018-03-12T07:16:51,482][INFO ][o.e.n.Node               ] node name [9hb_Gdj] derived from node ID [9hb_GdjPTH-qN-aMrqivkg]; set [node.name] to override
[2018-03-12T07:16:51,483][INFO ][o.e.n.Node               ] version[6.2.2], pid[624], build[10b1edd/2018-02-16T19:01:30.685723Z], OS[Linux/4.9.0-6-amd64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_161/25.161-b12]
[2018-03-12T07:16:51,483][INFO ][o.e.n.Node               ] JVM arguments [-Xms4g, -Xmx6g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.CY1BH4AF, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch]

mabialakienda · March 12, 2018, 6:40am

And the second scenario is about does 6 WARN you can check, Why its causing exception on http traffic?

dadoonet:

[2018-03-12T07:16:52,490][INFO ][o.e.p.PluginsService     ] [9hb_Gdj] loaded module [aggs-matrix-stats]
[2018-03-12T07:16:52,490][INFO ][o.e.p.PluginsService     ] [9hb_Gdj] loaded module [analysis-common]
[2018-03-12T07:16:52,491][INFO ][o.e.p.PluginsService     ] [9hb_Gdj] loaded module [ingest-common]
[2018-03-12T07:16:52,491][INFO ][o.e.p.PluginsService     ] [9hb_Gdj] loaded module [lang-expression]
[2018-03-12T07:16:52,491][INFO ][o.e.p.PluginsService     ] [9hb_Gdj] loaded module [lang-mustache]
[2018-03-12T07:16:52,491][INFO ][o.e.p.PluginsService     ] [9hb_Gdj] loaded module [lang-painless]
[2018-03-12T07:16:52,491][INFO ][o.e.p.PluginsService     ] [9hb_Gdj] loaded module [mapper-extras]
[2018-03-12T07:16:52,491][INFO ][o.e.p.PluginsService     ] [9hb_Gdj] loaded module [parent-join]
[2018-03-12T07:16:52,491][INFO ][o.e.p.PluginsService     ] [9hb_Gdj] loaded module [percolator]
[2018-03-12T07:16:52,491][INFO ][o.e.p.PluginsService     ] [9hb_Gdj] loaded module [rank-eval]
[2018-03-12T07:16:52,491][INFO ][o.e.p.PluginsService     ] [9hb_Gdj] loaded module [reindex]
[2018-03-12T07:16:52,491][INFO ][o.e.p.PluginsService     ] [9hb_Gdj] loaded module [repository-url]
[2018-03-12T07:16:52,492][INFO ][o.e.p.PluginsService     ] [9hb_Gdj] loaded module [transport-netty4]
[2018-03-12T07:16:52,492][INFO ][o.e.p.PluginsService     ] [9hb_Gdj] loaded module [tribe]
[2018-03-12T07:16:52,492][INFO ][o.e.p.PluginsService     ] [9hb_Gdj] no plugins loaded
[2018-03-12T07:16:56,674][INFO ][o.e.d.DiscoveryModule    ] [9hb_Gdj] using discovery type [zen]
[2018-03-12T07:16:57,363][INFO ][o.e.n.Node               ] initialized
[2018-03-12T07:16:57,364][INFO ][o.e.n.Node               ] [9hb_Gdj] starting ...
[2018-03-12T07:16:57,644][INFO ][o.e.t.TransportService   ] [9hb_Gdj] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
[2018-03-12T07:16:57,700][WARN ][o.e.b.BootstrapChecks    ] [9hb_Gdj] initial heap size [4294967296] not equal to maximum heap size [6442450944]; this can cause resize pauses and prevents mlockall from locking the entire heap
[2018-03-12T07:16:57,700][WARN ][o.e.b.BootstrapChecks    ] [9hb_Gdj] memory locking requested for elasticsearch process but memory is not locked
[2018-03-12T07:17:00,776][INFO ][o.e.c.s.MasterService    ] [9hb_Gdj] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {9hb_Gdj}{9hb_GdjPTH-qN-aMrqivkg}{y-3bzZdKR4SNU8jb78ETXw}{localhost}{127.0.0.1:9300}
[2018-03-12T07:17:00,781][INFO ][o.e.c.s.ClusterApplierService] [9hb_Gdj] new_master {9hb_Gdj}{9hb_GdjPTH-qN-aMrqivkg}{y-3bzZdKR4SNU8jb78ETXw}{localhost}{127.0.0.1:9300}, reason: apply cluster state (from master [master {9hb_Gdj}{9hb_GdjPTH-qN-aMrqivkg}{y-3bzZdKR4SNU8jb78ETXw}{localhost}{127.0.0.1:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-03-12T07:17:00,811][INFO ][o.e.h.n.Netty4HttpServerTransport] [9hb_Gdj] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200}
[2018-03-12T07:17:00,811][INFO ][o.e.n.Node               ] [9hb_Gdj] started
[2018-03-12T07:17:03,168][INFO ][o.e.g.GatewayService     ] [9hb_Gdj] recovered [20] indices into cluster_state
[2018-03-12T07:18:02,169][INFO ][o.e.c.m.MetaDataCreateIndexService] [9hb_Gdj] [logstash-2018.03.12] creating index, cause [auto(bulk api)], templates [logstash], shards [5]/[1], mappings [_default_]
[2018-03-12T07:18:14,661][INFO ][o.e.c.r.a.AllocationService] [9hb_Gdj] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[logstash-2015.05.20][0], [logstash-2015.05.20][4]] ...]).
[2018-03-12T07:18:15,447][INFO ][o.e.c.m.MetaDataMappingService] [9hb_Gdj] [logstash-2018.03.12/d98PBlRJS0qGRhpTp7ejUQ] create_mapping [doc]
[2018-03-12T07:18:17,351][INFO ][o.e.c.m.MetaDataMappingService] [9hb_Gdj] [logstash-2018.03.12/d98PBlRJS0qGRhpTp7ejUQ] update_mapping [doc]
[2018-03-12T07:18:17,542][INFO ][o.e.c.m.MetaDataMappingService] [9hb_Gdj] [logstash-2018.03.12/d98PBlRJS0qGRhpTp7ejUQ] update_mapping [doc]
[2018-03-12T07:18:43,933][WARN ][o.e.h.n.Netty4HttpServerTransport] [9hb_Gdj] caught exception while handling client http traffic, closing connection [id: 0xe4cf9658, L:/127.0.0.1:9200 - R:/127.0.0.1:45554]
java.io.IOException: Conexão fechada pela outra ponta
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:?]
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:?]
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:?]
	at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[?:?]
	at io.netty.buffer.PooledHeapByteBuf.setBytes(PooledHeapByteBuf.java:261) ~[netty-buffer-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106) ~[netty-buffer-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:343) ~[netty-transport-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
[2018-03-12T07:18:43,933][WARN ][o.e.h.n.Netty4HttpServerTransport] [9hb_Gdj] caught exception while handling client http traffic, closing connection [id: 0x4812b1af, L:/127.0.0.1:9200 - R:/127.0.0.1:45548]

dadoonet · March 12, 2018, 6:52am

I edited your posts to remove the unneeded citation you have been adding.
Could you do the same for the last 2 posts?

I'd recommend reading all this chapter: https://www.elastic.co/guide/en/elasticsearch/reference/current/setup.html

mabialakienda · March 12, 2018, 7:10am

Its to install java8 or java9?

dadoonet · March 12, 2018, 7:38am

Both works

system · April 9, 2018, 7:38am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Elasticsearch worked and then stopped Elasticsearch docker	12	1802	September 28, 2021
org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed Elasticsearch	2	2773	March 17, 2021
Merge Error with lucene 4.1 Elasticsearch	8	677	July 6, 2017
Elasticsearch crashing after a while Elasticsearch	4	995	July 6, 2017
Error in bulk indexing - this IndexWriter is closed Elasticsearch	6	4006	July 6, 2017

Hi guys I have problem with elasticsearch Java its stopping, please see logs below

Related topics