# Failed to execute bulk item (index) index **URL:** \ **Category:** Elasticsearch\ **Created:** [June 6, 2016, 2:43am UTC](https://discuss.elastic.co/t/failed-to-execute-bulk-item-index-index/51890 "2016-06-06T02:43:29Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![X-Mars](https://avatars.discourse-cdn.com/v4/letter/x/ea5d25/32.png) [@X-Mars](https://discuss.elastic.co/u/X-Mars)\ **Post date:** [June 6, 2016, 2:43am UTC](https://discuss.elastic.co/t/failed-to-execute-bulk-item-index-index/51890/1 "2016-06-06T02:43:30Z")
dear all ``` **the elasticsearch often have no response 。** ``` \*\* when it's NO RESPONSE ，i can stop the service with “service elasticsearch stop”，i olny can ”KILL -9”，Please help me。\*\* \*\* and i can not open the IP：9200 \*\* \*\* i got some logs\*\* `[2016-06-06 10:33:30,280][DEBUG][action.bulk] [Skullfire] [mars1-2016.06.05][2] failed to execute bulk item (index) index {[mars1-2016.06.05][tomcat][AVUjhuGSAZCFCIZWhp3G], source[{"@timestamp":"2016-06-05T16:04:27.798Z","beat":{"hostname":"MARS1","name":"MARS1"},"count":1,"fields":null,"input_type":"log","message":"127.0.0.1 - - [06/Jun/2016:00:00:00 +0800] \"GET /alfresco/service/api/solr/aclchangesets?fromTime=1462053874624\u0026toTime=1462975474624\u0026maxResults=2000 HTTP/1.1\" 200 128","offset":4799,"source":"/app/tomcat/logs/localhost_access_log2016-06-06.txt","type":"tomcat"}]} ProcessClusterEventTimeoutException[failed to process cluster event (put-mapping [tomcat]) within 30s] at org.elasticsearch.cluster.service.InternalClusterService$2$1.run(InternalClusterService.java:349) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [2016-06-06 10:33:34,747][WARN][cluster.service] [Skullfire] cluster state update task [shard-started ([mars2-2016.06.06][0], node[IfN3KNH_SEu0nz8eUGCamA], [P], v[1], s[INITIALIZING], a[id=BXA1MQUCRwGeuygEvF_HXQ], unassigned_info[[reason=INDEX_CREATED], at[2016-06-06T02:24:15.014Z]]), reason [after recovery from store],shard-started ([mars2-2016.06.06][1], node[IfN3KNH_SEu0nz8eUGCamA], [P], v[1], s[INITIALIZING], a[id=ZjfT926vTDGYoj1_ohKsOA], unassigned_info[[reason=INDEX_CREATED], at[2016-06-06T02:24:15.014Z]]), reason [after recovery from store],shard-started ([al030-2016.05.27][4], node[IfN3KNH_SEu0nz8eUGCamA], [P], v[11], s[INITIALIZING], a[id=B5C3PRJyT7KBL8OjyRk_zQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-06-06T02:22:38.427Z]]), reason [after recovery from store]] took 1.6m above the warn threshold of 30s [2016-06-06 10:33:37,858][WARN][transport] [Skullfire] Received response for a request that has timed out, sent [30989ms] ago, timed out [15360ms] ago, action [cluster:monitor/nodes/stats[n]], node [{Skullfire}{IfN3KNH_SEu0nz8eUGCamA}{172.16.2.32}{BL043.xwjrfw.cn/172.16.2.32:9300}], id [22296] [2016-06-06 10:34:05,985][DEBUG][action.admin.indices.mapping.put] [Skullfire] failed to put mappings on indices [[mars1-2016.06.05]], type [tomcat] ProcessClusterEventTimeoutException[failed to process cluster event (put-mapping [tomcat]) within 30s] at org.elasticsearch.cluster.service.InternalClusterService$2$1.run(InternalClusterService.java:349) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [2016-06-06 10:34:05,986][DEBUG][action.bulk] [Skullfire] [mars1-2016.06.05][1] failed to execute bulk item (index) index {[mars1-2016.06.05][tomcat][AVUjhuGSAZCFCIZWhp22], source[{"@timestamp":"2016-06-05T16:04:27.798Z","beat":{"hostname":"MARS1","name":"MARS1"},"count":1,"fields":null,"input_type":"log","message":"127.0.0.1 - - [06/Jun/2016:00:00:00 +0800] \"GET /alfresco/service/api/solr/aclchangesets?fromTime=1461132274624\u0026toTime=1461135874624\u0026maxResults=2000 HTTP/1.1\" 200 128","offset":1787,"source":"/app/tomcat/logs/localhost_access_log2016-06-06.txt","type":"tomcat"}]} ProcessClusterEventTimeoutException[failed to process cluster event (put-mapping [tomcat]) within 30s] at org.elasticsearch.cluster.service.InternalClusterService$2$1.run(InternalClusterService.java:349) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [2016-06-06 10:34:05,989][DEBUG][action.admin.indices.mapping.put] [Skullfire] failed to put mappings on indices [[mars1-2016.06.05]], type [tomcat] ProcessClusterEventTimeoutException[failed to process cluster event (put-mapping [tomcat]) within 30s] at org.elasticsearch.cluster.service.InternalClusterService$2$1.run(InternalClusterService.java:349) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)` ---
**Author:** ![X-Mars](https://avatars.discourse-cdn.com/v4/letter/x/ea5d25/32.png) [@X-Mars](https://discuss.elastic.co/u/X-Mars)\ **Post date:** [June 6, 2016, 2:52am UTC](https://discuss.elastic.co/t/failed-to-execute-bulk-item-index-index/51890/2 "2016-06-06T02:52:35Z")
`[2016-06-06 10:51:20,230][WARN][rest.suppressed] /_stats Params: {} java.lang.OutOfMemoryError: Java heap space at java.lang.AbstractStringBuilder.(AbstractStringBuilder.java:68) at java.lang.StringBuilder.(StringBuilder.java:89) at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.(TransportBroadcastByNodeAction.java:277) at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:222) at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:79) at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:137) at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:85) at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:58) at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359) at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:52) at org.elasticsearch.rest.BaseRestHandler$HeadersAndContextCopyClient.doExecute(BaseRestHandler.java:83) at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359) at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1226) at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.stats(AbstractClient.java:1546) at org.elasticsearch.rest.action.admin.indices.stats.RestIndicesStatsAction.handleRequest(RestIndicesStatsAction.java:102) at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:54) at org.elasticsearch.rest.RestController.executeHandler(RestController.java:205) at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:166) at org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:128) at org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:86) at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:449) at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:61) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.messageReceived(HttpPipeliningHandler.java:60) at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)` ---
**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\ **Post date:** [June 6, 2016, 3:46am UTC](https://discuss.elastic.co/t/failed-to-execute-bulk-item-index-index/51890/3 "2016-06-06T03:46:04Z")
> [@X-Mars](#): > > `java.lang.OutOfMemoryError: Java heap space` You should start by addressing this. What heap size are you running with? ---
**Author:** ![X-Mars](https://avatars.discourse-cdn.com/v4/letter/x/ea5d25/32.png) [@X-Mars](https://discuss.elastic.co/u/X-Mars)\ **Post date:** [June 6, 2016, 4:01am UTC](https://discuss.elastic.co/t/failed-to-execute-bulk-item-index-index/51890/4 "2016-06-06T04:01:08Z")
> [@X-Mars](#): > > java.lang.OutOfMemoryError: Java heap space sorry i don't know Where am I to edit it ---
**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\ **Post date:** [June 6, 2016, 4:06am UTC](https://discuss.elastic.co/t/failed-to-execute-bulk-item-index-index/51890/5 "2016-06-06T04:06:59Z")
> [@X-Mars](#): > > sorry > i don't know > Where am I to edit it You could start with the [docs on setting the heap](https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html) and ask for clarification on anything that is puzzling? ---
**Author:** ![X-Mars](https://avatars.discourse-cdn.com/v4/letter/x/ea5d25/32.png) [@X-Mars](https://discuss.elastic.co/u/X-Mars)\ **Post date:** [June 6, 2016, 4:15am UTC](https://discuss.elastic.co/t/failed-to-execute-bulk-item-index-index/51890/7 "2016-06-06T04:15:11Z")
when i start elasticsearch ,i use "/etc/init.d/elasticsearch start" , can i put "-Xmx10g -Xms10g" in to the "/etc/init.d/elasticsearch". thx ---
**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\ **Post date:** [June 6, 2016, 4:32am UTC](https://discuss.elastic.co/t/failed-to-execute-bulk-item-index-index/51890/8 "2016-06-06T04:32:56Z")
What version of Elasticsearch are you on? The stack trace above looks like it comes from 2.3.x, is that correct? ---
**Author:** ![X-Mars](https://avatars.discourse-cdn.com/v4/letter/x/ea5d25/32.png) [@X-Mars](https://discuss.elastic.co/u/X-Mars)\ **Post date:** [June 6, 2016, 4:42am UTC](https://discuss.elastic.co/t/failed-to-execute-bulk-item-index-index/51890/9 "2016-06-06T04:42:27Z")
> [@jasontedor](#): > > What version of Elasticsearch are you on? The stack trace above looks like it comes from 2.3.x, is that correct? Version: 2.3.3, Build: 218bdf1/2016-05-17T15:40:04Z, JVM: 1.8.0\_92 my machine with 4GB RAM i have set "export ES\_HEAP\_SIZE=2g" ，but i got a new error： `[2016-06-06 12:33:19,135][WARN][bootstrap] unable to install syscall filter: seccomp unavailable: requires kernel 3.5+ with CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER compiled in [2016-06-06 12:33:19,302][INFO][node] [Kukulcan] version[2.3.3], pid[26966], build[218bdf1/2016-05-17T15:40:04Z] [2016-06-06 12:33:19,303][INFO][node] [Kukulcan] initializing ... [2016-06-06 12:33:19,940][INFO][plugins] [Kukulcan] modules [reindex, lang-expression, lang-groovy], plugins [head], sites [head] [2016-06-06 12:33:19,961][INFO][env] [Kukulcan] using [1] data paths, mounts [[/app (/dev/vdb1)]], net usable_space [31.7gb], net total_space [49.2gb], spins? [possibly], types [ext4] [2016-06-06 12:33:19,961][INFO][env] [Kukulcan] heap size [1.9gb], compressed ordinary object pointers [true] [2016-06-06 12:33:23,119][INFO][node] [Kukulcan] initialized [2016-06-06 12:33:23,120][INFO][node] [Kukulcan] starting ... [2016-06-06 12:33:23,200][INFO][transport] [Kukulcan] publish_address {mars1.test.com/172.16.2.32:9300}, bound_addresses {172.16.2.32:9300} [2016-06-06 12:33:23,205][INFO][discovery] [Kukulcan] elasticsearch/mGCU0BAxS0qkDt4MVipaKg [2016-06-06 12:33:26,248][INFO][cluster.service] [Kukulcan] new_master {Kukulcan}{mGCU0BAxS0qkDt4MVipaKg}{172.16.2.32}{mars1.test.com/172.16.2.32:9300}, reason: zen-disco-join(elected_as_master, [0] joins received) [2016-06-06 12:33:26,287][INFO][http] [Kukulcan] publish_address {mars1.test.com/172.16.2.32:9200}, bound_addresses {172.16.2.32:9200} [2016-06-06 12:33:26,287][INFO][node] [Kukulcan] started [2016-06-06 12:33:29,830][INFO][gateway] [Kukulcan] recovered [562] indices into cluster_state [2016-06-06 12:34:01,869][INFO][node] [Kukulcan] stopping ... [2016-06-06 12:34:02,236][WARN][netty.channel.DefaultChannelPipeline] An exception was thrown by an exception handler. java.util.concurrent.RejectedExecutionException: Worker has already been shutdown at org.jboss.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:120) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72) at org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:56) at org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36) at org.jboss.netty.channel.socket.nio.AbstractNioChannelSink.execute(AbstractNioChannelSink.java:34) at org.jboss.netty.channel.DefaultChannelPipeline.execute(DefaultChannelPipeline.java:636) at org.jboss.netty.channel.Channels.fireExceptionCaughtLater(Channels.java:496) at org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:46) at org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:781) at org.jboss.netty.channel.Channels.write(Channels.java:725) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784) at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.handleDownstream(HttpPipeliningHandler.java:87) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) at org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:146) at org.elasticsearch.rest.action.support.RestResponseListener.processResponse(RestResponseListener.java:43) at org.elasticsearch.rest.action.support.RestActionListener.onResponse(RestActionListener.java:49) at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:89) at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:85) at org.elasticsearch.action.bulk.TransportBulkAction$2.finishHim(TransportBulkAction.java:356) ` ---
**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\ **Post date:** [June 6, 2016, 10:48am UTC](https://discuss.elastic.co/t/failed-to-execute-bulk-item-index-index/51890/10 "2016-06-06T10:48:25Z")
Three's no error there. There's a warning about the seccomp filters not being able to be installed because your kernel does not support them, and then it appears that you stopped the node intentionally. Note that the seccomp filters not being able to be installed is not an error, Elasticsearch will operate just fine without them, but your instance will be less secure. ---
**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\ **Post date:** [June 6, 2016, 10:50am UTC](https://discuss.elastic.co/t/failed-to-execute-bulk-item-index-index/51890/11 "2016-06-06T10:50:46Z")
> [@X-Mars](#): > > `[2016-06-06 12:33:29,830][INFO][gateway] [Kukulcan] recovered [562] indices into cluster_state` One additional comment. That is a _lot_ of indices for what appears to be a single-node cluster. That's very likely why you were running into heap pressure with the defaults. ---
**Author:** ![X-Mars](https://avatars.discourse-cdn.com/v4/letter/x/ea5d25/32.png) [@X-Mars](https://discuss.elastic.co/u/X-Mars)\ **Post date:** [June 8, 2016, 5:07am UTC](https://discuss.elastic.co/t/failed-to-execute-bulk-item-index-index/51890/12 "2016-06-08T05:07:45Z")
thk you very much I will be in the details of the investigation it's can closed ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:45pm UTC](https://discuss.elastic.co/t/failed-to-execute-bulk-item-index-index/51890/13 "2017-07-05T22:45:26Z")