# Timeout - Pipeline blocked

**URL:** <https://discuss.elastic.co/t/timeout-pipeline-blocked/97519>\
**Category:** Elasticsearch\
**Created:** [August 18, 2017, 6:46am UTC](https://discuss.elastic.co/t/timeout-pipeline-blocked/97519 "2017-08-18T06:46:37Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![g4nj4wizard](https://avatars.discourse-cdn.com/v4/letter/g/edb3f5/32.png) [@g4nj4wizard](https://discuss.elastic.co/u/g4nj4wizard)\
**Post date:** [August 18, 2017, 6:46am UTC](https://discuss.elastic.co/t/timeout-pipeline-blocked/97519/1 "2017-08-18T06:46:37Z")

</div>

Hi.

I'm new to this project and i am experiencing a big issue, since i added all the Clients (15 Server) for monitoring.  
Since i added these server, the response Time grew massively and now it doesn't even respond. I am not which settings need to be changed. I've red that sometimes stdout = rubydebug causes high cpu usage. Uncommenting it didnt help. I've followed this guide for ubuntu [http://www.itzgeek.com/how-tos/linux/ubuntu-how-tos/setup-elk-stack-ubuntu-16-04.html](http://www.itzgeek.com/how-tos/linux/ubuntu-how-tos/setup-elk-stack-ubuntu-16-04.html)

I'm running elasticsearch/logstash in a VM on our Hyper-V to monitor our Windows Server.  
VM Settings = 2 Virtual Cores + 6GB RAM  
What wonders me is, that Linux is not able to split the work over both CPUs. Either CPU 1 is constantly on 100% and CPU2 on 0-2% or vice versa.  
I already added more RAM. Tried first 16GB (100% usage) and also 32 (also 100% usage) I dont want to go higher, because there have to be other ways optimizing it.  
What can i do to make the server handle the incomming traffic? Java is using all RAM for its own processes.  
Most suspicious Log entries:

```
[winlogbeat-2017.08.16][[winlogbeat-2017.08.16][2]] CreateFailedEngineException[Create failed for [wineventlog#AV3wgGvwYERlAy38-ALy]]; nested: OutOfMemoryError[Java heap space];
        at org.elasticsearch.index.engine.InternalEngine.create(InternalEngine.java:351)
        at org.elasticsearch.index.shard.IndexShard.create(IndexShard.java:549)
        at org.elasticsearch.index.engine.Engine$Create.execute(Engine.java:810)
        at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:236)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:327)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:120)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:68)
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:657)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:287)
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:279)
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
        at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Java heap space

ProcessClusterEventTimeoutException[failed to process cluster event (put-mapping [wineventlog]) within 30s]
        at org.elasticsearch.cluster.service.InternalClusterService$2$1.run(InternalClusterService.java:361)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
[2017-08-17 16:26:27,147][DEBUG][action.admin.indices.mapping.put] [Sagittarius] failed to put mappings on indices [[winlogbeat-2016.11.04]], type [wineventlog]
ProcessClusterEventTimeoutException[failed to process cluster event (put-mapping [wineventlog]) within 30s]
        at org.elasticsearch.cluster.service.InternalClusterService$2$1.run(InternalClusterService.java:361)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

{:timestamp=>"2017-08-18T08:42:09.455000+0200", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
{:timestamp=>"2017-08-18T08:42:50.026000+0200", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Beats input", :exception=>LogStash::Inputs::Beats::InsertingToQueueTakeTooLong, :level=>:warn}
{:timestamp=>"2017-08-18T08:42:50.026000+0200", :message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker, :level=>:warn}

[2017-08-18 08:43:17,216][ERROR][cluster.action.shard] [Argus] unexpected failure during [shard-started ([winlogbeat-2017.08.16][4], node[fit1KJnrQS6QI4PftbbS7g], [P], v[11], s[INITIALIZING], a[id=GKXVDgJaRdeGAMsZn5w-Ag], unassigned_info[[reason=CLUSTER_RECOVERED], at[2017-08-18T06:02:05.660Z]]), reason [master {Argus}{fit1KJnrQS6QI4PftbbS7g}{127.0.0.1}{127.0.0.1:9300} marked shard as initializing, but shard state is [POST_RECOVERY], mark shard as started]]
java.lang.OutOfMemoryError: Java heap space
```

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [August 18, 2017, 7:34am UTC](https://discuss.elastic.co/t/timeout-pipeline-blocked/97519/2 "2017-08-18T07:34:19Z")

</div>

what heap size do you have configured for Elasticsearch?

---

<div class="post-metadata">

**Author:** ![g4nj4wizard](https://avatars.discourse-cdn.com/v4/letter/g/edb3f5/32.png) [@g4nj4wizard](https://discuss.elastic.co/u/g4nj4wizard)\
**Post date:** [August 18, 2017, 8:00am UTC](https://discuss.elastic.co/t/timeout-pipeline-blocked/97519/3 "2017-08-18T08:00:39Z")

</div>

I havent change that because i dont find a setting in the elasticsearch.config. where do i set that for ubuntu 16.04?  
I am usually operating on windows server. A hint where to set that option would be helpful!

---

<div class="post-metadata">

**Author:** ![g4nj4wizard](https://avatars.discourse-cdn.com/v4/letter/g/edb3f5/32.png) [@g4nj4wizard](https://discuss.elastic.co/u/g4nj4wizard)\
**Post date:** [August 18, 2017, 10:14am UTC](https://discuss.elastic.co/t/timeout-pipeline-blocked/97519/4 "2017-08-18T10:14:44Z")

</div>

found it out myself... its working now.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 15, 2017, 10:14am UTC](https://discuss.elastic.co/t/timeout-pipeline-blocked/97519/5 "2017-09-15T10:14:53Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.