# Logstash only using 2 core out of 4 core server

**URL:** <https://discuss.elastic.co/t/logstash-only-using-2-core-out-of-4-core-server/164640>\
**Category:** Logstash\
**Created:** [January 17, 2019, 1:41pm UTC](https://discuss.elastic.co/t/logstash-only-using-2-core-out-of-4-core-server/164640 "2019-01-17T13:41:45Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![varun1992](https://avatars.discourse-cdn.com/v4/letter/v/3da27b/32.png) [@varun1992](https://discuss.elastic.co/u/varun1992)\
**Post date:** [January 17, 2019, 1:41pm UTC](https://discuss.elastic.co/t/logstash-only-using-2-core-out-of-4-core-server/164640/1 "2019-01-17T13:41:45Z")

</div>

Logstash v6.4.3  
workers 8  
heap size 5gb

Logstash only using 2 core out of 4 core system.

 ![Capture](https://us1.discourse-cdn.com/elastic/original/3X/4/d/4d35235b6eba1f85c8a6caffbfca9f3d2b3c30d4.png)

logstash config file is

```
input {
  beats {
    port => 9092
  }
}

#---------------------OUTPUT SECTION----------------------------------------
output {
  elasticsearch {
    hosts => ["10.0.0.12:5044", "10.0.0.16:5044", "10.0.0.14:5044"]
    manage_template => false
    validate_after_inactivity => 10000
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }

   stdout { 
    codec => rubydebug
  }
}

```

How to balance cpu core utilization ?

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [January 17, 2019, 1:58pm UTC](https://discuss.elastic.co/t/logstash-only-using-2-core-out-of-4-core-server/164640/2 "2019-01-17T13:58:43Z")

</div>

Reading the [readme on this library](https://github.com/OpenHFT/Java-Thread-Affinity), it is **not** so easy to set thread CPU affinity and I don't know how Windows works with thread affinity.

---

<div class="post-metadata">

**Author:** ![varun1992](https://avatars.discourse-cdn.com/v4/letter/v/3da27b/32.png) [@varun1992](https://discuss.elastic.co/u/varun1992)\
**Post date:** [January 17, 2019, 2:24pm UTC](https://discuss.elastic.co/t/logstash-only-using-2-core-out-of-4-core-server/164640/3 "2019-01-17T14:24:19Z")

</div>

what is the relation with logstash and affinity ?

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [January 17, 2019, 2:42pm UTC](https://discuss.elastic.co/t/logstash-only-using-2-core-out-of-4-core-server/164640/4 "2019-01-17T14:42:10Z")

</div>

Logstash uses JRuby and runs on the Java JVM. The JVM delegates mapping of threads to cores to the OS.

---

<div class="post-metadata">

**Author:** ![varun1992](https://avatars.discourse-cdn.com/v4/letter/v/3da27b/32.png) [@varun1992](https://discuss.elastic.co/u/varun1992)\
**Post date:** [January 17, 2019, 2:48pm UTC](https://discuss.elastic.co/t/logstash-only-using-2-core-out-of-4-core-server/164640/5 "2019-01-17T14:48:53Z")

</div>

So how do i solve this issue ?

i am using windows server 2012 R2.

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [January 17, 2019, 2:59pm UTC](https://discuss.elastic.co/t/logstash-only-using-2-core-out-of-4-core-server/164640/6 "2019-01-17T14:59:54Z")

</div>

We can't at the Logstash level. Maybe Windows is holding back two CPUs for other things - like being a server.

---

<div class="post-metadata">

**Author:** ![varun1992](https://avatars.discourse-cdn.com/v4/letter/v/3da27b/32.png) [@varun1992](https://discuss.elastic.co/u/varun1992)\
**Post date:** [January 17, 2019, 3:01pm UTC](https://discuss.elastic.co/t/logstash-only-using-2-core-out-of-4-core-server/164640/7 "2019-01-17T15:01:14Z")

</div>

but other cores are less than 5% utilization

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 17, 2019, 3:10pm UTC](https://discuss.elastic.co/t/logstash-only-using-2-core-out-of-4-core-server/164640/8 "2019-01-17T15:10:18Z")

</div>

Do you really have no filters at all? Just one input and one output?

---

<div class="post-metadata">

**Author:** ![varun1992](https://avatars.discourse-cdn.com/v4/letter/v/3da27b/32.png) [@varun1992](https://discuss.elastic.co/u/varun1992)\
**Post date:** [January 17, 2019, 3:21pm UTC](https://discuss.elastic.co/t/logstash-only-using-2-core-out-of-4-core-server/164640/9 "2019-01-17T15:21:17Z")

</div>

yes i am filtering beat data, i want all the data from beat

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 17, 2019, 3:30pm UTC](https://discuss.elastic.co/t/logstash-only-using-2-core-out-of-4-core-server/164640/10 "2019-01-17T15:30:16Z")

</div>

I believe both inputs and outputs are single threaded. If you have one input and one output then that only requires two threads, so it will only use two CPUs.

---

<div class="post-metadata">

**Author:** ![varun1992](https://avatars.discourse-cdn.com/v4/letter/v/3da27b/32.png) [@varun1992](https://discuss.elastic.co/u/varun1992)\
**Post date:** [January 17, 2019, 3:38pm UTC](https://discuss.elastic.co/t/logstash-only-using-2-core-out-of-4-core-server/164640/11 "2019-01-17T15:38:09Z")

</div>

i am getting this error in logstash sometimes. any idea ?

```
[2019-01-17T09:12:06,095][INFO][org.logstash.beats.BeatsHandler] [local: 10.0.0.7:9092, remote: 70.99.118.61:50455] Handling exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 69
[2019-01-17T09:12:06,095][WARN][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 69
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.18.Final.jar:4.1.18.Final]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192]
Caused by: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 69
	at org.logstash.beats.BeatsParser.decode(BeatsParser.java:92) ~[logstash-input-beats-5.1.6.jar:?]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	... 8 more
[2019-01-17T09:12:06,095][WARN][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 69
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at 
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.18.Final.jar:4.1.18.Final]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192]
Caused by: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 69
	at org.logstash.beats.BeatsParser.decode(BeatsParser.java:92) ~[logstash-input-beats-5.1.6.jar:?]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	... 8 more
[2019-01-17T09:12:06,111][INFO][org.logstash.beats.BeatsHandler] [local: 10.0.0.7:9092, remote: 70.99.118.61:50455] Handling exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 84
[2019-01-17T09:12:06,111][INFO][org.logstash.beats.BeatsHandler] [local: 10.0.0.7:9092, remote: 70.99.118.61:54238] Handling exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 69
[2019-01-17T09:12:06,126][WARN][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 69
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.18.Final.jar:4.1.18.Final]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192]
Caused by: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 69
	at org.logstash.beats.BeatsParser.decode(BeatsParser.java:92) ~[logstash-input-beats-5.1.6.jar:?]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-all
	... 8 more
[2019-01-17T09:12:06,126][INFO][org.logstash.beats.BeatsHandler] [local: 10.0.0.7:9092, remote: 70.99.118.61:50454] Handling exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 84
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 17, 2019, 3:48pm UTC](https://discuss.elastic.co/t/logstash-only-using-2-core-out-of-4-core-server/164640/12 "2019-01-17T15:48:04Z")

</div>

Can you show us the beat and logstash configurations? Are there errors in the beat logs that correspond to the Invalid Frame errors (e.g. connection reset by peer)?

---

<div class="post-metadata">

**Author:** ![varun1992](https://avatars.discourse-cdn.com/v4/letter/v/3da27b/32.png) [@varun1992](https://discuss.elastic.co/u/varun1992)\
**Post date:** [January 17, 2019, 4:08pm UTC](https://discuss.elastic.co/t/logstash-only-using-2-core-out-of-4-core-server/164640/13 "2019-01-17T16:08:01Z")

</div>

which beat ? i will show filebeat is that ok ? logstash conf as above

```
#=========================== Filebeat inputs =============================

filebeat.inputs:

- type: log
  document_type: iis

  enabled: false

#============================= Filebeat modules ===============================

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml

  reload.enabled: false

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 3
  index.number_of_replicas: 1
  
setup.template.name: "filebeat-%{[beat.version]}-*"
setup.template.fields: "fields.yml"
setup.template.pattern: "filebeat-%{[beat.version]}-*"
setup.template.overwrite: true

#----------------------------- Logstash output --------------------------------
output.logstash:
  hosts: ["23.111.116.13:9092"]

#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
# hosts: ["localhost:5044"]
# index: "filebeat-%{[beat.version]}-%{+yyyy.MM.dd}"

#================================ Logging =====================================

logging.level: info
logging.to_files: true
logging.files:
  path: ${path.config}/logs
  name: filebeat
  keepfiles: 10
  permissions: 0644
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 17, 2019, 4:18pm UTC](https://discuss.elastic.co/t/logstash-only-using-2-core-out-of-4-core-server/164640/14 "2019-01-17T16:18:58Z")

</div>

> [@varun1992](#):
>
> [org.logstash.beats.BeatsHandler] [local: 10.0.0.7:9092, remote: 70.99.118.61:50455]

I find it impossible to reconcile the IP addresses in that logfile line with the configurations that you are posting. Your beats input appears to be listening on port 9092.

---

<div class="post-metadata">

**Author:** ![varun1992](https://avatars.discourse-cdn.com/v4/letter/v/3da27b/32.png) [@varun1992](https://discuss.elastic.co/u/varun1992)\
**Post date:** [January 17, 2019, 4:38pm UTC](https://discuss.elastic.co/t/logstash-only-using-2-core-out-of-4-core-server/164640/15 "2019-01-17T16:38:23Z")

</div>

sorry, I edited config, I copied wrong cinf before. my logstash port is 9092. elastic port is 5044

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 17, 2019, 4:46pm UTC](https://discuss.elastic.co/t/logstash-only-using-2-core-out-of-4-core-server/164640/16 "2019-01-17T16:46:28Z")

</div>

Are you able to log in to 70.99.118.61 and use lsof to find what process has port 50455 open?

---

<div class="post-metadata">

**Author:** ![varun1992](https://avatars.discourse-cdn.com/v4/letter/v/3da27b/32.png) [@varun1992](https://discuss.elastic.co/u/varun1992)\
**Post date:** [January 17, 2019, 4:53pm UTC](https://discuss.elastic.co/t/logstash-only-using-2-core-out-of-4-core-server/164640/17 "2019-01-17T16:53:24Z")

</div>

Is there any other way? That public ip used by more than 5 pc. so don't know which pc to check

---

<div class="post-metadata">

**Author:** ![varun1992](https://avatars.discourse-cdn.com/v4/letter/v/3da27b/32.png) [@varun1992](https://discuss.elastic.co/u/varun1992)\
**Post date:** [January 19, 2019, 1:30pm UTC](https://discuss.elastic.co/t/logstash-only-using-2-core-out-of-4-core-server/164640/18 "2019-01-19T13:30:38Z")

</div>

Any other way to solve issue ?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 19, 2019, 1:36pm UTC](https://discuss.elastic.co/t/logstash-only-using-2-core-out-of-4-core-server/164640/19 "2019-01-19T13:36:45Z")

</div>

Well I would guess there is something configured to expect 9092 to be elasticsearch, so you could switch your configuration around to have elasticsearch on 9092 and use 5044 for beats. This would have the added advantage of being consistent with the expectations of the rest of the planet.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 16, 2019, 1:36pm UTC](https://discuss.elastic.co/t/logstash-only-using-2-core-out-of-4-core-server/164640/20 "2019-02-16T13:36:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.