# Filebeat logs have these error messages in the log file
**URL:** https://discuss.elastic.co/t/filebeat-logs-have-these-error-messages-in-the-log-file/147631
**Category:** Beats
**Tags:** filebeat
**Created:** [September 6, 2018, 8:28pm UTC](https://discuss.elastic.co/t/filebeat-logs-have-these-error-messages-in-the-log-file/147631 "2018-09-06T20:28:13Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@Sindhu\_Reddy](https://discuss.elastic.co/u/Sindhu_Reddy)
#### Post date: [September 6, 2018, 8:28pm UTC](https://discuss.elastic.co/t/filebeat-logs-have-these-error-messages-in-the-log-file/147631/1 "2018-09-06T20:28:13Z")
I am getting the below error pretty often, not sure why this error occurs. Any information regarding this problem is appreciated.The logstashcluster has healthy instances and it is up and running.
Filebeat Logs :
| 2018-09-06T06:33:47.067-0600 | ERROR | pipeline/output.go:92 | Failed to publish events: client is not connected |
| --- | --- | --- | --- |
| 2018-09-06T06:33:47.093-0600 | ERROR | logstash/async.go:235 | Failed to publish events caused by: EOF |
| 2018-09-06T06:33:47.131-0600 | ERROR | logstash/async.go:235 | Failed to publish events caused by: client is not connected |
| 2018-09-06T06:33:48.168-0600 | ERROR | pipeline/output.go:92 | Failed to publish events: client is not connected |
| 2018-09-06T06:33:48.199-0600 | ERROR | logstash/async.go:235 | Failed to publish events caused by: EOF |
| 2018-09-06T06:33:48.235-0600 | ERROR | logstash/async.go:235 | Failed to publish events caused by: client is not connected |
| 2018-09-06T06:33:49.269-0600 | ERROR | pipeline/output.go:92 | Failed to publish events: client is not connected |
| 2018-09-06T06:33:49.299-0600 | ERROR | logstash/async.go:235 | Failed to publish events caused by: EOF |
| 2018-09-06T06:33:49.341-0600 | ERROR | logstash/async.go:235 | Failed to publish events caused by: client is not connected |
Filebeat config :
filebeat.prospectors:
- type: log
enabled: true
paths:
- D:\inetpub\splunklog\*\*
filebeat.config.modules:
path: ${path.config}/modules.d/\*.yml
reload.enabled: false
setup.template.settings:
index.number\_of\_shards: 3
setup.kibana:
output.logstash:
hosts: ["example.int:5044"]
fields\_under\_root: true
fields:
"@stackId": "abc\_def"
logging.level: info
logging.to\_files: true
logging.files:
path: C:\filebeat\filebeat-6.2.4\logs
name: filebeat
keepfiles: 7
permissions: 0644
Logstash Config:
input {
beats {
port =\> 5044
}
}
filter {
if [source] =~ /._avro.json._/ {
mutate {
add\_field =\> { "@type" =\> "BUSINESS\_EVENT"}
}
} else if [source] =~ /._[bB]usiness._/ {
json {
source =\> "message"
}
mutate {
add\_field =\> { "@type" =\> "BUSINESS"}
remove\_field =\> ["message"]
}
} else if [source] =~ /._validation._/ {
json {
source =\> "message"
}
mutate {
add\_field =\> { "@type" =\> "VALIDATION"}
remove\_field =\> ["message"]
}
} else if [source] =~ /._[fF]orensic._/ {
json {
source =\> "message"
}
mutate {
add\_field =\> { "@type" =\> "FORENSIC"}
remove\_field =\> ["message"]
}
} else if [source] =~ /._[mM]etrics._/ {
json {
source =\> "message"
}
mutate {
add\_field =\> { "@type" =\> "METRIC"}
remove\_field =\> ["message"]
}
} else if [source] =~ /._[eE]rror._/ {
json {
source =\> "message"
}
mutate {
add\_field =\> { "@type" =\> "ERROR"}
remove\_field =\> ["message"]
}
} else {
mutate {
add\_field =\> { "@type" =\> "UNKNOWN"}
}
}
}
output
{
kinesis {
stream\_name =\> "firehydrant"
region =\> "us-east-1"
randomized\_partition\_key =\> true
aggregation\_enabled =\> true
max\_pending\_records =\> 10000
}
}
logstash version is 5.6.3
filebeat version is 6.2.4
---
### Author:  [@kvch](https://discuss.elastic.co/u/kvch)
#### Post date: [September 6, 2018, 10:20pm UTC](https://discuss.elastic.co/t/filebeat-logs-have-these-error-messages-in-the-log-file/147631/2 "2018-09-06T22:20:48Z")
What do you see in the logs of Logstash?
---
### Author:  [@Sindhu\_Reddy](https://discuss.elastic.co/u/Sindhu_Reddy)
#### Post date: [September 6, 2018, 10:23pm UTC](https://discuss.elastic.co/t/filebeat-logs-have-these-error-messages-in-the-log-file/147631/3 "2018-09-06T22:23:12Z")
[2018-09-06T15:58:16,032][INFO][org.logstash.beats.BeatsHandler] [local: 10.155.34.52:5044, remote: 10.155.56.178:14290] Handling exception: failed to allocate 16777216 byte(s) of direct memory (used: 1023410176, max: 1037959168)
[2018-09-06T15:58:16,049][INFO][org.logstash.beats.BeatsHandler] [local: 10.155.34.52:5044, remote: 10.155.56.178:14290] Handling exception: failed to allocate 16777216 byte(s) of direct memory (used: 1023410176, max: 1037959168)
[2018-09-06T15:58:16,049][INFO][org.logstash.beats.BeatsHandler] [local: 10.155.34.52:5044, remote: 10.155.56.178:14290] Handling exception: failed to allocate 16777216 byte(s) of direct memory (used: 1023410176, max: 1037959168)[2018-09-06T15:58:19,078][WARN][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 1023410176, max: 1037959168)
at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:640) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:594) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:764) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:740) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:244) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:214) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:146) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:324) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:176) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:137) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:125) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.18.Final.jar:4.1.18.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0\_181]
---
### Author:  [@kvch](https://discuss.elastic.co/u/kvch)
#### Post date: [September 7, 2018, 12:10pm UTC](https://discuss.elastic.co/t/filebeat-logs-have-these-error-messages-in-the-log-file/147631/4 "2018-09-07T12:10:12Z")
Your LS instance seems to be out of memory, thus it cannot receive more events from Filebeat. Try increasing the memory for LS or decrease the load on that LS instance.
---
### Author:  [@Sindhu\_Reddy](https://discuss.elastic.co/u/Sindhu_Reddy)
#### Post date: [September 7, 2018, 5:03pm UTC](https://discuss.elastic.co/t/filebeat-logs-have-these-error-messages-in-the-log-file/147631/5 "2018-09-07T17:03:59Z")
I changed my instance types of LS , almost doubled the number of instances i had so that each of them gets less load but i am still getting this error.
[2018-09-07T10:50:43,701][WARN][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 1023485784, max: 1037959168)
at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:640) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:594) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:764) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:740) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:244) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:214) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:146) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:324) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:176) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:137) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:125) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.18.Final.jar:4.1.18.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0\_181]
[2018-09-07T10:50:43,701][WARN][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 1023485784, max: 1037959168)
at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:640) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:594) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:764) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:740) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:244) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:214) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:146) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:324) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:176) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:137) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:125) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.18.Final.jar:4.1.18.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0\_181]
---
### Author:  [@kvch](https://discuss.elastic.co/u/kvch)
#### Post date: [September 8, 2018, 12:39am UTC](https://discuss.elastic.co/t/filebeat-logs-have-these-error-messages-in-the-log-file/147631/6 "2018-09-08T00:39:55Z")
Possible ways to decrease the load on LS:
- Decrease the value of `pipelining` in `filebeat.yml`
- Decrease the option `bulk_max_size` in `filebeat.yml`
Unfortunately, it's a known problem. There is an open issue: [https://github.com/logstash-plugins/logstash-input-beats/issues/309](https://github.com/logstash-plugins/logstash-input-beats/issues/309)
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 6, 2018, 12:40am UTC](https://discuss.elastic.co/t/filebeat-logs-have-these-error-messages-in-the-log-file/147631/7 "2018-10-06T00:40:04Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.