# Missing a lot of logs sent via filebeats

**URL:** <https://discuss.elastic.co/t/missing-a-lot-of-logs-sent-via-filebeats/61185>\
**Category:** Beats\
**Created:** [September 21, 2016, 9:41pm UTC](https://discuss.elastic.co/t/missing-a-lot-of-logs-sent-via-filebeats/61185 "2016-09-21T21:41:35Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Maxwell\_Flanders](https://avatars.discourse-cdn.com/v4/letter/m/eb8c5e/32.png) [@Maxwell\_Flanders](https://discuss.elastic.co/u/Maxwell_Flanders)\
**Post date:** [September 21, 2016, 9:41pm UTC](https://discuss.elastic.co/t/missing-a-lot-of-logs-sent-via-filebeats/61185/1 "2016-09-21T21:41:35Z")

</div>

We seem to be losing a lot of logs from just a couple of servers that we collect and send via filebeats. This is our basic configuration:

```
filebeat:
  prospectors: []
  registry_file: "/var/lib/filebeat/registry"
  config_dir: "/etc/filebeat/conf.d"
output:
  logstash:
    enabled: true
    hosts:
    - 10.252.250.30:5044
    - 10.252.250.53:5044
    - 10.252.250.59:5044
    - 10.252.250.60:5044
    loadbalance: true

logging:
    level: info
    to_syslog: true

```

and our prospectors file looks like:

```
filebeat:
  prospectors:
  - paths:
    - "/opt/bro/logs/current/conn.log"
    document_type: bro
    fields:
      type: conn
  - paths:
    - "/opt/bro/logs/current/files.log"
    document_type: bro
    fields:
      type: files

```

What happens is when we start filebeats, it collects logs and send them through logstash to kibana and we see logs for about a minute, and then they stop, and we begin to receive errors like this one:

```
Sep 21 21:06:05 nat-gateway-manager-uscen-a-c001-n001 /usr/bin/filebeat[21443]: single.go:76: Error publishing events (retrying): EOF
Sep 21 21:06:05 nat-gateway-manager-uscen-a-c001-n001 /usr/bin/filebeat[21443]: single.go:152: send fail
Sep 21 21:06:05 nat-gateway-manager-uscen-a-c001-n001 /usr/bin/filebeat[21443]: single.go:159: backoff retry: 1m0s
Sep 21 21:07:15 nat-gateway-manager-uscen-a-c001-n001 /usr/bin/filebeat[21443]: single.go:76: Error publishing events (retrying): EOF
Sep 21 21:07:15 nat-gateway-manager-uscen-a-c001-n001 /usr/bin/filebeat[21443]: single.go:152: send fail
Sep 21 21:07:15 nat-gateway-manager-uscen-a-c001-n001 /usr/bin/filebeat[21443]: single.go:159: backoff retry: 1m0s
Sep 21 21:08:25 nat-gateway-manager-uscen-a-c001-n001 /usr/bin/filebeat[21443]: single.go:76: Error publishing events (retrying): EOF
Sep 21 21:08:25 nat-gateway-manager-uscen-a-c001-n001 /usr/bin/filebeat[21443]: single.go:152: send fail
Sep 21 21:08:25 nat-gateway-manager-uscen-a-c001-n001 /usr/bin/filebeat[21443]: single.go:159: backoff retry: 1m0s
Sep 21 21:09:35 nat-gateway-manager-uscen-a-c001-n001 /usr/bin/filebeat[21443]: single.go:76: Error publishing events (retrying): EOF
Sep 21 21:09:35 nat-gateway-manager-uscen-a-c001-n001 /usr/bin/filebeat[21443]: single.go:152: send fail
Sep 21 21:09:35 nat-gateway-manager-uscen-a-c001-n001 /usr/bin/filebeat[21443]: single.go:159: backoff retry: 1m0s
Sep 21 21:10:35 nat-gateway-manager-uscen-a-c001-n001 /usr/bin/filebeat[21443]: single.go:126: Connecting error publishing events (retrying): dial tcp IPADDRESS:5044: getsockopt: connection refused

```

After that point, we only see sparse, intermittent logs, even though there are many more on the server.

Our logstash input for beats looks like this:

```
beats {
    port => 5044
    congestion_threshold => 25
}

```

We collect a ton of logs via filebeats (syslogs from hundreds of servers) and send them through the exact same logstash servers, and all of them seem to make it into elasticsearch just fine. However, on just a couple of our servers where we collect a high volume of logs, we encounter this issue where we get some logs and then they stop flowing, and then we only see random bits of logs intermittently. Have you guys seen an issue like this before?? How can we configure filebeats to manage particularly high loads??

Another note: during troubleshooting, we dropped the logstash instances that it forwards logs to from 4 to a single instance. When we do that, what we seem to find is that it no longer MISSES any logs, but chugs through the backlog incredibly slowly. If we add more servers to the loadbalancing scheme, it becomes less reliable.

Any help appreciated!!!

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [September 22, 2016, 8:02am UTC](https://discuss.elastic.co/t/missing-a-lot-of-logs-sent-via-filebeats/61185/2 "2016-09-22T08:02:31Z")

</div>

Which version of filebeat and logstash are you using? Do you have any logs from the logstash side?

@steffens Can you have a look here?

---

<div class="post-metadata">

**Author:** ![il.bert](https://avatars.discourse-cdn.com/v4/letter/i/ac91a4/32.png) [@il.bert](https://discuss.elastic.co/u/il.bert)\
**Post date:** [September 22, 2016, 8:07am UTC](https://discuss.elastic.co/t/missing-a-lot-of-logs-sent-via-filebeats/61185/3 "2016-09-22T08:07:42Z")

</div>

up! I am facing the same problem, changed the congestion\_threshold too and also reduced the number of loggers machine (thus drastically reduced the throughput)...still the same problem

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [September 22, 2016, 8:13am UTC](https://discuss.elastic.co/t/missing-a-lot-of-logs-sent-via-filebeats/61185/4 "2016-09-22T08:13:55Z")

</div>

@il.bert Could you also share filebeat version, logstash version and both configs? In addition it would be good to know which version of the beats-input-plugin is used.

---

<div class="post-metadata">

**Author:** ![il.bert](https://avatars.discourse-cdn.com/v4/letter/i/ac91a4/32.png) [@il.bert](https://discuss.elastic.co/u/il.bert)\
**Post date:** [September 22, 2016, 8:23am UTC](https://discuss.elastic.co/t/missing-a-lot-of-logs-sent-via-filebeats/61185/5 "2016-09-22T08:23:23Z")

</div>

LS I used both 2.4 and 5.0  
FB is 5alpha6

FB is the following

for each file I have this path conf

```
  fields_under_root: true
  ignore_older: 10m
  close_inactive: 2m
  clean_inactive: 15m

#========================= Filebeat global options ============================

filebeat.spool_size: 10000
filebeat.idle_timeout: 10s

#----------------------------- Logstash output --------------------------------
output.logstash:
  hosts: ["10.246.85.242:5044", "10.246.85.243:5044"]
  template.name: "filebeat"
  loadbalance: true
  bulk_max_size: 1000  
  template.path: "filebeat.template.json"
  template.overwrite: false

```

in LS i tried LOTS OF configuration 2, 3 and 4 instances of LS with different conf  
the one that get me better results is the following, different configurations allowed me to pass a very smaller throughput (10% of the load) with this one I have almost 75%

```
path.data: /var/lib/logstash
pipeline.workers: 24
pipeline.output.workers: 1
pipeline.batch.size: 15000
path.config: /etc/logstash/conf.d
config.reload.automatic: true
config.reload.interval: 30
log.level: verbose
path.log: /var/log/logstash/logstash.log

```

with

```
-Xms12g
-Xmx12g

```

I have opened also this [FileBeat EOF Error](https://discuss.elastic.co/t/filebeat-eof-error/60882/5) and this [FileBeat file is falling under ignore older](https://discuss.elastic.co/t/filebeat-file-is-falling-under-ignore-older/61210) for other errors i am facing

hope you will help asap as my POC for my manager should be ready by next week

---

<div class="post-metadata">

**Author:** ![Maxwell\_Flanders](https://avatars.discourse-cdn.com/v4/letter/m/eb8c5e/32.png) [@Maxwell\_Flanders](https://discuss.elastic.co/u/Maxwell_Flanders)\
**Post date:** [September 22, 2016, 1:33pm UTC](https://discuss.elastic.co/t/missing-a-lot-of-logs-sent-via-filebeats/61185/6 "2016-09-22T13:33:16Z")

</div>

I am using:

`filebeat: 1.2.3`  
`beats-input-plugin: 2.0.3`, later tested on 2.2.9 with no fix.  
`logstash: 2.1.0`

I don't see any logstash errors. I believe before we changed `congestion_threshold => 25` in logstash-beats-input, we had some

`{:timestamp=>"2016-09-21T21:07:15.641000+0000", :message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::CircuitBreaker::HalfOpenBreaker, :level=>:warn}`

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [September 23, 2016, 6:37am UTC](https://discuss.elastic.co/t/missing-a-lot-of-logs-sent-via-filebeats/61185/7 "2016-09-23T06:37:06Z")

</div>

It seems like this thread could be related? [FileBeat EOF Error](https://discuss.elastic.co/t/filebeat-eof-error/60882/17)

@il.bert As you are on both threads, because you can share some insights?

---

<div class="post-metadata">

**Author:** ![Maxwell\_Flanders](https://avatars.discourse-cdn.com/v4/letter/m/eb8c5e/32.png) [@Maxwell\_Flanders](https://discuss.elastic.co/u/Maxwell_Flanders)\
**Post date:** [September 23, 2016, 4:30pm UTC](https://discuss.elastic.co/t/missing-a-lot-of-logs-sent-via-filebeats/61185/8 "2016-09-23T16:30:01Z")

</div>

Haha whoops, I posted my conclusion to the wrong thread. Here it is again:

Hey guys, I found the root of our issue was actually on a misconfigured filter on the logstash end that was clogging the filter workers queue. I guess that manifested itself eventually in filebeats's logs.

---

<div class="post-metadata">

**Author:** ![il.bert](https://avatars.discourse-cdn.com/v4/letter/i/ac91a4/32.png) [@il.bert](https://discuss.elastic.co/u/il.bert)\
**Post date:** [October 7, 2016, 9:56am UTC](https://discuss.elastic.co/t/missing-a-lot-of-logs-sent-via-filebeats/61185/9 "2016-10-07T09:56:55Z")

</div>

For me there seems to be a different problem!

I posted a workaround here, but it is not a solution [FileBeat EOF Error](https://discuss.elastic.co/t/filebeat-eof-error/60882)  
If you could read it, any help will be really appreciated!

> [@FileBeat EOF Error](https://discuss.elastic.co/t/filebeat-eof-error/60882/25):
>
> Hi, sadly our test did not proceed well, none of the above workaround worked 100%, we were still getting that error. As we had the opportunity, we moved to a larger cluster: 3 HUGE machines. We expected one of them to be able to take the whole load but we used 3 for test. It happened again. EOF Thus it could not be a problem of fully loaded machine so we moved to changing other parameters. We found out that removing load\_balance in filebeat solved the problem, or at least it seems to. Again…

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 12, 2016, 9:42pm UTC](https://discuss.elastic.co/t/missing-a-lot-of-logs-sent-via-filebeats/61185/10 "2016-10-12T21:42:00Z")

</div>

This topic was automatically closed after 21 days. New replies are no longer allowed.