# Logs still being shipped with filebeat purged

**URL:** <https://discuss.elastic.co/t/logs-still-being-shipped-with-filebeat-purged/144601>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [August 16, 2018, 12:33am UTC](https://discuss.elastic.co/t/logs-still-being-shipped-with-filebeat-purged/144601 "2018-08-16T00:33:32Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ianjack](https://avatars.discourse-cdn.com/v4/letter/i/ecccb3/32.png) [@ianjack](https://discuss.elastic.co/u/ianjack)\
**Post date:** [August 16, 2018, 12:33am UTC](https://discuss.elastic.co/t/logs-still-being-shipped-with-filebeat-purged/144601/1 "2018-08-16T00:33:32Z")

</div>

I attempted to use "exclude\_lines:" and didn't have any success with filebeat v5.8.3

- input\_type: log  
paths:
  - /var/log/auth.log
  - /var/log/syslog
  - /var/log/ufw.log
  - /var/log/mail.log  
exclude\_files: [".gz$"]  
exclude\_lines: ['_.bdb\_equality\_candidates._']

This:

processors:  
drop\_event.when.regexp:  
system.auth.message: ['._bdb\_equality\_candidates._']

Log entries:

Aug 15 00:28:07 authv2 slapd[14027]: \<= bdb\_equality\_candidates: (carLicense) not indexed

I tried this in the filebeat.yml config file and the syslog module. Tried to use a regex (Yes, it was correct) in the module syslog config file. I didn't have any luck from the examples I saw in the forums. I abandoned it, attempted to simply exclude the syslog file, logs are still being shipped, turned off filebeat, logs are still being shipped. Purged it from the system, logs are being shipped. All the logs are up to date and lines match.

The registry is purged. Filebeat isn't on the system and there is no other system producing these log entries and yet they are still showing up.

What is forwarding these log entries? Is there a lock, queue or something pushing these files out? Filebeat has been off this system for about 24 hours. Syslog isn't set to to push these logs independent of filebeat.

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [August 16, 2018, 6:47am UTC](https://discuss.elastic.co/t/logs-still-being-shipped-with-filebeat-purged/144601/2 "2018-08-16T06:47:16Z")

</div>

What is the output of `which -a filebeat`?

---

<div class="post-metadata">

**Author:** ![ianjack](https://avatars.discourse-cdn.com/v4/letter/i/ecccb3/32.png) [@ianjack](https://discuss.elastic.co/u/ianjack)\
**Post date:** [August 17, 2018, 4:14pm UTC](https://discuss.elastic.co/t/logs-still-being-shipped-with-filebeat-purged/144601/3 "2018-08-17T16:14:30Z")

</div>

Nothing.

Apparently those entries were queued up in the logstash server. There were millions of lines still being added to the indices long after filebeat was off. Filebeat is very effective at grabbing and shipping those logs, logstash still had to process them. It took a long time to complete.

The exclude\_lines prolly do work but wasn't acting on lines already shipped to logstash due to the backlog of lines to proccess.

Well, that's embarrassing.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 14, 2018, 4:14pm UTC](https://discuss.elastic.co/t/logs-still-being-shipped-with-filebeat-purged/144601/4 "2018-09-14T16:14:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.