# Filebeat show unwanted logs

**URL:** <https://discuss.elastic.co/t/filebeat-show-unwanted-logs/95179>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [July 31, 2017, 11:56am UTC](https://discuss.elastic.co/t/filebeat-show-unwanted-logs/95179 "2017-07-31T11:56:43Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![abu.sayeed](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@abu.sayeed](https://discuss.elastic.co/u/abu.sayeed)\
**Post date:** [July 31, 2017, 11:56am UTC](https://discuss.elastic.co/t/filebeat-show-unwanted-logs/95179/1 "2017-07-31T11:56:43Z")

</div>

#cat /etc/filebeat/filebeat.yml

filebeat.modules:  
filebeat.prospectors:

- input\_type: log  
paths:

logging.to\_files: true  
logging.files:  
rotateeverybytes: 10485760 # = 10MB

keepfiles: 7

##########

Kibana discover show:

July 31st 2017, 16:54:28.274   
@timestamp:July 31st 2017, 16:54:28.27 offset: 10,067,175 @version:  
1 input\_type: beat.hostname:vNTDACLSWEBP005 [beat.name](http://beat.name):vNTDACLSWEBP005  
beat.version:5.4.3 host: vNTDACLSWEBP005 source: /home/local/user/nsms/logs/Logs/nsms-server.log message: 2017-07-31 16:54:26,865 : [INFO] http-nio-8080-exec-57 [c.n.s.g.SmsGatewayGPHttpGet:281] Setting cmp param value for smsGateway [GP:DAC:GET}  
[GP:DAC:GET} | SmsResult

But I want:  
July 31st 2017, 16:54:28.274

**host:** vNTDACLSWEBP005  
**source:** /home/local/user/nsms/logs/Logs/nsms-server.log  
**message:** 2017-07-31 16:54:26,865 : [INFO] http-nio-8080-exec-57 [c.n.s.g.SmsGatewayGPHttpGet:281] Setting cmp param value for smsGateway [GP:DAC:GET}  
[GP:DAC:GET} | SmsResult

How can remove unwanted logs. Like:  
@timestamp:July 31st 2017, 16:54:28.27  
offset: 10,067,175  
@version: 5.4.3  
beat.hostname:vNTDACLSWEBP005  
[beat.name](http://beat.name):vNTDACLSWEBP005  
beat.version:5.4.3

Please anybody help me.

Thanks all.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [July 31, 2017, 12:49pm UTC](https://discuss.elastic.co/t/filebeat-show-unwanted-logs/95179/2 "2017-07-31T12:49:45Z")

</div>

I don't think you can remove the @timestamp, it is a necessary field, and I'm not sure about the @version field either, but the beat fields and the offset can be removed using mutate in your filter pipeline on logstash.

```auto
filter {
    mutate {
       remove_field => ["beat", "offset"]
    }
}

```

---

<div class="post-metadata">

**Author:** ![abhiroyg](https://avatars.discourse-cdn.com/v4/letter/a/e9bcb4/32.png) [@abhiroyg](https://discuss.elastic.co/u/abhiroyg)\
**Post date:** [July 31, 2017, 2:48pm UTC](https://discuss.elastic.co/t/filebeat-show-unwanted-logs/95179/3 "2017-07-31T14:48:04Z")

</div>

Except @timestamp, you can remove any other field including @version.  
I tried removing @timestamp, but I got errors. So, I ended up keeping it.  
Please use the code suggested by @leandrojmp for removing fields.

---

<div class="post-metadata">

**Author:** ![abu.sayeed](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@abu.sayeed](https://discuss.elastic.co/u/abu.sayeed)\
**Post date:** [August 1, 2017, 5:57am UTC](https://discuss.elastic.co/t/filebeat-show-unwanted-logs/95179/4 "2017-08-01T05:57:10Z")

</div>

> [@abhiroyg](#):
>
> @version

no change after using mutate in my filter pipeline on logstash.

mutate {  
remove\_field =\> ["beat", "offset"]  
}

---

<div class="post-metadata">

**Author:** ![abhiroyg](https://avatars.discourse-cdn.com/v4/letter/a/e9bcb4/32.png) [@abhiroyg](https://discuss.elastic.co/u/abhiroyg)\
**Post date:** [August 1, 2017, 6:08am UTC](https://discuss.elastic.co/t/filebeat-show-unwanted-logs/95179/5 "2017-08-01T06:08:23Z")

</div>

Hi,

Please add the field name you want to remove to that list. Here, since you want to remove @version. Your mutate filter would look like this.

```
mutate {
    remove_field => ["@version", "beat", "offset"]
}

```

Hope it helps.

---

<div class="post-metadata">

**Author:** ![abu.sayeed](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@abu.sayeed](https://discuss.elastic.co/u/abu.sayeed)\
**Post date:** [August 1, 2017, 7:02am UTC](https://discuss.elastic.co/t/filebeat-show-unwanted-logs/95179/6 "2017-08-01T07:02:06Z")

</div>

Logstash-pipeline conf

filter {  
mutate {  
remove\_field =\> ["\_id", "\_type", "\_index", "\_score", "@version", "beat", "offset"]  
}  
}

@timestamp August 1st 2017, 12:56:54.893  
t **\_id** AV2clVn9SwgmMqjSZLw1  
t **\_index** filebeat-2017.08.01

# **\_score** -

t **\_type** log  
t host vNTDACLSWEBP005  
t **input\_type** log  
t message 2017-08-01 12:56:54,133 : [INFO] http-nio-8080-exec-7 [c.n.j.JdbcService:-1] Executing SP ACT\_sms with Action [UPDATE]  
t source /home/local/user/nsms/logs/Logs/nsms-server.log  
t **tags** beats\_input\_codec\_plain\_applied  
t **type** log

how can remove highlights log.  
thanks all

---

<div class="post-metadata">

**Author:** ![abhiroyg](https://avatars.discourse-cdn.com/v4/letter/a/e9bcb4/32.png) [@abhiroyg](https://discuss.elastic.co/u/abhiroyg)\
**Post date:** [August 1, 2017, 7:15am UTC](https://discuss.elastic.co/t/filebeat-show-unwanted-logs/95179/7 "2017-08-01T07:15:03Z")

</div>

Hi,

You can't remove **\_id** , **\_index** , **\_score** , **\_type**. They are required by elasticsearch. You can try reading elasticsearch documentation to know why.

You can remove **input\_type** , **type** , **tags** using the syntax I mentioned in my previous reply.

Generally tags can be helpful. So, if you want to disable that particular tag (_beats\_input\_codec\_plain\_applied_). Add `include_codec_tag => false` to your beats plugin.  
Example:

```
input {
    beats {
        include_codec_tag => false
        <other configuration>
    }
}

```

Hope it helps.

---

<div class="post-metadata">

**Author:** ![abu.sayeed](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@abu.sayeed](https://discuss.elastic.co/u/abu.sayeed)\
**Post date:** [August 2, 2017, 6:59am UTC](https://discuss.elastic.co/t/filebeat-show-unwanted-logs/95179/8 "2017-08-02T06:59:25Z")

</div>

It's works. Thanks all for helping me. I'm very very glad for your help.

---

<div class="post-metadata">

**Author:** ![abu.sayeed](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@abu.sayeed](https://discuss.elastic.co/u/abu.sayeed)\
**Post date:** [August 2, 2017, 7:11am UTC](https://discuss.elastic.co/t/filebeat-show-unwanted-logs/95179/9 "2017-08-02T07:11:29Z")

</div>

**How can split log in column permanently?**

Kibana Discovery logs format:

**host** :VM\_5 **source** :/home/user/user.log **message** :smsStatus : [SENT]

But I wish log format:

**host:**** source: ****message:**  
VM\_5 /home/user/user.log smsStatus : [SENT]

---

<div class="post-metadata">

**Author:** ![abhiroyg](https://avatars.discourse-cdn.com/v4/letter/a/e9bcb4/32.png) [@abhiroyg](https://discuss.elastic.co/u/abhiroyg)\
**Post date:** [August 2, 2017, 7:22am UTC](https://discuss.elastic.co/t/filebeat-show-unwanted-logs/95179/10 "2017-08-02T07:22:54Z")

</div>

Hi,

Can you please post this as a new question in Kibana ?

Thanks

---

<div class="post-metadata">

**Author:** ![abu.sayeed](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@abu.sayeed](https://discuss.elastic.co/u/abu.sayeed)\
**Post date:** [August 2, 2017, 7:36am UTC](https://discuss.elastic.co/t/filebeat-show-unwanted-logs/95179/11 "2017-08-02T07:36:10Z")

</div>

OK Thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 30, 2017, 7:36am UTC](https://discuss.elastic.co/t/filebeat-show-unwanted-logs/95179/12 "2017-08-30T07:36:24Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.