# How to use logstash filter

**URL:** <https://discuss.elastic.co/t/how-to-use-logstash-filter/68671>\
**Category:** Logstash\
**Created:** [December 12, 2016, 8:06am UTC](https://discuss.elastic.co/t/how-to-use-logstash-filter/68671 "2016-12-12T08:06:45Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![JohnnyS](https://avatars.discourse-cdn.com/v4/letter/j/6de8d8/32.png) [@JohnnyS](https://discuss.elastic.co/u/JohnnyS)\
**Post date:** [December 12, 2016, 8:06am UTC](https://discuss.elastic.co/t/how-to-use-logstash-filter/68671/1 "2016-12-12T08:06:45Z")

</div>

Hi

As I'm new to Logstash I just wanted to ask you some filter questions, probably you guys will be able to help me on this and get me starting...

These are the logs I'm sending to Logstash (example):

```
   "message" => "[12/12/16 4:13:19:608 CET] 0000003c LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://tam-uat.bc:389.",
  "@version" => "1",
"@timestamp" => "2016-12-12T07:54:04.154Z",
      "type" => "log",
"input_type" => "log",
    "fields" => nil,
    "offset" => 13542,
     "count" => 1,
      "beat" => {
    "hostname" => "el2081.bc",
        "name" => "el2081.bc"
},
    "source" => "/opt/websphere/logs/poma1/pom_bpel_c1_n1_m1/SystemOut.log",
      "host" => "el2081.bc",
      "tags" => [
    [0] "beats_input_codec_plain_applied"
]

```

}  
{  
"message" =\> "[12/12/16 6:03:28:922 CET] 0000003c LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://tam-uat.bc:389.",  
"@version" =\> "1",  
"@timestamp" =\> "2016-12-12T07:54:04.154Z",  
"offset" =\> 13690,  
"type" =\> "log",  
"input\_type" =\> "log",  
"count" =\> 1,  
"fields" =\> nil,  
"beat" =\> {  
"hostname" =\> "el2081.bc",  
"name" =\> "el2081.bc"  
},  
"source" =\> "/opt/websphere/logs/poma1/pom\_bpel\_c1\_n1\_m1/SystemOut.log",  
"host" =\> "el2081.bc",  
"tags" =\> [  
[0] "beats\_input\_codec\_plain\_applied"  
]  
}  
{  
"message" =\> "[12/12/16 7:53:15:398 CET] 0000003c LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://tam-uat.bc:389.",  
"@version" =\> "1",  
"@timestamp" =\> "2016-12-12T07:54:04.154Z",  
"beat" =\> {  
"hostname" =\> "el2081.bc",  
"name" =\> "el2081.bc"  
},  
"offset" =\> 13838,  
"type" =\> "log",  
"input\_type" =\> "log",  
"fields" =\> nil,  
"source" =\> "/opt/websphere/logs/poma1/pom\_bpel\_c1\_n1\_m1/SystemOut.log",  
"count" =\> 1,  
"host" =\> "el2081.bc",  
"tags" =\> [  
[0] "beats\_input\_codec\_plain\_applied"

Now, how do I make my filter to show only the host, message and timestamp? Can somebody help me out please?

Kind regards

Johnny

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 12, 2016, 8:13am UTC](https://discuss.elastic.co/t/how-to-use-logstash-filter/68671/2 "2016-12-12T08:13:27Z")

</div>

You mean you want to permanently delete all fields except `@timestamp`, `host`, and `message`?

---

<div class="post-metadata">

**Author:** ![JohnnyS](https://avatars.discourse-cdn.com/v4/letter/j/6de8d8/32.png) [@JohnnyS](https://discuss.elastic.co/u/JohnnyS)\
**Post date:** [December 12, 2016, 8:15am UTC](https://discuss.elastic.co/t/how-to-use-logstash-filter/68671/3 "2016-12-12T08:15:27Z")

</div>

Yes indeed.

---

<div class="post-metadata">

**Author:** ![Martin\_Duris](https://avatars.discourse-cdn.com/v4/letter/m/b77776/32.png) [@Martin\_Duris](https://discuss.elastic.co/u/Martin_Duris)\
**Post date:** [December 12, 2016, 8:20am UTC](https://discuss.elastic.co/t/how-to-use-logstash-filter/68671/4 "2016-12-12T08:20:28Z")

</div>

Maybe try this one ?  
[https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html)  
(you can name fields which you want to remove)

Or maybe you can try create new event with fields you want  
[https://www.elastic.co/guide/en/logstash/current/plugins-filters-clone.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-clone.html)

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 12, 2016, 8:33am UTC](https://discuss.elastic.co/t/how-to-use-logstash-filter/68671/5 "2016-12-12T08:33:09Z")

</div>

The prune filter allows you to express exactly that; keep a specified set of fields but delete the rest.

---

<div class="post-metadata">

**Author:** ![JohnnyS](https://avatars.discourse-cdn.com/v4/letter/j/6de8d8/32.png) [@JohnnyS](https://discuss.elastic.co/u/JohnnyS)\
**Post date:** [December 12, 2016, 8:34am UTC](https://discuss.elastic.co/t/how-to-use-logstash-filter/68671/6 "2016-12-12T08:34:20Z")

</div>

I tried with this filter now:

beats{  
port=\>5055  
}  
}  
filter {  
mutate {  
remove =\> ["offset","count","type","input\_type","fields"]

}  
}  
output {  
stdout { codec =\> rubydebug }

```
    # elasticsearch {
    # hosts => ["el2597.bc:9200", "el2598.bc:9200"]
    # document_type => "tuxsrv"
    # index => "logstash-tuxsrv-%{+YYYY.MM.dd}"
    # }
    }

```

Now I don't have any outcome anymore...? Is this normal?

[monadm@el2599.bc::ES-TEST]/mwo/home/monadm/elk/logstash/bin # ./logstash -f /mwo/home/monadm/mwo\_logstash\_fb/bpm.conf  
You are using a deprecated config setting "remove" set in mutate. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=\>"remove", :plugin=\>\<LogStash::Filters::Mutate remove=\>["offset", "count", "type", "input\_type", "fields"]\>, :level=\>:warn}  
Settings: Default pipeline workers: 2  
Pipeline main started

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 12, 2016, 8:57am UTC](https://discuss.elastic.co/t/how-to-use-logstash-filter/68671/7 "2016-12-12T08:57:25Z")

</div>

As the message says, `remove` is deprecated. Use `remove_field` instead. Not sure it'll help, but let's rule it out first.

---

<div class="post-metadata">

**Author:** ![JohnnyS](https://avatars.discourse-cdn.com/v4/letter/j/6de8d8/32.png) [@JohnnyS](https://discuss.elastic.co/u/JohnnyS)\
**Post date:** [December 12, 2016, 9:07am UTC](https://discuss.elastic.co/t/how-to-use-logstash-filter/68671/8 "2016-12-12T09:07:44Z")

</div>

I used it like this and it works now, thanks.

filter {  
mutate {  
remove\_field =\> ["offset","count","type","input\_type","fields","tags"]

}  
}

One more question related to filtering, which command in the filter do I use to show messages with only the correct pattern? As an example, I have different patterns like:

Threads hung  
ORACLE not available  
No suitable messaging engine is available  
JMS exceptions  
Test Flush  
A messaging engine communication error occured  
A internal messaging engine error occured, (look at Oracle level)  
Transaction Timeouts  
Connection to resource not available  
exception on commit for datasource $$UN\_FLEX\_1  
Failed to connect to database on $$ID\_HOST  
Out Of Memory  
Invalid user/passwd (CWF)  
CORBA.NO\_PERMISSION detected  
Connection is closed Error (DSRA9110E) detected  
CORBA.NO\_IMPLEMENT No available target (backend)  
CORBA.NO\_RESPONSE (time outs)  
Unable to send via POST (time outs)  
Error in storeproc (restart component)  
Request flow runtime instance is unavailable  
AMF-Message

Can you maybe advise me? Thanks.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 12, 2016, 9:48am UTC](https://discuss.elastic.co/t/how-to-use-logstash-filter/68671/9 "2016-12-12T09:48:37Z")

</div>

You can e.g. use conditionals to selectively apply a drop filter.

[https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html)

---

<div class="post-metadata">

**Author:** ![JohnnyS](https://avatars.discourse-cdn.com/v4/letter/j/6de8d8/32.png) [@JohnnyS](https://discuss.elastic.co/u/JohnnyS)\
**Post date:** [December 21, 2016, 8:41am UTC](https://discuss.elastic.co/t/how-to-use-logstash-filter/68671/10 "2016-12-21T08:41:02Z")

</div>

Thanks a lot already for your support.

Just a small question related, this is my outcome now:

{  
"message" =\> " A",  
"@version" =\> "1",  
"@timestamp" =\> "2016-12-21T08:35:40.461Z",  
"input\_type" =\> "log",  
"host" =\> "el2083.bc",  
"mwo\_domain" =\> "From /opt/websphere/logs/mopa1/amf/mop/Root.log.2016-12-21"  
}

and I would like to have for the mwo\_domain field that the outcome is like:

"mwo\_domain" =\> "From bpmmopa1

Can you help me a bit as I'm not used to reg.ex.

Thank you.

Johnny

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 18, 2017, 8:41am UTC](https://discuss.elastic.co/t/how-to-use-logstash-filter/68671/11 "2017-01-18T08:41:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.