How to use logstash filter

Elastic Stack Logstash
1

Hi

As I'm new to Logstash I just wanted to ask you some filter questions, probably you guys will be able to help me on this and get me starting...

These are the logs I'm sending to Logstash (example):

   "message" => "[12/12/16 4:13:19:608 CET] 0000003c LdapRegistryI A   SECJ0419I: The user registry is currently connected to the LDAP server ldap://tam-uat.bc:389.",
  "@version" => "1",
"@timestamp" => "2016-12-12T07:54:04.154Z",
      "type" => "log",
"input_type" => "log",
    "fields" => nil,
    "offset" => 13542,
     "count" => 1,
      "beat" => {
    "hostname" => "el2081.bc",
        "name" => "el2081.bc"
},
    "source" => "/opt/websphere/logs/poma1/pom_bpel_c1_n1_m1/SystemOut.log",
      "host" => "el2081.bc",
      "tags" => [
    [0] "beats_input_codec_plain_applied"
]

}
{
"message" => "[12/12/16 6:03:28:922 CET] 0000003c LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://tam-uat.bc:389.",
"@version" => "1",
"@timestamp" => "2016-12-12T07:54:04.154Z",
"offset" => 13690,
"type" => "log",
"input_type" => "log",
"count" => 1,
"fields" => nil,
"beat" => {
"hostname" => "el2081.bc",
"name" => "el2081.bc"
},
"source" => "/opt/websphere/logs/poma1/pom_bpel_c1_n1_m1/SystemOut.log",
"host" => "el2081.bc",
"tags" => [
[0] "beats_input_codec_plain_applied"
]
}
{
"message" => "[12/12/16 7:53:15:398 CET] 0000003c LdapRegistryI A SECJ0419I: The user registry is currently connected to the LDAP server ldap://tam-uat.bc:389.",
"@version" => "1",
"@timestamp" => "2016-12-12T07:54:04.154Z",
"beat" => {
"hostname" => "el2081.bc",
"name" => "el2081.bc"
},
"offset" => 13838,
"type" => "log",
"input_type" => "log",
"fields" => nil,
"source" => "/opt/websphere/logs/poma1/pom_bpel_c1_n1_m1/SystemOut.log",
"count" => 1,
"host" => "el2081.bc",
"tags" => [
[0] "beats_input_codec_plain_applied"

Now, how do I make my filter to show only the host, message and timestamp? Can somebody help me out please?

Kind regards

Johnny

2

You mean you want to permanently delete all fields except @timestamp, host, and message?

3

Yes indeed.

4

Maybe try this one ?
https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html
(you can name fields which you want to remove)

Or maybe you can try create new event with fields you want
https://www.elastic.co/guide/en/logstash/current/plugins-filters-clone.html

5

The prune filter allows you to express exactly that; keep a specified set of fields but delete the rest.

6

I tried with this filter now:

beats{
port=>5055
}
}
filter {
mutate {
remove => ["offset","count","type","input_type","fields"]

}
}
output {
stdout { codec => rubydebug }

    #   elasticsearch {
    #       hosts => ["el2597.bc:9200", "el2598.bc:9200"]
    #       document_type => "tuxsrv"
    #       index => "logstash-tuxsrv-%{+YYYY.MM.dd}"
    #   }
    }

Now I don't have any outcome anymore...? Is this normal?

[monadm@el2599.bc::ES-TEST]/mwo/home/monadm/elk/logstash/bin # ./logstash -f /mwo/home/monadm/mwo_logstash_fb/bpm.conf
You are using a deprecated config setting "remove" set in mutate. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"remove", :plugin=><LogStash::Filters::Mutate remove=>["offset", "count", "type", "input_type", "fields"]>, :level=>:warn}
Settings: Default pipeline workers: 2
Pipeline main started

7

As the message says, remove is deprecated. Use remove_field instead. Not sure it'll help, but let's rule it out first.

8

I used it like this and it works now, thanks.

filter {
mutate {
remove_field => ["offset","count","type","input_type","fields","tags"]

}
}

One more question related to filtering, which command in the filter do I use to show messages with only the correct pattern? As an example, I have different patterns like:

Threads hung
ORACLE not available
No suitable messaging engine is available
JMS exceptions
Test Flush
A messaging engine communication error occured
A internal messaging engine error occured, (look at Oracle level)
Transaction Timeouts
Connection to resource not available
exception on commit for datasource $$UN_FLEX_1
Failed to connect to database on $$ID_HOST
Out Of Memory
Invalid user/passwd (CWF)
CORBA.NO_PERMISSION detected
Connection is closed Error (DSRA9110E) detected
CORBA.NO_IMPLEMENT No available target (backend)
CORBA.NO_RESPONSE (time outs)
Unable to send via POST (time outs)
Error in storeproc (restart component)
Request flow runtime instance is unavailable
AMF-Message

Can you maybe advise me? Thanks.

9

You can e.g. use conditionals to selectively apply a drop filter.

https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html

10

Thanks a lot already for your support.

Just a small question related, this is my outcome now:

{
"message" => " A",
"@version" => "1",
"@timestamp" => "2016-12-21T08:35:40.461Z",
"input_type" => "log",
"host" => "el2083.bc",
"mwo_domain" => "From /opt/websphere/logs/mopa1/amf/mop/Root.log.2016-12-21"
}

and I would like to have for the mwo_domain field that the outcome is like:

"mwo_domain" => "From bpmmopa1

Can you help me a bit as I'm not used to reg.ex.

Thank you.

Johnny

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.