Can't remove fields in logstash 5.2.2

Elastic Stack Logstash
1

Hi all,
I'm trying to remove the "remove" field and the default fields (host, @version, etc...) but can't seems to achieve this goal.

Here is my conf file:
input {
file {
path => ["E:/TestLog.txt"]
start_position => "beginning"
}
}
filter {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{WORD:vendor}: %{TIMESTAMP_ISO8601:localEventTime} - %{WORD:remove} - [%{IP:userIP}] %{USERNAME:username}(%{WORD:group})[] - %{GREEDYDATA:vpnMsg}" }
}
}
output {
stdout {
codec => rubydebug
}
}

and from some reason, when trying to add:
mutate {remove_field => ['remove']}

I get an error grokparsefailure

Can somebody help me?
Thanks.

2

and from some reason, when trying to add:
mutate {remove_field => ['remove']}

I get an error grokparsefailure

Sorry, I don't buy that story. Instead of doing that just don't capture the field in the first place, i.e. replace %{WORD:remove} with %{WORD}.

I'm not sure @version can be deleted.

3

I've changed the WORD:remove to WORD as you suggested.
When running the conf now I receive the following error:
"path" => "E:/TestLog.txt",
"@timestamp" => 2017-03-14T19:51:04.449Z,
"@version" => "1",
"host" => "Orr-PC",
"message" => "\r",
"tags" => [
[0] "_grokparsefailure"

why? is something wrong with my conf file?
This is the updated conf file:

input {
file {
path => ["E:/TestLog.txt"]
start_position => "beginning"
}
}
filter {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{WORD:vendor}: %{TIMESTAMP_ISO8601:localEventTime} - %{WORD} - [%{IP:userIP}] %{USERNAME:username}(%{WORD:group}) - %{GREEDYDATA:vpnMsg}" }
}
}
output {
stdout {
codec => rubydebug
}
}

and this is the log file i'm trying to parse:

2017-02-14T13:39:33+02:00 PulseSecure: 2017-02-14 13:39:33 - ive - [10.16.4.225] dpnini(Users) - Testing Password realm restrictions failed for dpnini/Users

4

btw - the parsing is working on grok debugger and gives the expected output:
%{TIMESTAMP_ISO8601:timestamp} %{WORD:vendor}: %{TIMESTAMP_ISO8601:localEventTime} - %{WORD} - [%{IP:userIP}] %{USERNAME:username}(%{WORD:group})[] - %{GREEDYDATA:vpnMsg}

5

why? is something wrong with my conf file?

It looks like the input contains an empty line that you're trying to parse. You can use a conditional to drop empty lines.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.