Pattern Assistance

Elastic Stack Logstash
#1

Hi :slight_smile:
can you please help me understand the issue?
this is my log:

DEBUG 2017-02-21 17:49:13,431 "local Ip":172.33.82.82 "Algo":Extraction "Status":"Constructor" "Current":0 "Pending":0 "Done":0 "All":0 "TaskId":425

I tried the following:

input {
beats {
port => "5043"
}
}

filter {
grok {
match => [
"message",
"%{WORD} (?%{TIMESTAMP_ISO8601}+\d\d:\d\d) %{IP} %{GREEDYDATA:kvdata}"
]
}
kv {
field_split => " "
value_split => ":"
source => "kvdata"
remove_field => "kvdata"
}
}

output {
elasticsearch {
hosts => "localhost:9200"
}
stdout {}
}

not working...

can you please help?
thank you!

#2

The string you're trying to match prefixes the IP address with "local Ip:" but you're not including that literal string in your expression. If you're using the kv filter for the rest of the string why not deal with the "local Ip" field in the same way?

#3

ok I'll try, thanks!

#4

hi!
I did the following and it's still not working:

input {
beats {
port => "5043"
}
}

filter {
grok {
match => [
"message",
"%{WORD} (?%{TIMESTAMP_ISO8601}+\d\d:\d\d) %{IP} %{GREEDYDATA:kvdata}"
]
}
kv {
field_split => " "
value_split => ":"
source => "kvdata"
remove_field => "kvdata"
}
}

output {
elasticsearch {
hosts => "localhost:9200"
}
stdout {}
}

i'm getting the tag :

beats_input_codec_plain_applied, _grokparsefailure

in the Kibana.

can you please assist?
maybe it has something to do with the filebeat.template.json file?
what am i missing?...
thank you!

#5

I don't get it. Your grok expression is the same as last time so unless you input is different you'll get the same unwanted output.

#6

I've copied by mistake the old pattern. the new one is the same without the IP:

input {
beats {
port => "5043"
}
}
filter {
grok {
match => [
"message",
"%{WORD} (?%{TIMESTAMP_ISO8601}+\d\d:\d\d) %{GREEDYDATA:kvdata}"
]
}
kv {
field_split => " "
value_split => ":"
source => "kvdata"
remove_field => "kvdata"
}
}
output {
elasticsearch {
hosts => "localhost:9200"
}
stdout {}
}

still getting the same issue..

#7

Why can't you use this pattern:
%{WORD} %{TIMESTAMP_ISO8601} %{GREEDYDATA:kvdata}

#8

tried... didn't work..

#9

Be systematic. Try the shortest possible pattern, ^%{WORD}. Does that work? Yes? Then add the next token, i.e. %{WORD} %{TIMESTAMP_ISO8601} Does that work? Use the grok constructor web site to gradually build your expressions until you're comfortable doing it on your own.

#10

Are you sure? I tried the exact pattern and your log line in online grok debugger https://grokdebug.herokuapp.comhttps://grokdebug.herokuapp.com/ and it works.

#11

i'm sure... it's not working... tried also only with %{WORD}.
in the Logstash logs there's now : "string index out of range" and also in the Kibana _grokparsefailure.
i'm working with Docker if that changes anything...

#12

Perhaps you have another filter that's giving you the _grokparsefailure tag. Check all files in /etc/logstash/conf.d (if that's where you store your configuration files).

#13

that's the only file in /etc/logstash/conf.d..

#14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.