Field doesn't create when field value is empty

Elastic Stack Logstash
1

hi elastic team,

i would like to ask one thing.

when i check my log in https://grokdebug.herokuapp.com/, i got expect result

my test log is

2018-03-07 10:13:56,256 (null) 24 WARN : TEST.Controllers.PaymentController : zlbjkexpwcb5pai1zrsexpgp ::: TEST : {"version":"7.2","request_timestamp":"2018-03-07 10:13:36","sid":"764764000000962","ref_id":"","desc":"TESTTEST"} : 5932958ms

{
"log_timestamp": [
[
"2018-03-07 10:13:56,256"
]
],
"threadno": [
[
"24"
]
],
"log_level": [
[
"WARN"
]
],
"session_id": [
[
"zlbjkexpwcb5pai1zrsexpgp"
]
],
"sid": [
[
"764764000000962"
]
],
"ref_id": [
[
"""
]
]
}

but when i ship my log to logstash, ref_id field is missing when its value is empty.

how do i get ref_id field if it's value is empty or not????

i'm running logstash 6.1.1 and i use grok filter

2

What does your configuration look like?

3

sorry i forgot to paste it.

here it is

input {
stdin {
type => "true"
}
}

filter {
grok {
match => { "message" => [ "^(?<log_timestamp>\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{1,3}).?(?\d+).?%{WORD:log_level}.?TEST.?\s+:\s+(?<session_id>.?)\s{0,}:::.?"request_timestamp":"(?.?)","sid":"(?.?)","ref_id":"(?<ref_id>.?)","desc":"(?.?)""]}
add_field => { "api_code" => "test" }
}

if ![ref_id] {
mutate { add_field => { "ref_id" => "-" } }
}

}

output {
stdout {
codec => rubydebug
}
}

right now i add ![ref_id]. this way ok for now. but i have no idea it is best practice or not.
plz, advice me. thank you :slight_smile:

4

Don't parse JSON with a grok filter. Use a json filter.

5

Does JSON filter support for writing patterns??

6

No, the JSON filter parses JSON. Use a grok filter to extract the timestamp, loglevel, and what else to separate fields, including the JSON string. Then use the json filter to parse the field with the JSON.

7

i see thanks @magnusbaeck

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.