Logstash output file has additional fields

Elastic Stack Logstash
1

Hi,

My config is as below
filter { grok { match => { "message" => "%{DATE_EU:mydate}%{SPACE}%{HOUR:myhour}.%{MINUTE:myminute}.%{SECOND:mysecond}%{SPACE}%{GREEDYDATA:realdata}"} }

and my input is as below
16.04.2015 00.02.31:125 RECV MO ID:routernode6@nbpagpapp6:520920746 CID:C50836 DA:95999-E164A OA:6281230252381-E164N EX:T NI:smpp-mo N/A UD:OFF\ MD

and my output is as below
{"message":"16.04.2015 00.02.31:125 RECV MO ID:routernode6@nbpagpapp6:520920746 CID:C50836 DA:95999-E164A OA:6281230252381-E164N EX:T NI:smpp-mo N/A UD:OFF\\ MD","@version":"1","@timestamp":"2016-05-12T11:10:49.936Z","path":"/home/test/project/logstash_sample_data/api_trace_input.log","host":"vijayvm.mav.com","mydate":"16.04.2015","myhour":"00","myminute":"02","mysecond":"31:125","realdata":"RECV MO ID:routernode6@nbpagpapp6:520920746 CID:C50836 DA:95999-E164A OA:6281230252381-E164N EX:T NI:smpp-mo N/A UD:OFF\\ MD"}

My question is based on my grok filter, I have to get only below as output.
{"mydate":"16.04.2015","myhour":"00","myminute":"02","mysecond":"31:125","realdata":"RECV MO ID:routernode6@nbpagpapp6:520920746 CID:C50836 DA:95999-E164A OA:6281230252381-E164N EX:T NI:smpp-mo N/A UD:OFF\\ MD"}}

But May I know why is it printing additional items like "message", "path" and "host" ?

2

The file input adds the host and path fields and the message field is a leftover from the original input that you grokked. If you don't want to have the fields you can remove them with the remove_field option of e.g. a mutate filter.

1 Like
3

thanks @magnusbaeck, I could remove it with mutate filter. I understood 'host' and 'path' are added by file input. But that 'message' doesnt seems to be the leftover of grok.

4

But that 'message' doesnt seems to be the leftover of grok.

Your grok filter parses the message field so you obviously have a message field. If you don't need that field after you've parsed it you have to remove it yourself.