Logstash grok matching entire filebeat message instead of message inside

Elastic Stack Logstash
1

I have setup Filebeat -> Kafka -> Logstash -> Elasticseatch system.

I have filebeat output like;

{"@timestamp":"2016-11-04T05:16:16.108Z","beat":{"hostname":"localhost","name":"localhost","version":"5.0.0"},"fields":{"logtype":"logfile"},"input_type":"log","message":"\u003cOct 31, 2016 6:37:40:678 AM\u003e \u003cdataa\u003e \u003cdatab\u003e \u003cdatac\u003e \u003datad\u003e \u003cdatae\u003e \u003cdataf\u003e \u003cgatag\u003e\n \u003cdatah\u003e","offset":273,"source":"/logfiles/logfile.log","type":"logfile"}

I have setup logstash grok to match this message like;

grok {
	match => { "message" => "%{GREEDYDATA:messageDataA}" }
}

But it is pushing entire beat data like @timestamp, hostname, etc (the whole filebeat output line I have mentioned above) to messageDataA field. But I want to push only the message part inside filebeat message to be pushed. How can I do this? I am using 5.0 GA of Filebeat and Logstash.

2

It sounds like you're missing a codec => json setting in your kafka input. Please show your configuration and the output of a stdout { codec => rubydebug } output.

Using grok to copy data from one field to another is inefficient and unnecessary. Use a mutate filter to rename the field instead (or copy the field value with add_field if you really want to keep both fields).

1 Like
3

codec => json solved the issue. Thanks a lot :slight_smile: