# Configure kv plugin to handle multi line messages **URL:** \ **Category:** Logstash\ **Created:** [January 24, 2019, 8:09am UTC](https://discuss.elastic.co/t/configure-kv-plugin-to-handle-multi-line-messages/165549 "2019-01-24T08:09:11Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![yasko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yasko/32/40151_2.png) [@yasko](https://discuss.elastic.co/u/yasko)\ **Post date:** [January 24, 2019, 8:09am UTC](https://discuss.elastic.co/t/configure-kv-plugin-to-handle-multi-line-messages/165549/1 "2019-01-24T08:09:11Z")
I'm collecting logs using filebeat and sending them to logstash for processing. The log format is: ``` timestamp: 2019-01-24T07:04:42.322Z, level: INFO, message: something happened ``` In logstash I've setup the kv plugin to create the fields and values. All works well until a multi line message is received. In filebeat I've configured the input to handle multi line messages - here's the config: ``` - type: log multiline: pattern: '^timestamp' negate: true match: after paths: - /mnt/logs/*.log ``` Filebeat sends the entire message and also logstash, but the kv plugin seems to handle it only until a new line is reached: ``` input { beats {...} } filter { kv { field_split_pattern => "\s*,\s" value_split => ":" } } output {...} ``` When I put the _message_ value in square brackets all lines of the message are included, the problem is that the message can have all kinds of brackets ([SPARQL queries](https://en.wikipedia.org/wiki/SPARQL)) and that breaks the message again to only a single line. So, is there a way I can configure the kv plugin to handle multiline messages? ---
**Author:** ![shrikantgulia](https://avatars.discourse-cdn.com/v4/letter/s/c68b51/32.png) [@shrikantgulia](https://discuss.elastic.co/u/shrikantgulia)\ **Post date:** [January 24, 2019, 9:52am UTC](https://discuss.elastic.co/t/configure-kv-plugin-to-handle-multi-line-messages/165549/2 "2019-01-24T09:52:31Z")
Hello @yasko, Can you please post some logs Best Regards Shrikant ---
**Author:** ![yasko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yasko/32/40151_2.png) [@yasko](https://discuss.elastic.co/u/yasko)\ **Post date:** [January 24, 2019, 10:15am UTC](https://discuss.elastic.co/t/configure-kv-plugin-to-handle-multi-line-messages/165549/3 "2019-01-24T10:15:51Z")
@shrikantgulia Sure Filebeat: ``` { "level": "debug", "timestamp": "2019-01-24T07:23:15.429Z", "logger": "publish", "caller": "pipeline/processor.go:308", "message": "Publish event: {\n \"@timestamp\": \"2019-01-24T07:23:15.429Z\",\n \"@metadata\": {\n \"beat\": \"filebeat\",\n \"type\": \"doc\",\n \"version\": \"6.5.4\"\n },\n \"log\": {\n \"flags\": [\n \"multiline\"\n]\n },\n \"message\": \"timestamp: 2019-01-24T07:04:42.322Z, level: INFO, message: just this line\\nof the\\nmessage is stored\",\n \"prospector\": {\n \"type\": \"log\"\n },\n \"input\": {\n \"type\": \"log\"\n },\n \"beat\": {\n \"name\": \"cujo\",\n \"hostname\": \"af02ce3109dd\",\n \"version\": \"6.5.4\"\n },\n \"host\": {\n \"name\": \"cujo\"\n },\n \"source\": \"/mnt/logs/sample.log\",\n \"offset\": 0\n}" } { "level": "debug", "timestamp": "2019-01-24T07:23:15.429Z", "logger": "publish", "caller": "pipeline/processor.go:308", "message": "Publish event: {\n \"@timestamp\": \"2019-01-24T07:23:15.429Z\",\n \"@metadata\": {\n \"beat\": \"filebeat\",\n \"type\": \"doc\",\n \"version\": \"6.5.4\"\n },\n \"prospector\": {\n \"type\": \"log\"\n },\n \"input\": {\n \"type\": \"log\"\n },\n \"beat\": {\n \"name\": \"cujo\",\n \"hostname\": \"af02ce3109dd\",\n \"version\": \"6.5.4\"\n },\n \"host\": {\n \"name\": \"cujo\"\n },\n \"offset\": 99,\n \"log\": {\n \"flags\": [\n \"multiline\"\n]\n },\n \"message\": \"timestamp: 2019-01-24T07:04:42.322Z, level: INFO, message: [proper multi\\nline\\nmessage because of the brackets]\",\n \"source\": \"/mnt/logs/sample.log\"\n}" } { "level": "debug", "timestamp": "2019-01-24T07:23:20.430Z", "logger": "publish", "caller": "pipeline/processor.go:308", "message": "Publish event: {\n \"@timestamp\": \"2019-01-24T07:23:15.429Z\",\n \"@metadata\": {\n \"beat\": \"filebeat\",\n \"type\": \"doc\",\n \"version\": \"6.5.4\"\n },\n \"source\": \"/mnt/logs/sample.log\",\n \"offset\": 210,\n \"log\": {\n \"flags\": [\n \"multiline\"\n]\n },\n \"message\": \"timestamp: 2019-01-24T07:04:42.322Z, level: INFO, message: [but if I have brackets in\\nthe [message] it doesnt work again]\",\n \"prospector\": {\n \"type\": \"log\"\n },\n \"input\": {\n \"type\": \"log\"\n },\n \"beat\": {\n \"hostname\": \"af02ce3109dd\",\n \"version\": \"6.5.4\",\n \"name\": \"cujo\"\n },\n \"host\": {\n \"name\": \"cujo\"\n }\n}" } ``` Logstash: ``` [2019-01-24T10:08:15,300][DEBUG][logstash.pipeline] filter received {"event"=>{"input"=>{"type"=>"log"}, "prospector"=>{"type"=>"log"}, "message"=>"timestamp: 2019-01-24T07:04:42.322Z, level: INFO, message: just this line\nof the \nmessage is stored", "@timestamp"=>2019-01-24T10:07:10.976Z, "beat"=>{"hostname"=>"0b3b67f06ae8", "name"=>"cujo", "version"=>"6.5.4"}, "source"=>"/mnt/logs/sample.log", "log"=>{"flags"=>["multiline"]}, "tags"=>["kv", "beats_input_codec_plain_applied"], "@version"=>"1", "host"=>{"name"=>"cujo"}, "offset"=>0}} [2019-01-24T10:08:15,301][DEBUG][logstash.pipeline] filter received {"event"=>{"input"=>{"type"=>"log"}, "prospector"=>{"type"=>"log"}, "message"=>"timestamp: 2019-01-24T07:23:15.429Z, level: INFO, message: [proper multi\nline\nmessage because of the brackets]", "@timestamp"=>2019-01-24T10:07:10.976Z, "beat"=>{"hostname"=>"0b3b67f06ae8", "name"=>"cujo", "version"=>"6.5.4"}, "source"=>"/mnt/logs/sample.log", "log"=>{"flags"=>["multiline"]}, "tags"=>["kv", "beats_input_codec_plain_applied"], "@version"=>"1", "host"=>{"name"=>"cujo"}, "offset"=>100}} [2019-01-24T10:08:15,309][DEBUG][logstash.pipeline] filter received {"event"=>{"input"=>{"type"=>"log"}, "prospector"=>{"type"=>"log"}, "message"=>"timestamp: 2019-01-24T07:04:42.322Z, level: INFO, message: [but if I have brackets in\nthe [message] it doesnt work again]", "@timestamp"=>2019-01-24T10:07:10.976Z, "beat"=>{"hostname"=>"0b3b67f06ae8", "name"=>"cujo", "version"=>"6.5.4"}, "source"=>"/mnt/logs/sample.log", "log"=>{"flags"=>["multiline"]}, "tags"=>["kv", "beats_input_codec_plain_applied"], "@version"=>"1", "host"=>{"name"=>"cujo"}, "offset"=>211}} [2019-01-24T10:08:15,370][DEBUG][logstash.pipeline] output received {"event"=>{"input"=>{"type"=>"log"}, "level"=>"INFO", "prospector"=>{"type"=>"log"}, "message"=>"[but if I have brackets in", "@timestamp"=>2019-01-24T07:04:42.322Z, "beat"=>{"hostname"=>"0b3b67f06ae8", "name"=>"cujo", "version"=>"6.5.4"}, "source"=>"/mnt/logs/sample.log", "log"=>{"flags"=>["multiline"]}, "tags"=>["kv", "beats_input_codec_plain_applied"], "@version"=>"1", "host"=>{"name"=>"cujo"}, "offset"=>211}} [2019-01-24T10:08:15,370][DEBUG][logstash.pipeline] output received {"event"=>{"input"=>{"type"=>"log"}, "level"=>"INFO", "prospector"=>{"type"=>"log"}, "message"=>"just this line", "@timestamp"=>2019-01-24T07:04:42.322Z, "beat"=>{"hostname"=>"0b3b67f06ae8", "name"=>"cujo", "version"=>"6.5.4"}, "source"=>"/mnt/logs/sample.log", "log"=>{"flags"=>["multiline"]}, "tags"=>["kv", "beats_input_codec_plain_applied"], "@version"=>"1", "host"=>{"name"=>"cujo"}, "offset"=>0}} [2019-01-24T10:08:15,373][DEBUG][logstash.pipeline] output received {"event"=>{"input"=>{"type"=>"log"}, "level"=>"INFO", "prospector"=>{"type"=>"log"}, "message"=>"proper multi\nline\nmessage because of the brackets", "@timestamp"=>2019-01-24T07:23:15.429Z, "beat"=>{"hostname"=>"0b3b67f06ae8", "name"=>"cujo", "version"=>"6.5.4"}, "source"=>"/mnt/logs/sample.log", "log"=>{"flags"=>["multiline"]}, "tags"=>["kv", "beats_input_codec_plain_applied"], "@version"=>"1", "host"=>{"name"=>"cujo"}, "offset"=>100}} ``` These are the test log messages: ``` timestamp: 2019-01-24T07:04:42.322Z, level: INFO, message: just this line of the message is stored timestamp: 2019-01-24T07:23:15.429Z, level: INFO, message: [proper multi line message because of the brackets] timestamp: 2019-01-24T07:04:42.322Z, level: INFO, message: [but if I have brackets in the [message] it doesnt work again] ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 21, 2019, 10:16am UTC](https://discuss.elastic.co/t/configure-kv-plugin-to-handle-multi-line-messages/165549/4 "2019-02-21T10:16:15Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.