# Logstash codecs (multiline,json) uncomplete messages from file lines **URL:** \ **Category:** Logstash\ **Created:** [January 12, 2018, 11:56am UTC](https://discuss.elastic.co/t/logstash-codecs-multiline-json-uncomplete-messages-from-file-lines/115286 "2018-01-12T11:56:10Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Ged](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ged/32/65761_2.png) [@Ged](https://discuss.elastic.co/u/Ged)\ **Post date:** [January 12, 2018, 11:56am UTC](https://discuss.elastic.co/t/logstash-codecs-multiline-json-uncomplete-messages-from-file-lines/115286/1 "2018-01-12T11:56:10Z")
Hi, Elasticsearch 5.5.0 Logstash 5.5.0 logstash-codec-multiline 3.0.5 logstash-codec-json 3.0.3. I'm reading file (NFS) containing JSON lines using above codecs and from time to time receiving only part of line as message value: _Line from log:_ **{"Validation::Ms":1,"Authentication::Ms":4,"Profile::XId::FindProfile::Ms":2,"SearchQuery::** Ms":2,"Authorization::Ms":22,"DataRule::evaluate::Ms":0,"ActionCode":"XCC\_PrSearchRQ","SessionId":"ID-ptnhlp602-35350-1512548256092-0-103068496","AthId":"Shared/IDL:Sess\/SessMgr:1\.0.IDL/Common/!SMS\/RESE!SMSLB\/RES.LB!-3190237126642510194!910228!0!1!E2E-1","ThreadId":"8c80d70d","RequestType":"ProfileSearch","ClCd":"XV","DomainId":"XBNM","ClCnCd":"MYS","ObjectType":"PROFILE","User::ID":"1236","User::Group":"XBNM","User::Domain":"AA","User::HasView":true,"User::HasOSView":false,"DatabaseCalls::Ms::sum":81,"DatabaseCalls::Ms::count":64,"DatabaseCalls::Ms::avg":1.27,"DatabaseCalls::Ms::min":1,"DatabaseCalls::Ms::max":7,"Result":"SUCCESS","TotalTime::Ms":157,"Timestamp":"2018-01-12 11:34:09.743"} _Message value:_ Ms":2,"Authorization::Ms":22,"DataRule::evaluate::Ms":0,"ActionCode":"XCC\_PrSearchRQ","SessionId":"ID-ptnhlp602-35350-1512548256092-0-103068496","AthId":"Shared/IDL:Sess\/SessMgr:1\.0.IDL/Common/!SMS\/RESE!SMSLB\/RES.LB!-3190237126642510194!910228!0!1!E2E-1","ThreadId":"8c80d70d","RequestType":"ProfileSearch","ClCd":"XV","DomainId":"XBNM","ClCnCd":"MYS","ObjectType":"PROFILE","User::ID":"1236","User::Group":"XBNM","User::Domain":"AA","User::HasView":true,"User::HasOSView":false,"DatabaseCalls::Ms::sum":81,"DatabaseCalls::Ms::count":64,"DatabaseCalls::Ms::avg":1.27,"DatabaseCalls::Ms::min":1,"DatabaseCalls::Ms::max":7,"Result":"SUCCESS","TotalTime::Ms":157,"Timestamp":"2018-01-12 11:34:09.743"} Logstash configuration: ########################### INPUT ########################### input { file { codec =\> multiline { pattern =\> '^' negate =\> true what =\> previous } path =\> [ "file1.log", "file2.log", "file3.log", "file4.log", "file5.log", "file6.log", "file7.log", "file8.log" ] sincedb\_path =\> "/apps/elk5/logstash/sincedb-tn-ppp\_metrics" type =\> "mymetrics" } } ########################### FILTER ########################### filter { if [type] == "mymetrics" { json { source =\> message } ########################### OUTPUT ########################### output { if [type] == "mymetrics" { elasticsearch { hosts =\> '127.0.0.1:9200' index =\> "mymetrics-%{+YYYY.MM.dd}" } } } What can cause above ? May it be NFS ? Thanks in advance ! ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 16, 2018, 4:39pm UTC](https://discuss.elastic.co/t/logstash-codecs-multiline-json-uncomplete-messages-from-file-lines/115286/2 "2018-01-16T16:39:28Z")
What's the point of the multiline configuration? _All_ lines match `^`, and in the example you posted there are no linebreaks within the JSON document. ---
**Author:** ![Ged](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ged/32/65761_2.png) [@Ged](https://discuss.elastic.co/u/Ged)\ **Post date:** [January 16, 2018, 5:05pm UTC](https://discuss.elastic.co/t/logstash-codecs-multiline-json-uncomplete-messages-from-file-lines/115286/3 "2018-01-16T17:05:49Z")
logstash-codec-json works expects single JSON messages, so i need to extract them first when reading file containing JSON lines (JSON messages delimited by \n) using logstash-codec-multiline plugin. It works in most cases but problem occurs in about 0,3% of all processed log lines. Both strings from example in my previous post apply to single log line: - first is just a single line from log file - second is related "message" value. As you can see beginning of data in "message" value is missing comparing to log line. I mean that part: {"Validation::Ms":1,"Authentication::Ms":4,"Profile::XId::FindProfile::Ms":2,"SearchQuery:: I've marked difference in bold in first post. Do you think it's multiline problem ? I've suggested NFS after reading this: [https://discuss.elastic.co/t/logstash-vs-nfs-null-characters/25918](https://discuss.elastic.co/t/logstash-vs-nfs-null-characters/25918) Thanks ! ---
**Author:** ![Ged](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ged/32/65761_2.png) [@Ged](https://discuss.elastic.co/u/Ged)\ **Post date:** [January 20, 2018, 5:18pm UTC](https://discuss.elastic.co/t/logstash-codecs-multiline-json-uncomplete-messages-from-file-lines/115286/4 "2018-01-20T17:18:33Z")
@magnusbaeck Now i know what you meant - multiline is not needed for this configuration when i get complete JSON message in log in single line. After further investigation seems like reading via NFS causes this problem. So I'll contiune in thread [https://discuss.elastic.co/t/logstash-codecs-multiline-json-parseerror-illegal-character/115292](https://discuss.elastic.co/t/logstash-codecs-multiline-json-parseerror-illegal-character/115292) if needed. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 17, 2018, 5:18pm UTC](https://discuss.elastic.co/t/logstash-codecs-multiline-json-uncomplete-messages-from-file-lines/115286/5 "2018-02-17T17:18:36Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.