Logstash codecs (multiline,json) uncomplete messages from file lines

Elastic Stack Logstash
1

Hi,

Elasticsearch 5.5.0
Logstash 5.5.0
logstash-codec-multiline 3.0.5
logstash-codec-json 3.0.3.

I'm reading file (NFS) containing JSON lines using above codecs and from time to time receiving only part of line as message value:

Line from log:

{"Validation::Ms":1,"Authentication::Ms":4,"Profile::XId::FindProfile::Ms":2,"SearchQuery:: Ms":2,"Authorization::Ms":22,"DataRule::evaluate::Ms":0,"ActionCode":"XCC_PrSearchRQ","SessionId":"ID-ptnhlp602-35350-1512548256092-0-103068496","AthId":"Shared/IDL:Sess\/SessMgr:1\.0.IDL/Common/!SMS\/RESE!SMSLB\/RES.LB!-3190237126642510194!910228!0!1!E2E-1","ThreadId":"8c80d70d","RequestType":"ProfileSearch","ClCd":"XV","DomainId":"XBNM","ClCnCd":"MYS","ObjectType":"PROFILE","User::ID":"1236","User::Group":"XBNM","User::Domain":"AA","User::HasView":true,"User::HasOSView":false,"DatabaseCalls::Ms::sum":81,"DatabaseCalls::Ms::count":64,"DatabaseCalls::Ms::avg":1.27,"DatabaseCalls::Ms::min":1,"DatabaseCalls::Ms::max":7,"Result":"SUCCESS","TotalTime::Ms":157,"Timestamp":"2018-01-12 11:34:09.743"}

Message value:

Ms":2,"Authorization::Ms":22,"DataRule::evaluate::Ms":0,"ActionCode":"XCC_PrSearchRQ","SessionId":"ID-ptnhlp602-35350-1512548256092-0-103068496","AthId":"Shared/IDL:Sess\/SessMgr:1\.0.IDL/Common/!SMS\/RESE!SMSLB\/RES.LB!-3190237126642510194!910228!0!1!E2E-1","ThreadId":"8c80d70d","RequestType":"ProfileSearch","ClCd":"XV","DomainId":"XBNM","ClCnCd":"MYS","ObjectType":"PROFILE","User::ID":"1236","User::Group":"XBNM","User::Domain":"AA","User::HasView":true,"User::HasOSView":false,"DatabaseCalls::Ms::sum":81,"DatabaseCalls::Ms::count":64,"DatabaseCalls::Ms::avg":1.27,"DatabaseCalls::Ms::min":1,"DatabaseCalls::Ms::max":7,"Result":"SUCCESS","TotalTime::Ms":157,"Timestamp":"2018-01-12 11:34:09.743"}

Logstash configuration:
########################### INPUT ###########################
input {
file {
codec => multiline {
pattern => '^'
negate => true
what => previous
}
path => [
"file1.log",
"file2.log",
"file3.log",
"file4.log",
"file5.log",
"file6.log",
"file7.log",
"file8.log"
]
sincedb_path => "/apps/elk5/logstash/sincedb-tn-ppp_metrics"
type => "mymetrics"
}
}

########################### FILTER ###########################
filter {
if [type] == "mymetrics"
{
json {
source => message
}

########################### OUTPUT ###########################
output {
if [type] == "mymetrics"
{
elasticsearch {
hosts => '127.0.0.1:9200'
index => "mymetrics-%{+YYYY.MM.dd}"
}
}
}

What can cause above ? May it be NFS ?

Thanks in advance !

2

What's the point of the multiline configuration? All lines match ^, and in the example you posted there are no linebreaks within the JSON document.

3

logstash-codec-json works expects single JSON messages, so i need to extract them first when reading file containing JSON lines (JSON messages delimited by \n) using logstash-codec-multiline plugin.

It works in most cases but problem occurs in about 0,3% of all processed log lines.

Both strings from example in my previous post apply to single log line:

  • first is just a single line from log file
  • second is related "message" value.

As you can see beginning of data in "message" value is missing comparing to log line. I mean that part:
{"Validation::Ms":1,"Authentication::Ms":4,"Profile::XId::FindProfile::Ms":2,"SearchQuery::

I've marked difference in bold in first post.

Do you think it's multiline problem ?

I've suggested NFS after reading this:
https://discuss.elastic.co/t/logstash-vs-nfs-null-characters/25918

Thanks !

4

@magnusbaeck
Now i know what you meant - multiline is not needed for this configuration when i get complete JSON message in log in single line.

After further investigation seems like reading via NFS causes this problem.

So I'll contiune in thread
https://discuss.elastic.co/t/logstash-codecs-multiline-json-parseerror-illegal-character/115292
if needed.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.