Logstash km filter

gopisa · September 17, 2018, 3:36pm

Hi All,

need help on Km filter.

i am using the logstash 2.3V. my logs files look like below
}
"version" : "123"
"Keywords" : "1",
"Level" : "Verbose",
"Message" : "",
"Opcode" : "Info",
}

logstash conf:

input {
file {
path => "C:\Medseek\Logs\testfilter.log"
start_position => "beginning"
codec => multiline {
pattern => "^{"
negate => true
what => previous
}
}
}

filter {
mutate {

remove_field => ['message']

}

kv {

  value_split => ":"
  field_split => ",\n\s"

}
}

output:

""Keywords"" ==> "1",
""Level"" ==> "Verbose",
""Message"" ==> "",
""Opcode"" ==> "Info",

In every line starting "" it's coming . i tried to remove but no lock..

can you please help me on this

Thanks in advance

Regards
gopi

magnusbaeck · September 17, 2018, 4:28pm

You have a JSON log. Use a json filter to parse it, not kv.

gopisa · September 18, 2018, 3:25am

Hi,

My logs Are not in hain format. There is no kama(,) in between open and closed braket, so those logs are in text format kindly advise me to which filter to use to resolve the issue.

}
"version" : "123"
"Keywords" : "1",
"Level" : "Verbose",
"Message" : "",
"Opcode" : "Info",
}
}
"version" : "123"
"Keywords" : "1",
"Level" : "Verbose",
"Message" : "",
"Opcode" : "Info",
}

magnusbaeck · September 18, 2018, 6:25am

Do the messages really start with }? Is there really a comma before the closing brace? I would've expected the log to look like this:

{
"version" : "123"
"Keywords" : "1",
"Level" : "Verbose",
"Message" : "",
"Opcode" : "Info"
}
{
"version" : "123"
"Keywords" : "1",
"Level" : "Verbose",
"Message" : "",
"Opcode" : "Info"
}

gopisa · September 18, 2018, 7:22am

Hi ,

thank you for helping me on this issue.

my logs file totally look like below .

{
"SourceId" : "45687-8-9-8",
"EventId" : "1",
"Keywords" : "1",
"Level" : "Verbose",
"Message" : "",
"Opcode" : "Info",
"Task" : "345 WriteVerbose",
"Version" : "0",
"Timestamp" : "2018-09-17T15:42:53.4479069Z",
"Payload_message" : "Message received by doc. Set logging level above verbose to disable this message.",
"Payload_MessageType" : "GetPatientDemographics",
"Payload_MedseekPatientId" : "",
"Payload_OriginatingMessageId" : "15d7de04-ce8e-4g4g5-ac29-5rr54r5r2r",
"EOE" : ""
}
{
"SourceId" : "1548-789-65",
"EventId" : "1",
"Keywords" : "1",
"Level" : "Verbose",
"Message" : "",
"Opcode" : "Info",
"Task" : "555 WriteVerbose",
"Version" : "0",
"Timestamp" : "2018-09-17T15:42:55.1911694Z",
"Payload_message" : "Message received by PatientSvcTwoWayImplementation. Set logging level above verbose to disable this message.",
"Payload_MessageType" : "GetCustomDocument",
"Payload_MedseekPatientId" : "55555f55-4527-a7tt-f06ggggwssse4",
"Payload_OriginatingMessageId" : "555-5ec4-461b-666-8ffff5f6",
"EOE" : ""
}

If i used json filters .it's coming like below. pFA

logstash configuration is

input {
file {
path => "C:\Medseek\Logs\testfilter.log"
start_position => "beginning"

}
}

filter {
json {
source => "message"
remove_field => ["message"]
}
}

could you please help me on this .

Regards
gopi

magnusbaeck · September 18, 2018, 7:25am

Don't remove your multiline codec.

Please don't post screenshots when you can copy/paste the plain text instead.

gopisa · September 18, 2018, 9:37am

Hi ,

Thanks, It's coming correct format but getting below error .

←[33mFailed to flush outgoing items {:outgoing_count=>2, :exception=>"NoMethodError", :backtrace=>["C:/SoftwareInstalls/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-output-azure_loganalytics-0.2.0/lib/logstash/outputs/azure_loganalytics.rb:92:in flush'", "C:/SoftwareInstalls/logstash-2.3.2/ vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:219:inbuffer_flush'", "org/jruby/RubyHash.java:1342:in each'", "C:/SoftwareInstalls/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:216:inbuffer_flush'", "C:/SoftwareInstalls/logstash-2.3.2/vendor/bundle/jrub
y/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:112:in buffer_initialize'", "org/jruby/RubyKernel.java:1479:inloop'", "C:/SoftwareInstalls/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:110:in `buffer_initialize'"], :level=>:warn}←[0m

Thanks in advance

Regards
gopi

gopisa · September 18, 2018, 9:55am

Hi ,

total error msg.

C:\SoftwareInstalls\logstash-2.3.2>bin\logstash.bat -f conf\some.conf
io/console not supported; tty will not be manipulated
Settings: Default pipeline workers: 2
Pipeline main started
←[33mError parsing json {:source=>"message", :raw=>"}\r", :exception=>#<LogStash::Json::ParserError: Unexpected close marker '}': expected ']' (for ROOT starting at [Source: [B@3669e012; line: 1, column: 0])
at [Source: [B@3669e012; line: 1, column: 2]>, :level=>:warn}←[0m
{
"@timestamp" => "2018-09-18T09:51:48.298Z",
"message" => "}\r",
"@version" => "1",
"path" => "C:\Medseek\Logs\testfilter.log",
"host" => "AZ-AQA-COM-01",
"tags" => [
[0] "_jsonparsefailure"
]
}
{
"@timestamp" => "2018-09-18T09:51:48.298Z",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "C:\Medseek\Logs\testfilter.log",
"host" => "AZ-AQA-COM-01",
"SourceId" => "555666-cdd0-547e-222-888",
"EventId" => "1",
"Keywords" => "1",
"Level" => "Verbose",
"Message" => "",
"Opcode" => "Info",
"Task" => "66558 WriteVerbose",
"Version" => "0",
"Timestamp" => "2018-09-17T18:21:59.9311294Z",
"Payload_message" => "Message received by PatientSvcTwoWayImplementation. Set logging level above verbose to disable this message.",
"Payload_MessageType" => "GetCustomDocument",
"Payload_MedseekPatientId" => "555-14ea-426b-885h5-adbb55",
"Payload_OriginatingMessageId" => "88888-4224-4648-888-777333",
"EOE" => ""
}
←[33mFailed to flush outgoing items {:outgoing_count=>1, :exception=>"NoMethodError", :backtrace=>["C:/SoftwareInstalls/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-output-azure_loganalytics-0.2.0/lib/logstash/outputs/azure_loganalytics.rb:92:in flush'", "C:/SoftwareInstalls/logstash-2.3.2/ vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:219:inbuffer_flush'", "org/jruby/RubyHash.java:1342:in each'", "C:/SoftwareInstalls/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:216:inbuffer_flush'", "C:/SoftwareInstalls/logstash-2.3.2/vendor/bundle/jrub
y/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:112:in buffer_initialize'", "org/jruby/RubyKernel.java:1479:inloop'", "C:/SoftwareInstalls/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:110:in `buffer_initialize'"], :level=>:warn}←[0m

Thanks in advance

Regards
Gopi

gopisa · September 18, 2018, 1:51pm

Hi ,

please find the total error message bellow. logs are sending correctly now. but i am getting small error msg..

C:\SoftwareInstalls\logstash-2.3.2>bin\logstash.bat -f conf\some.conf
io/console not supported; tty will not be manipulated
Settings: Default pipeline workers: 2
Pipeline main started
←[33mError parsing json {:source=>"message", :raw=>"}\r", :exception=>#<LogStash::Json::ParserError: Unexpected close marker '}': expected ']' (for ROOT starting at [Source: [B@3669e012; line: 1, column: 0])
at [Source: [B@3669e012; line: 1, column: 2]>, :level=>:warn}←[0m
{
"@timestamp" => "2018-09-18T09:51:48.298Z",
"message" => "}\r",
"@version" => "1",
"path" => "C:\Medseek\Logs\testfilter.log",
"host" => "AZ-AQA-COM-01",
"tags" => [
[0] "_jsonparsefailure"
]
}
{
"@timestamp" => "2018-09-18T09:51:48.298Z",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "C:\Medseek\Logs\testfilter.log",
"host" => "AZ-AQA-COM-01",
"SourceId" => "555666-cdd0-547e-222-888",
"EventId" => "1",
"Keywords" => "1",
"Level" => "Verbose",
"Message" => "",
"Opcode" => "Info",
"Task" => "66558 WriteVerbose",
"Version" => "0",
"Timestamp" => "2018-09-17T18:21:59.9311294Z",
"Payload_message" => "Message received by PatientSvcTwoWayImplementation. Set logging level above verbose to disable this message.",
"Payload_MessageType" => "GetCustomDocument",
"Payload_MedseekPatientId" => "555-14ea-426b-885h5-adbb55",
"Payload_OriginatingMessageId" => "88888-4224-4648-888-777333",
"EOE" => ""
}

could you please help me on this.

gopisa · September 19, 2018, 2:56pm

Hi ,
Thanks for helping me on this .. i am facing below error, while running the logstash . could you please help me on this.

←[33mError parsing json {:source=>"message", :raw=>"}\r", :exception=>#<LogStash::Json::ParserError: Unexpected close marker '}': expected ']' (for ROOT starting at [Source: [B@6ff05fdd; line: 1, column: 0]) at [Source: [B@6ff05fdd; line:…

Thanks in advance .

Thanks
gopi

gopisa · September 19, 2018, 3:03pm

if the log file field is multiple lines getting error msg..

"payload_exception " : "this is the exception while connecting the server
server abc ip of em server
abcd
efgh
exception not resolved"

this type of field not processing , what need to change in logstash configuration .

Thanks in advance

gopi

magnusbaeck · September 19, 2018, 8:19pm

If you get a Logstash event containing just } there's something wrong with your multiline codec settings. Can you isolate exactly what the input that gives that result looks like? With the multiline codec settings you posted earlier it's not clear why you're having this problem.

gopisa · September 20, 2018, 8:40am

Hi ,

please find the logstash configuration and logs format bellow.

logstash conf;

input {
file {
path => "C:\Med\Logs\testfilter.log"
start_position => "beginning"
codec => multiline {
pattern => "^{"
negate => true
what => previous

}
}

}

filter {
json {
source => "message"
remove_field => [ "message", "tags"]
}

}

logs format ;

{
"SourceId" : "45687-8-9-8",
"EventId" : "1",
"Keywords" : "1",
"Level" : "Verbose",
"Message" : "",
"Opcode" : "Info",
"Task" : "345 WriteVerbose",
"Version" : "0",
"Timestamp" : "2018-09-17T15:42:53.4479069Z",
"Payload_message" : "Message received by doc. Set logging level above verbose to disable this message.",
"Payload_MessageType" : "GetPatientDemographics",
"Payload_MedseekPatientId" : "",
"payload_exception " : "this is the exception while connecting the server
server abc ip of em server
abcd
efgh
exception not resolved"
"Payload_OriginatingMessageId" : "15d7de04-ce8e-4g4g5-ac29-5rr54r5r2r",
"EOE" : ""
}
{
"SourceId" : "1548-789-65",
"EventId" : "1",
"Keywords" : "1",
"Level" : "Verbose",
"Message" : "",
"Opcode" : "Info",
"Task" : "555 WriteVerbose",
"Version" : "0",
"Timestamp" : "2018-09-17T15:42:55.1911694Z",
"Payload_message" : "Message received by PatientSvcTwoWayImplementation. Set logging level above verbose to disable this message.",
"Payload_MessageType" : "GetCustomDocument",
"Payload_MedseekPatientId" : "55555f55-4527-a7tt-f06ggggwssse4",
"Payload_OriginatingMessageId" : "555-5ec4-461b-666-8ffff5f6",
"EOE" : ""
}

In the logs files payload-exception field came getting error msg, might be issue with the multiple line of that payload-exception field ..what pattern need to use for that . can you please help me on this .

Thanks in advance

Regards
gopi

magnusbaeck · September 20, 2018, 8:49am

Literal newline characters inside JSON strings isn't allowed. They must be encoded as \n. However, I don't think that's the problem. This works fine for me:

$ cat test.config 
input {
  stdin {
    codec => multiline {
      pattern => "^\{"
      negate => true
      what => previous
    }
  }
}
output { stdout { codec => rubydebug } }
$ cat data 
{
"SourceId" : "45687-8-9-8",
"EventId" : "1",
"Keywords" : "1",
"Level" : "Verbose",
"Message" : "",
"Opcode" : "Info",
"Task" : "345 WriteVerbose",
"Version" : "0",
"Timestamp" : "2018-09-17T15:42:53.4479069Z",
"Payload_message" : "Message received by doc. Set logging level above
verbose to disable this message.",
"Payload_MessageType" : "GetPatientDemographics",
"Payload_MedseekPatientId" : "",
"payload_exception " : "this is the exception while connecting the
server
server abc ip of em server
abcd
efgh
exception not resolved"
"Payload_OriginatingMessageId" : "15d7de04-ce8e-4g4g5-ac29-5rr54r5r2r",
"EOE" : ""
}
{
"SourceId" : "1548-789-65",
"EventId" : "1",
"Keywords" : "1",
"Level" : "Verbose",
"Message" : "",
"Opcode" : "Info",
"Task" : "555 WriteVerbose",
"Version" : "0",
"Timestamp" : "2018-09-17T15:42:55.1911694Z",
"Payload_message" : "Message received by PatientSvcTwoWayImplementation.
Set logging level above verbose to disable this message.",
"Payload_MessageType" : "GetCustomDocument",
"Payload_MedseekPatientId" : "55555f55-4527-a7tt-f06ggggwssse4",
"Payload_OriginatingMessageId" : "555-5ec4-461b-666-8ffff5f6",
"EOE" : ""
}
$ /opt/logstash/bin/logstash -f test.config < data
Settings: Default pipeline workers: 8
Pipeline main started
{
    "@timestamp" => "2018-09-20T08:49:16.141Z",
       "message" => "{\n\"SourceId\" : \"45687-8-9-8\",\n\"EventId\" : \"1\",\n\"Keywords\" : \"1\",\n\"Level\" : \"Verbose\",\n\"Message\" : \"\",\n\"Opcode\" : \"Info\",\n\"Task\" : \"345 WriteVerbose\",\n\"Version\" : \"0\",\n\"Timestamp\" : \"2018-09-17T15:42:53.4479069Z\",\n\"Payload_message\" : \"Message received by doc. Set logging level above\nverbose to disable this message.\",\n\"Payload_MessageType\" : \"GetPatientDemographics\",\n\"Payload_MedseekPatientId\" : \"\",\n\"payload_exception \" : \"this is the exception while connecting the\nserver\nserver abc ip of em server\nabcd\nefgh\nexception not resolved\"\n\"Payload_OriginatingMessageId\" : \"15d7de04-ce8e-4g4g5-ac29-5rr54r5r2r\",\n\"EOE\" : \"\"\n}",
      "@version" => "1",
          "tags" => [
        [0] "multiline"
    ],
          "host" => "lnxolofon"
}
Pipeline main has been shutdown
stopping pipeline {:id=>"main"}

gopisa · September 20, 2018, 11:06am

Hi,

Thank you for helping me on this.
yes it's working with above configuration, if we are not used any filters getting below output . , in that out put total my log is coming in message field. if i used json filters, multi line field (payload-exception field ) is not working. getting error .

output

{
"@timestamp" => "2018-09-20T10:35:56.993Z",
"message" => "{\r\n"SourceId" : "99fc846d-cdd0-547e-babb-54c514616e9d",\r\n"EventId" : "1",\r\n"Keywords" : "1",\r\n"Level" : "Verbose",\r\n"Message" : "",\r\n"Opcode" : "Info",\r\n"Task" : "65533 WriteVerbose",\r\n"Version" : "0",\r\n"Timestamp" : "2018-09-
18T15:19:40.6927540Z",\r\n"Payload_message" : "Message received by NotificationSvcTwoWayImplementation. Set logging level above verbose to disable this message.",\r\n"Payload_MessageType" : "GetPatientNotifications",\r\n"Payload_MedseekPatientId" : "d209854e-c3c1-4999-aade-20819559c5da"
,\r\n"payload_exception " : "this is the exception while connecting the server\r\nserver abc ip of em server\r\nabcd\r\nefgh\r\nexception not resolved"\r\n"Payload_OriginatingMessageId" : "234d3d88-2a9f-4fcb-b983-8dc08dd36bbc",\r\n"EOE" : ""\r\n}\r",
"@version" => "1",
"tags" => [
[0] "multiline",
[1] "_jsonparsefailure"
],
"path" => "C:\Medseek\Logs\testfilter.log",
}

logstash configuration

input {
file {
path => "C:\Medseek\Logs\testfilter.log"
start_position => "beginning"
codec => multiline {
pattern => "^{"
negate => true
what => previous

}
}

}

filter {
json {
source => "message"
remove_field => [ "message", "tags"]
}

}

output for above logstash conf

{
"@timestamp" => "2018-09-20T10:50:25.723Z",
"@version" => "1",
"path" => "C:\Medseek\Logs\testfilter.log",
"host" => "AZ-AQA-COM-01",
"SourceId" => "444444-cdd0-547e-babb-1212121",
"EventId" => "1",
"Keywords" => "1",
"Level" => "Verbose",
"Message" => "",
"Opcode" => "Info",
"Task" => "65533 WriteVerbose",
"Version" => "0454",
"Timestamp" => "2018-09-18T15:19:40.6927540Z",
"Payload_message" => "Message received this message.",
"Payload_MessageType" => "getting",
"Payload_MedseekPatientId" => "444-454545-4999-aade-45454",
"Payload_OriginatingMessageId" => "454-2a9f-454545454-b983-454545454",
"EOE" => ""
}

expecting output:

{
"@timestamp" => "2018-09-20T10:50:25.723Z",
"@version" => "1",
"path" => "C:\Medseek\Logs\testfilter.log",
"host" => "AZ-AQA-COM-01",
"SourceId" => "444444-cdd0-547e-babb-1212121",
"EventId" => "1",
"Keywords" => "1",
"Level" => "Verbose",
"Message" => "",
"Opcode" => "Info",
"Task" => "65533 WriteVerbose",
"Version" => "0454",
"Timestamp" => "2018-09-18T15:19:40.6927540Z",
"Payload_message" => "Message received this message.",
"Payload_MessageType" => "getting",
"Payload_MedseekPatientId" => "444-454545-4999-aade-45454",
"Payload_Exception" : "at System.Xml..6666666.exception exception exception exception exception
at office.Xml.454544544.exception exception exception exception exception
at ntegration.555555.exception exception exception exception exception
"Payload_OriginatingMessageId" => "454-2a9f-454545454-b983-454545454",
"EOE" => ""
}

multi line field (payload-exception field ) also should come one filed . i tried but no luck.

another issue is if the log is contain three logs files if i ran the logstash, last log is not processing .

Thanks in advance

Gopi

magnusbaeck · September 20, 2018, 11:27am

multi line field (payload-exception field ) also should come one filed . i tried but no luck.

I can't tell off the top of my head what's going on and I don't have time to dig into it further.

another issue is if the log is contain three logs files if i ran the logstash, last log is not processing .

Set the multiline codec's auto_flush_interval option to something reasonably low, like 5.

gopisa · September 20, 2018, 12:12pm

No Problem. thanks you very much for helping me on configuring logstash . grate support ..

gopisa · September 24, 2018, 8:52am

Hi ,

if you have time can you please help me on this issue .

Thanks in advance

Regards
gopi

system · October 22, 2018, 8:52am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.