Logstash km filter


(gopisa) #1

Hi All,

need help on Km filter.

i am using the logstash 2.3V. my logs files look like below
}
"version" : "123"
"Keywords" : "1",
"Level" : "Verbose",
"Message" : "",
"Opcode" : "Info",
}

logstash conf:

input {
file {
path => "C:\Medseek\Logs\testfilter.log"
start_position => "beginning"
codec => multiline {
pattern => "^{"
negate => true
what => previous
}
}
}

filter {
mutate {

remove_field => ['message']

}

kv {

  value_split => ":"
  field_split => ",\n\s"

}
}

output:

""Keywords"" ==> "1",
""Level"" ==> "Verbose",
""Message"" ==> "",
""Opcode"" ==> "Info",

In every line starting "" it's coming . i tried to remove but no lock..

can you please help me on this

Thanks in advance

Regards
gopi


(Magnus Bäck) #2

You have a JSON log. Use a json filter to parse it, not kv.


(gopisa) #3

Hi,

My logs Are not in hain format. There is no kama(,) in between open and closed braket, so those logs are in text format kindly advise me to which filter to use to resolve the issue.

}
"version" : "123"
"Keywords" : "1",
"Level" : "Verbose",
"Message" : "",
"Opcode" : "Info",
}
}
"version" : "123"
"Keywords" : "1",
"Level" : "Verbose",
"Message" : "",
"Opcode" : "Info",
}


(Magnus Bäck) #4

Do the messages really start with }? Is there really a comma before the closing brace? I would've expected the log to look like this:

{
"version" : "123"
"Keywords" : "1",
"Level" : "Verbose",
"Message" : "",
"Opcode" : "Info"
}
{
"version" : "123"
"Keywords" : "1",
"Level" : "Verbose",
"Message" : "",
"Opcode" : "Info"
}

(gopisa) #5

Hi ,

thank you for helping me on this issue.

my logs file totally look like below .

{
"SourceId" : "45687-8-9-8",
"EventId" : "1",
"Keywords" : "1",
"Level" : "Verbose",
"Message" : "",
"Opcode" : "Info",
"Task" : "345 WriteVerbose",
"Version" : "0",
"Timestamp" : "2018-09-17T15:42:53.4479069Z",
"Payload_message" : "Message received by doc. Set logging level above verbose to disable this message.",
"Payload_MessageType" : "GetPatientDemographics",
"Payload_MedseekPatientId" : "",
"Payload_OriginatingMessageId" : "15d7de04-ce8e-4g4g5-ac29-5rr54r5r2r",
"EOE" : ""
}
{
"SourceId" : "1548-789-65",
"EventId" : "1",
"Keywords" : "1",
"Level" : "Verbose",
"Message" : "",
"Opcode" : "Info",
"Task" : "555 WriteVerbose",
"Version" : "0",
"Timestamp" : "2018-09-17T15:42:55.1911694Z",
"Payload_message" : "Message received by PatientSvcTwoWayImplementation. Set logging level above verbose to disable this message.",
"Payload_MessageType" : "GetCustomDocument",
"Payload_MedseekPatientId" : "55555f55-4527-a7tt-f06ggggwssse4",
"Payload_OriginatingMessageId" : "555-5ec4-461b-666-8ffff5f6",
"EOE" : ""
}

If i used json filters .it's coming like below. pFA

logstash configuration is

input {
file {
path => "C:\Medseek\Logs\testfilter.log"
start_position => "beginning"

    }
}

filter {
json {
source => "message"
remove_field => ["message"]
}
}

could you please help me on this .

Regards
gopi


(Magnus Bäck) #6

Don't remove your multiline codec.

Please don't post screenshots when you can copy/paste the plain text instead.


(gopisa) #7

Hi ,

Thanks, It's coming correct format but getting below error .

←[33mFailed to flush outgoing items {:outgoing_count=>2, :exception=>"NoMethodError", :backtrace=>["C:/SoftwareInstalls/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-output-azure_loganalytics-0.2.0/lib/logstash/outputs/azure_loganalytics.rb:92:in flush'", "C:/SoftwareInstalls/logstash-2.3.2/ vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:219:inbuffer_flush'", "org/jruby/RubyHash.java:1342:in each'", "C:/SoftwareInstalls/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:216:inbuffer_flush'", "C:/SoftwareInstalls/logstash-2.3.2/vendor/bundle/jrub
y/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:112:in buffer_initialize'", "org/jruby/RubyKernel.java:1479:inloop'", "C:/SoftwareInstalls/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:110:in `buffer_initialize'"], :level=>:warn}←[0m

Thanks in advance

Regards
gopi


(gopisa) #8

Hi ,

total error msg.

C:\SoftwareInstalls\logstash-2.3.2>bin\logstash.bat -f conf\some.conf
io/console not supported; tty will not be manipulated
Settings: Default pipeline workers: 2
Pipeline main started
←[33mError parsing json {:source=>"message", :raw=>"}\r", :exception=>#<LogStash::Json::ParserError: Unexpected close marker '}': expected ']' (for ROOT starting at [Source: [B@3669e012; line: 1, column: 0])
at [Source: [B@3669e012; line: 1, column: 2]>, :level=>:warn}←[0m
{
"@timestamp" => "2018-09-18T09:51:48.298Z",
"message" => "}\r",
"@version" => "1",
"path" => "C:\Medseek\Logs\testfilter.log",
"host" => "AZ-AQA-COM-01",
"tags" => [
[0] "_jsonparsefailure"
]
}
{
"@timestamp" => "2018-09-18T09:51:48.298Z",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "C:\Medseek\Logs\testfilter.log",
"host" => "AZ-AQA-COM-01",
"SourceId" => "555666-cdd0-547e-222-888",
"EventId" => "1",
"Keywords" => "1",
"Level" => "Verbose",
"Message" => "",
"Opcode" => "Info",
"Task" => "66558 WriteVerbose",
"Version" => "0",
"Timestamp" => "2018-09-17T18:21:59.9311294Z",
"Payload_message" => "Message received by PatientSvcTwoWayImplementation. Set logging level above verbose to disable this message.",
"Payload_MessageType" => "GetCustomDocument",
"Payload_MedseekPatientId" => "555-14ea-426b-885h5-adbb55",
"Payload_OriginatingMessageId" => "88888-4224-4648-888-777333",
"EOE" => ""
}
←[33mFailed to flush outgoing items {:outgoing_count=>1, :exception=>"NoMethodError", :backtrace=>["C:/SoftwareInstalls/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-output-azure_loganalytics-0.2.0/lib/logstash/outputs/azure_loganalytics.rb:92:in flush'", "C:/SoftwareInstalls/logstash-2.3.2/ vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:219:inbuffer_flush'", "org/jruby/RubyHash.java:1342:in each'", "C:/SoftwareInstalls/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:216:inbuffer_flush'", "C:/SoftwareInstalls/logstash-2.3.2/vendor/bundle/jrub
y/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:112:in buffer_initialize'", "org/jruby/RubyKernel.java:1479:inloop'", "C:/SoftwareInstalls/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:110:in `buffer_initialize'"], :level=>:warn}←[0m

Thanks in advance

Regards
Gopi


(gopisa) #9

Hi ,

please find the total error message bellow. logs are sending correctly now. but i am getting small error msg..

C:\SoftwareInstalls\logstash-2.3.2>bin\logstash.bat -f conf\some.conf
io/console not supported; tty will not be manipulated
Settings: Default pipeline workers: 2
Pipeline main started
←[33mError parsing json {:source=>"message", :raw=>"}\r", :exception=>#<LogStash::Json::ParserError: Unexpected close marker '}': expected ']' (for ROOT starting at [Source: [B@3669e012; line: 1, column: 0])
at [Source: [B@3669e012; line: 1, column: 2]>, :level=>:warn}←[0m
{
"@timestamp" => "2018-09-18T09:51:48.298Z",
"message" => "}\r",
"@version" => "1",
"path" => "C:\Medseek\Logs\testfilter.log",
"host" => "AZ-AQA-COM-01",
"tags" => [
[0] "_jsonparsefailure"
]
}
{
"@timestamp" => "2018-09-18T09:51:48.298Z",
"@version" => "1",
"tags" => [
[0] "multiline"
],
"path" => "C:\Medseek\Logs\testfilter.log",
"host" => "AZ-AQA-COM-01",
"SourceId" => "555666-cdd0-547e-222-888",
"EventId" => "1",
"Keywords" => "1",
"Level" => "Verbose",
"Message" => "",
"Opcode" => "Info",
"Task" => "66558 WriteVerbose",
"Version" => "0",
"Timestamp" => "2018-09-17T18:21:59.9311294Z",
"Payload_message" => "Message received by PatientSvcTwoWayImplementation. Set logging level above verbose to disable this message.",
"Payload_MessageType" => "GetCustomDocument",
"Payload_MedseekPatientId" => "555-14ea-426b-885h5-adbb55",
"Payload_OriginatingMessageId" => "88888-4224-4648-888-777333",
"EOE" => ""
}

could you please help me on this.


(gopisa) #10

Hi ,
Thanks for helping me on this .. i am facing below error, while running the logstash . could you please help me on this.

←[33mError parsing json {:source=>"message", :raw=>"}\r", :exception=>#<LogStash::Json::ParserError: Unexpected close marker '}': expected ']' (for ROOT starting at [Source: [B@6ff05fdd; line: 1, column: 0]) at [Source: [B@6ff05fdd; line:…

Thanks in advance .

Thanks
gopi


(gopisa) #11

if the log file field is multiple lines getting error msg..

"payload_exception " : "this is the exception while connecting the server
server abc ip of em server
abcd
efgh
exception not resolved"

this type of field not processing , what need to change in logstash configuration .

Thanks in advance

gopi


(Magnus Bäck) #12

If you get a Logstash event containing just } there's something wrong with your multiline codec settings. Can you isolate exactly what the input that gives that result looks like? With the multiline codec settings you posted earlier it's not clear why you're having this problem.


(gopisa) #13

Hi ,

please find the logstash configuration and logs format bellow.

logstash conf;

input {
file {
path => "C:\Med\Logs\testfilter.log"
start_position => "beginning"
codec => multiline {
pattern => "^{"
negate => true
what => previous

    }
}

}

filter {
json {
source => "message"
remove_field => [ "message", "tags"]
}

}

logs format ;

{
"SourceId" : "45687-8-9-8",
"EventId" : "1",
"Keywords" : "1",
"Level" : "Verbose",
"Message" : "",
"Opcode" : "Info",
"Task" : "345 WriteVerbose",
"Version" : "0",
"Timestamp" : "2018-09-17T15:42:53.4479069Z",
"Payload_message" : "Message received by doc. Set logging level above verbose to disable this message.",
"Payload_MessageType" : "GetPatientDemographics",
"Payload_MedseekPatientId" : "",
"payload_exception " : "this is the exception while connecting the server
server abc ip of em server
abcd
efgh
exception not resolved"
"Payload_OriginatingMessageId" : "15d7de04-ce8e-4g4g5-ac29-5rr54r5r2r",
"EOE" : ""
}
{
"SourceId" : "1548-789-65",
"EventId" : "1",
"Keywords" : "1",
"Level" : "Verbose",
"Message" : "",
"Opcode" : "Info",
"Task" : "555 WriteVerbose",
"Version" : "0",
"Timestamp" : "2018-09-17T15:42:55.1911694Z",
"Payload_message" : "Message received by PatientSvcTwoWayImplementation. Set logging level above verbose to disable this message.",
"Payload_MessageType" : "GetCustomDocument",
"Payload_MedseekPatientId" : "55555f55-4527-a7tt-f06ggggwssse4",
"Payload_OriginatingMessageId" : "555-5ec4-461b-666-8ffff5f6",
"EOE" : ""
}

In the logs files payload-exception field came getting error msg, might be issue with the multiple line of that payload-exception field ..what pattern need to use for that . can you please help me on this .

Thanks in advance

Regards
gopi


(Magnus Bäck) #14

Literal newline characters inside JSON strings isn't allowed. They must be encoded as \n. However, I don't think that's the problem. This works fine for me:

$ cat test.config 
input {
  stdin {
    codec => multiline {
      pattern => "^\{"
      negate => true
      what => previous
    }
  }
}
output { stdout { codec => rubydebug } }
$ cat data 
{
"SourceId" : "45687-8-9-8",
"EventId" : "1",
"Keywords" : "1",
"Level" : "Verbose",
"Message" : "",
"Opcode" : "Info",
"Task" : "345 WriteVerbose",
"Version" : "0",
"Timestamp" : "2018-09-17T15:42:53.4479069Z",
"Payload_message" : "Message received by doc. Set logging level above
verbose to disable this message.",
"Payload_MessageType" : "GetPatientDemographics",
"Payload_MedseekPatientId" : "",
"payload_exception " : "this is the exception while connecting the
server
server abc ip of em server
abcd
efgh
exception not resolved"
"Payload_OriginatingMessageId" : "15d7de04-ce8e-4g4g5-ac29-5rr54r5r2r",
"EOE" : ""
}
{
"SourceId" : "1548-789-65",
"EventId" : "1",
"Keywords" : "1",
"Level" : "Verbose",
"Message" : "",
"Opcode" : "Info",
"Task" : "555 WriteVerbose",
"Version" : "0",
"Timestamp" : "2018-09-17T15:42:55.1911694Z",
"Payload_message" : "Message received by PatientSvcTwoWayImplementation.
Set logging level above verbose to disable this message.",
"Payload_MessageType" : "GetCustomDocument",
"Payload_MedseekPatientId" : "55555f55-4527-a7tt-f06ggggwssse4",
"Payload_OriginatingMessageId" : "555-5ec4-461b-666-8ffff5f6",
"EOE" : ""
}
$ /opt/logstash/bin/logstash -f test.config < data
Settings: Default pipeline workers: 8
Pipeline main started
{
    "@timestamp" => "2018-09-20T08:49:16.141Z",
       "message" => "{\n\"SourceId\" : \"45687-8-9-8\",\n\"EventId\" : \"1\",\n\"Keywords\" : \"1\",\n\"Level\" : \"Verbose\",\n\"Message\" : \"\",\n\"Opcode\" : \"Info\",\n\"Task\" : \"345 WriteVerbose\",\n\"Version\" : \"0\",\n\"Timestamp\" : \"2018-09-17T15:42:53.4479069Z\",\n\"Payload_message\" : \"Message received by doc. Set logging level above\nverbose to disable this message.\",\n\"Payload_MessageType\" : \"GetPatientDemographics\",\n\"Payload_MedseekPatientId\" : \"\",\n\"payload_exception \" : \"this is the exception while connecting the\nserver\nserver abc ip of em server\nabcd\nefgh\nexception not resolved\"\n\"Payload_OriginatingMessageId\" : \"15d7de04-ce8e-4g4g5-ac29-5rr54r5r2r\",\n\"EOE\" : \"\"\n}",
      "@version" => "1",
          "tags" => [
        [0] "multiline"
    ],
          "host" => "lnxolofon"
}
Pipeline main has been shutdown
stopping pipeline {:id=>"main"}

(gopisa) #15

Hi,

Thank you for helping me on this.
yes it's working with above configuration, if we are not used any filters getting below output . , in that out put total my log is coming in message field. if i used json filters, multi line field (payload-exception field ) is not working. getting error .

output

{
"@timestamp" => "2018-09-20T10:35:56.993Z",
"message" => "{\r\n"SourceId" : "99fc846d-cdd0-547e-babb-54c514616e9d",\r\n"EventId" : "1",\r\n"Keywords" : "1",\r\n"Level" : "Verbose",\r\n"Message" : "",\r\n"Opcode" : "Info",\r\n"Task" : "65533 WriteVerbose",\r\n"Version" : "0",\r\n"Timestamp" : "2018-09-
18T15:19:40.6927540Z",\r\n"Payload_message" : "Message received by NotificationSvcTwoWayImplementation. Set logging level above verbose to disable this message.",\r\n"Payload_MessageType" : "GetPatientNotifications",\r\n"Payload_MedseekPatientId" : "d209854e-c3c1-4999-aade-20819559c5da"
,\r\n"payload_exception " : "this is the exception while connecting the server\r\nserver abc ip of em server\r\nabcd\r\nefgh\r\nexception not resolved"\r\n"Payload_OriginatingMessageId" : "234d3d88-2a9f-4fcb-b983-8dc08dd36bbc",\r\n"EOE" : ""\r\n}\r",
"@version" => "1",
"tags" => [
[0] "multiline",
[1] "_jsonparsefailure"
],
"path" => "C:\Medseek\Logs\testfilter.log",
}

logstash configuration

input {
file {
path => "C:\Medseek\Logs\testfilter.log"
start_position => "beginning"
codec => multiline {
pattern => "^{"
negate => true
what => previous

    }
}

}

filter {
json {
source => "message"
remove_field => [ "message", "tags"]
}

}

output for above logstash conf

{
"@timestamp" => "2018-09-20T10:50:25.723Z",
"@version" => "1",
"path" => "C:\Medseek\Logs\testfilter.log",
"host" => "AZ-AQA-COM-01",
"SourceId" => "444444-cdd0-547e-babb-1212121",
"EventId" => "1",
"Keywords" => "1",
"Level" => "Verbose",
"Message" => "",
"Opcode" => "Info",
"Task" => "65533 WriteVerbose",
"Version" => "0454",
"Timestamp" => "2018-09-18T15:19:40.6927540Z",
"Payload_message" => "Message received this message.",
"Payload_MessageType" => "getting",
"Payload_MedseekPatientId" => "444-454545-4999-aade-45454",
"Payload_OriginatingMessageId" => "454-2a9f-454545454-b983-454545454",
"EOE" => ""
}

expecting output:

{
"@timestamp" => "2018-09-20T10:50:25.723Z",
"@version" => "1",
"path" => "C:\Medseek\Logs\testfilter.log",
"host" => "AZ-AQA-COM-01",
"SourceId" => "444444-cdd0-547e-babb-1212121",
"EventId" => "1",
"Keywords" => "1",
"Level" => "Verbose",
"Message" => "",
"Opcode" => "Info",
"Task" => "65533 WriteVerbose",
"Version" => "0454",
"Timestamp" => "2018-09-18T15:19:40.6927540Z",
"Payload_message" => "Message received this message.",
"Payload_MessageType" => "getting",
"Payload_MedseekPatientId" => "444-454545-4999-aade-45454",
"Payload_Exception" : "at System.Xml..6666666.exception exception exception exception exception
at office.Xml.454544544.exception exception exception exception exception
at ntegration.555555.exception exception exception exception exception
"Payload_OriginatingMessageId" => "454-2a9f-454545454-b983-454545454",
"EOE" => ""
}

multi line field (payload-exception field ) also should come one filed . i tried but no luck.

another issue is if the log is contain three logs files if i ran the logstash, last log is not processing .

Thanks in advance

Gopi


(Magnus Bäck) #16

multi line field (payload-exception field ) also should come one filed . i tried but no luck.

I can't tell off the top of my head what's going on and I don't have time to dig into it further.

another issue is if the log is contain three logs files if i ran the logstash, last log is not processing .

Set the multiline codec's auto_flush_interval option to something reasonably low, like 5.


(gopisa) #17

No Problem. thanks you very much for helping me on configuring logstash . grate support ..


(gopisa) #18

Hi ,

if you have time can you please help me on this issue .

Thanks in advance

Regards
gopi


(system) #19

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.