# Logstash km filter

**URL:** <https://discuss.elastic.co/t/logstash-km-filter/148857>\
**Category:** Logstash\
**Created:** [September 17, 2018, 3:36pm UTC](https://discuss.elastic.co/t/logstash-km-filter/148857 "2018-09-17T15:36:34Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![gopisa](https://avatars.discourse-cdn.com/v4/letter/g/7993a0/32.png) [@gopisa](https://discuss.elastic.co/u/gopisa)\
**Post date:** [September 17, 2018, 3:36pm UTC](https://discuss.elastic.co/t/logstash-km-filter/148857/1 "2018-09-17T15:36:34Z")

</div>

Hi All,

need help on Km filter.

i am using the logstash 2.3V. my logs files look like below  
}  
"version" : "123"  
"Keywords" : "1",  
"Level" : "Verbose",  
"Message" : "",  
"Opcode" : "Info",  
}

logstash conf:

input {  
file {  
path =\> "C:\Medseek\Logs\testfilter.log"  
start\_position =\> "beginning"  
codec =\> multiline {  
pattern =\> "^{"  
negate =\> true  
what =\> previous  
}  
}  
}

filter {  
mutate {

```
remove_field => ['message']

```

}

kv {

```
  value_split => ":"
  field_split => ",\n\s"

```

}  
}

output:

""Keywords"" ==\> "1",  
""Level"" ==\> "Verbose",  
""Message"" ==\> "",  
""Opcode"" ==\> "Info",

In every line starting "" it's coming . i tried to remove but no lock..

can you please help me on this

Thanks in advance

Regards  
gopi

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 17, 2018, 4:28pm UTC](https://discuss.elastic.co/t/logstash-km-filter/148857/2 "2018-09-17T16:28:07Z")

</div>

You have a JSON log. Use a json filter to parse it, not kv.

---

<div class="post-metadata">

**Author:** ![gopisa](https://avatars.discourse-cdn.com/v4/letter/g/7993a0/32.png) [@gopisa](https://discuss.elastic.co/u/gopisa)\
**Post date:** [September 18, 2018, 3:25am UTC](https://discuss.elastic.co/t/logstash-km-filter/148857/3 "2018-09-18T03:25:11Z")

</div>

Hi,

My logs Are not in hain format. There is no kama(,) in between open and closed braket, so those logs are in text format kindly advise me to which filter to use to resolve the issue.

}  
"version" : "123"  
"Keywords" : "1",  
"Level" : "Verbose",  
"Message" : "",  
"Opcode" : "Info",  
}  
}  
"version" : "123"  
"Keywords" : "1",  
"Level" : "Verbose",  
"Message" : "",  
"Opcode" : "Info",  
}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 18, 2018, 6:25am UTC](https://discuss.elastic.co/t/logstash-km-filter/148857/4 "2018-09-18T06:25:07Z")

</div>

Do the messages really _start_ with `}`? Is there really a comma _before_ the closing brace? I would've expected the log to look like this:

```nohighlight
{
"version" : "123"
"Keywords" : "1",
"Level" : "Verbose",
"Message" : "",
"Opcode" : "Info"
}
{
"version" : "123"
"Keywords" : "1",
"Level" : "Verbose",
"Message" : "",
"Opcode" : "Info"
}

```

---

<div class="post-metadata">

**Author:** ![gopisa](https://avatars.discourse-cdn.com/v4/letter/g/7993a0/32.png) [@gopisa](https://discuss.elastic.co/u/gopisa)\
**Post date:** [September 18, 2018, 7:22am UTC](https://discuss.elastic.co/t/logstash-km-filter/148857/5 "2018-09-18T07:22:49Z")

</div>

Hi ,

thank you for helping me on this issue.

my logs file totally look like below .

{  
"SourceId" : "45687-8-9-8",  
"EventId" : "1",  
"Keywords" : "1",  
"Level" : "Verbose",  
"Message" : "",  
"Opcode" : "Info",  
"Task" : "345 WriteVerbose",  
"Version" : "0",  
"Timestamp" : "2018-09-17T15:42:53.4479069Z",  
"Payload\_message" : "Message received by doc. Set logging level above verbose to disable this message.",  
"Payload\_MessageType" : "GetPatientDemographics",  
"Payload\_MedseekPatientId" : "",  
"Payload\_OriginatingMessageId" : "15d7de04-ce8e-4g4g5-ac29-5rr54r5r2r",  
"EOE" : ""  
}  
{  
"SourceId" : "1548-789-65",  
"EventId" : "1",  
"Keywords" : "1",  
"Level" : "Verbose",  
"Message" : "",  
"Opcode" : "Info",  
"Task" : "555 WriteVerbose",  
"Version" : "0",  
"Timestamp" : "2018-09-17T15:42:55.1911694Z",  
"Payload\_message" : "Message received by PatientSvcTwoWayImplementation. Set logging level above verbose to disable this message.",  
"Payload\_MessageType" : "GetCustomDocument",  
"Payload\_MedseekPatientId" : "55555f55-4527-a7tt-f06ggggwssse4",  
"Payload\_OriginatingMessageId" : "555-5ec4-461b-666-8ffff5f6",  
"EOE" : ""  
}

If i used json filters .it's coming like below. pFA

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/6/66ffa5aed6babb243e058537c7ceb2e295d48444.png)

logstash configuration is

input {  
file {  
path =\> "C:\Medseek\Logs\testfilter.log"  
start\_position =\> "beginning"

```
    }
}

```

filter {  
json {  
source =\> "message"  
remove\_field =\> ["message"]  
}  
}

could you please help me on this .

Regards  
gopi

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 18, 2018, 7:25am UTC](https://discuss.elastic.co/t/logstash-km-filter/148857/6 "2018-09-18T07:25:14Z")

</div>

Don't remove your multiline codec.

Please don't post screenshots when you can copy/paste the plain text instead.

---

<div class="post-metadata">

**Author:** ![gopisa](https://avatars.discourse-cdn.com/v4/letter/g/7993a0/32.png) [@gopisa](https://discuss.elastic.co/u/gopisa)\
**Post date:** [September 18, 2018, 9:37am UTC](https://discuss.elastic.co/t/logstash-km-filter/148857/7 "2018-09-18T09:37:20Z")

</div>

Hi ,

Thanks, It's coming correct format but getting below error .

←[33mFailed to flush outgoing items {:outgoing\_count=\>2, :exception=\>"NoMethodError", :backtrace=\>["C:/SoftwareInstalls/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-output-azure\_loganalytics-0.2.0/lib/logstash/outputs/azure\_loganalytics.rb:92:in `flush'", "C:/SoftwareInstalls/logstash-2.3.2/ vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:219:in`buffer\_flush'", "org/jruby/RubyHash.java:1342:in `each'", "C:/SoftwareInstalls/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:216:in`buffer\_flush'", "C:/SoftwareInstalls/logstash-2.3.2/vendor/bundle/jrub  
y/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:112:in `buffer_initialize'", "org/jruby/RubyKernel.java:1479:in`loop'", "C:/SoftwareInstalls/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:110:in `buffer\_initialize'"], :level=\>:warn}←[0m

Thanks in advance

Regards  
gopi

---

<div class="post-metadata">

**Author:** ![gopisa](https://avatars.discourse-cdn.com/v4/letter/g/7993a0/32.png) [@gopisa](https://discuss.elastic.co/u/gopisa)\
**Post date:** [September 18, 2018, 9:55am UTC](https://discuss.elastic.co/t/logstash-km-filter/148857/8 "2018-09-18T09:55:54Z")

</div>

Hi ,

total error msg.

C:\SoftwareInstalls\logstash-2.3.2\>bin\logstash.bat -f conf\some.conf  
io/console not supported; tty will not be manipulated  
Settings: Default pipeline workers: 2  
Pipeline main started  
←[33mError parsing json {:source=\>"message", :raw=\>"}\r", :exception=\>#\<LogStash::Json::ParserError: Unexpected close marker '}': expected ']' (for ROOT starting at [Source: [B@3669e012; line: 1, column: 0])  
at [Source: [B@3669e012; line: 1, column: 2]\>, :level=\>:warn}←[0m  
{  
"@timestamp" =\> "2018-09-18T09:51:48.298Z",  
"message" =\> "}\r",  
"@version" =\> "1",  
"path" =\> "C:\Medseek\Logs\testfilter.log",  
"host" =\> "AZ-AQA-COM-01",  
"tags" =\> [  
[0] "\_jsonparsefailure"  
]  
}  
{  
"@timestamp" =\> "2018-09-18T09:51:48.298Z",  
"@version" =\> "1",  
"tags" =\> [  
[0] "multiline"  
],  
"path" =\> "C:\Medseek\Logs\testfilter.log",  
"host" =\> "AZ-AQA-COM-01",  
"SourceId" =\> "555666-cdd0-547e-222-888",  
"EventId" =\> "1",  
"Keywords" =\> "1",  
"Level" =\> "Verbose",  
"Message" =\> "",  
"Opcode" =\> "Info",  
"Task" =\> "66558 WriteVerbose",  
"Version" =\> "0",  
"Timestamp" =\> "2018-09-17T18:21:59.9311294Z",  
"Payload\_message" =\> "Message received by PatientSvcTwoWayImplementation. Set logging level above verbose to disable this message.",  
"Payload\_MessageType" =\> "GetCustomDocument",  
"Payload\_MedseekPatientId" =\> "555-14ea-426b-885h5-adbb55",  
"Payload\_OriginatingMessageId" =\> "88888-4224-4648-888-777333",  
"EOE" =\> ""  
}  
←[33mFailed to flush outgoing items {:outgoing\_count=\>1, :exception=\>"NoMethodError", :backtrace=\>["C:/SoftwareInstalls/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-output-azure\_loganalytics-0.2.0/lib/logstash/outputs/azure\_loganalytics.rb:92:in `flush'", "C:/SoftwareInstalls/logstash-2.3.2/ vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:219:in`buffer\_flush'", "org/jruby/RubyHash.java:1342:in `each'", "C:/SoftwareInstalls/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:216:in`buffer\_flush'", "C:/SoftwareInstalls/logstash-2.3.2/vendor/bundle/jrub  
y/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:112:in `buffer_initialize'", "org/jruby/RubyKernel.java:1479:in`loop'", "C:/SoftwareInstalls/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:110:in `buffer\_initialize'"], :level=\>:warn}←[0m

Thanks in advance

Regards  
Gopi

---

<div class="post-metadata">

**Author:** ![gopisa](https://avatars.discourse-cdn.com/v4/letter/g/7993a0/32.png) [@gopisa](https://discuss.elastic.co/u/gopisa)\
**Post date:** [September 18, 2018, 1:51pm UTC](https://discuss.elastic.co/t/logstash-km-filter/148857/9 "2018-09-18T13:51:32Z")

</div>

Hi ,

please find the total error message bellow. logs are sending correctly now. but i am getting small error msg..

C:\SoftwareInstalls\logstash-2.3.2\>bin\logstash.bat -f conf\some.conf  
io/console not supported; tty will not be manipulated  
Settings: Default pipeline workers: 2  
Pipeline main started  
←[33mError parsing json {:source=\>"message", :raw=\>"}\r", :exception=\>#\<LogStash::Json::ParserError: Unexpected close marker '}': expected ']' (for ROOT starting at [Source: [B@3669e012; line: 1, column: 0])  
at [Source: [B@3669e012; line: 1, column: 2]\>, :level=\>:warn}←[0m  
{  
"@timestamp" =\> "2018-09-18T09:51:48.298Z",  
"message" =\> "}\r",  
"@version" =\> "1",  
"path" =\> "C:\Medseek\Logs\testfilter.log",  
"host" =\> "AZ-AQA-COM-01",  
"tags" =\> [  
[0] "\_jsonparsefailure"  
]  
}  
{  
"@timestamp" =\> "2018-09-18T09:51:48.298Z",  
"@version" =\> "1",  
"tags" =\> [  
[0] "multiline"  
],  
"path" =\> "C:\Medseek\Logs\testfilter.log",  
"host" =\> "AZ-AQA-COM-01",  
"SourceId" =\> "555666-cdd0-547e-222-888",  
"EventId" =\> "1",  
"Keywords" =\> "1",  
"Level" =\> "Verbose",  
"Message" =\> "",  
"Opcode" =\> "Info",  
"Task" =\> "66558 WriteVerbose",  
"Version" =\> "0",  
"Timestamp" =\> "2018-09-17T18:21:59.9311294Z",  
"Payload\_message" =\> "Message received by PatientSvcTwoWayImplementation. Set logging level above verbose to disable this message.",  
"Payload\_MessageType" =\> "GetCustomDocument",  
"Payload\_MedseekPatientId" =\> "555-14ea-426b-885h5-adbb55",  
"Payload\_OriginatingMessageId" =\> "88888-4224-4648-888-777333",  
"EOE" =\> ""  
}

could you please help me on this.

---

<div class="post-metadata">

**Author:** ![gopisa](https://avatars.discourse-cdn.com/v4/letter/g/7993a0/32.png) [@gopisa](https://discuss.elastic.co/u/gopisa)\
**Post date:** [September 19, 2018, 2:56pm UTC](https://discuss.elastic.co/t/logstash-km-filter/148857/10 "2018-09-19T14:56:52Z")

</div>

Hi ,  
Thanks for helping me on this .. i am facing below error, while running the logstash . could you please help me on this.

←[33mError parsing json {:source=\>"message", :raw=\>"}\r", :exception=\>#\<LogStash::Json::ParserError: Unexpected close marker '}': expected ']' (for ROOT starting at [Source: [B@6ff05fdd; line: 1, column: 0]) at [Source: [B@6ff05fdd; line:…

Thanks in advance .

Thanks  
gopi

---

<div class="post-metadata">

**Author:** ![gopisa](https://avatars.discourse-cdn.com/v4/letter/g/7993a0/32.png) [@gopisa](https://discuss.elastic.co/u/gopisa)\
**Post date:** [September 19, 2018, 3:03pm UTC](https://discuss.elastic.co/t/logstash-km-filter/148857/11 "2018-09-19T15:03:03Z")

</div>

if the log file field is multiple lines getting error msg..

"payload\_exception " : "this is the exception while connecting the server  
server abc ip of em server  
abcd  
efgh  
exception not resolved"

this type of field not processing , what need to change in logstash configuration .

Thanks in advance

gopi

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 19, 2018, 8:19pm UTC](https://discuss.elastic.co/t/logstash-km-filter/148857/12 "2018-09-19T20:19:41Z")

</div>

If you get a Logstash event containing just `}` there's something wrong with your multiline codec settings. Can you isolate exactly what the input that gives that result looks like? With the multiline codec settings you posted earlier it's not clear why you're having this problem.

---

<div class="post-metadata">

**Author:** ![gopisa](https://avatars.discourse-cdn.com/v4/letter/g/7993a0/32.png) [@gopisa](https://discuss.elastic.co/u/gopisa)\
**Post date:** [September 20, 2018, 8:40am UTC](https://discuss.elastic.co/t/logstash-km-filter/148857/13 "2018-09-20T08:40:59Z")

</div>

Hi ,

please find the logstash configuration and logs format bellow.

## logstash conf;

input {  
file {  
path =\> "C:\Med\Logs\testfilter.log"  
start\_position =\> "beginning"  
codec =\> multiline {  
pattern =\> "^{"  
negate =\> true  
what =\> previous

```
    }
}

```

}

filter {  
json {  
source =\> "message"  
remove\_field =\> ["message", "tags"]  
}

}

## logs format ;

{  
"SourceId" : "45687-8-9-8",  
"EventId" : "1",  
"Keywords" : "1",  
"Level" : "Verbose",  
"Message" : "",  
"Opcode" : "Info",  
"Task" : "345 WriteVerbose",  
"Version" : "0",  
"Timestamp" : "2018-09-17T15:42:53.4479069Z",  
"Payload\_message" : "Message received by doc. Set logging level above verbose to disable this message.",  
"Payload\_MessageType" : "GetPatientDemographics",  
"Payload\_MedseekPatientId" : "",  
"payload\_exception " : "this is the exception while connecting the server  
server abc ip of em server  
abcd  
efgh  
exception not resolved"  
"Payload\_OriginatingMessageId" : "15d7de04-ce8e-4g4g5-ac29-5rr54r5r2r",  
"EOE" : ""  
}  
{  
"SourceId" : "1548-789-65",  
"EventId" : "1",  
"Keywords" : "1",  
"Level" : "Verbose",  
"Message" : "",  
"Opcode" : "Info",  
"Task" : "555 WriteVerbose",  
"Version" : "0",  
"Timestamp" : "2018-09-17T15:42:55.1911694Z",  
"Payload\_message" : "Message received by PatientSvcTwoWayImplementation. Set logging level above verbose to disable this message.",  
"Payload\_MessageType" : "GetCustomDocument",  
"Payload\_MedseekPatientId" : "55555f55-4527-a7tt-f06ggggwssse4",  
"Payload\_OriginatingMessageId" : "555-5ec4-461b-666-8ffff5f6",  
"EOE" : ""  
}

In the logs files payload-exception field came getting error msg, might be issue with the multiple line of that payload-exception field ..what pattern need to use for that . can you please help me on this .

Thanks in advance

Regards  
gopi

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 20, 2018, 8:49am UTC](https://discuss.elastic.co/t/logstash-km-filter/148857/14 "2018-09-20T08:49:47Z")

</div>

Literal newline characters inside JSON strings isn't allowed. They must be encoded as `\n`. However, I don't think that's the problem. This works fine for me:

```nohighlight
$ cat test.config 
input {
  stdin {
    codec => multiline {
      pattern => "^\{"
      negate => true
      what => previous
    }
  }
}
output { stdout { codec => rubydebug } }
$ cat data 
{
"SourceId" : "45687-8-9-8",
"EventId" : "1",
"Keywords" : "1",
"Level" : "Verbose",
"Message" : "",
"Opcode" : "Info",
"Task" : "345 WriteVerbose",
"Version" : "0",
"Timestamp" : "2018-09-17T15:42:53.4479069Z",
"Payload_message" : "Message received by doc. Set logging level above
verbose to disable this message.",
"Payload_MessageType" : "GetPatientDemographics",
"Payload_MedseekPatientId" : "",
"payload_exception " : "this is the exception while connecting the
server
server abc ip of em server
abcd
efgh
exception not resolved"
"Payload_OriginatingMessageId" : "15d7de04-ce8e-4g4g5-ac29-5rr54r5r2r",
"EOE" : ""
}
{
"SourceId" : "1548-789-65",
"EventId" : "1",
"Keywords" : "1",
"Level" : "Verbose",
"Message" : "",
"Opcode" : "Info",
"Task" : "555 WriteVerbose",
"Version" : "0",
"Timestamp" : "2018-09-17T15:42:55.1911694Z",
"Payload_message" : "Message received by PatientSvcTwoWayImplementation.
Set logging level above verbose to disable this message.",
"Payload_MessageType" : "GetCustomDocument",
"Payload_MedseekPatientId" : "55555f55-4527-a7tt-f06ggggwssse4",
"Payload_OriginatingMessageId" : "555-5ec4-461b-666-8ffff5f6",
"EOE" : ""
}
$ /opt/logstash/bin/logstash -f test.config < data
Settings: Default pipeline workers: 8
Pipeline main started
{
    "@timestamp" => "2018-09-20T08:49:16.141Z",
       "message" => "{\n\"SourceId\" : \"45687-8-9-8\",\n\"EventId\" : \"1\",\n\"Keywords\" : \"1\",\n\"Level\" : \"Verbose\",\n\"Message\" : \"\",\n\"Opcode\" : \"Info\",\n\"Task\" : \"345 WriteVerbose\",\n\"Version\" : \"0\",\n\"Timestamp\" : \"2018-09-17T15:42:53.4479069Z\",\n\"Payload_message\" : \"Message received by doc. Set logging level above\nverbose to disable this message.\",\n\"Payload_MessageType\" : \"GetPatientDemographics\",\n\"Payload_MedseekPatientId\" : \"\",\n\"payload_exception \" : \"this is the exception while connecting the\nserver\nserver abc ip of em server\nabcd\nefgh\nexception not resolved\"\n\"Payload_OriginatingMessageId\" : \"15d7de04-ce8e-4g4g5-ac29-5rr54r5r2r\",\n\"EOE\" : \"\"\n}",
      "@version" => "1",
          "tags" => [
        [0] "multiline"
    ],
          "host" => "lnxolofon"
}
Pipeline main has been shutdown
stopping pipeline {:id=>"main"}

```

---

<div class="post-metadata">

**Author:** ![gopisa](https://avatars.discourse-cdn.com/v4/letter/g/7993a0/32.png) [@gopisa](https://discuss.elastic.co/u/gopisa)\
**Post date:** [September 20, 2018, 11:06am UTC](https://discuss.elastic.co/t/logstash-km-filter/148857/15 "2018-09-20T11:06:57Z")

</div>

Hi,

Thank you for helping me on this.  
yes it's working with above configuration, if we are not used any filters getting below output . , in that out put total my log is coming in message field. if i used json filters, multi line field (payload-exception field ) is not working. getting error .

## output

{  
"@timestamp" =\> "2018-09-20T10:35:56.993Z",  
"message" =\> "{\r\n"SourceId" : "99fc846d-cdd0-547e-babb-54c514616e9d",\r\n"EventId" : "1",\r\n"Keywords" : "1",\r\n"Level" : "Verbose",\r\n"Message" : "",\r\n"Opcode" : "Info",\r\n"Task" : "65533 WriteVerbose",\r\n"Version" : "0",\r\n"Timestamp" : "2018-09-  
18T15:19:40.6927540Z",\r\n"Payload\_message" : "Message received by NotificationSvcTwoWayImplementation. Set logging level above verbose to disable this message.",\r\n"Payload\_MessageType" : "GetPatientNotifications",\r\n"Payload\_MedseekPatientId" : "d209854e-c3c1-4999-aade-20819559c5da"  
,\r\n"payload\_exception " : "this is the exception while connecting the server\r\nserver abc ip of em server\r\nabcd\r\nefgh\r\nexception not resolved"\r\n"Payload\_OriginatingMessageId" : "234d3d88-2a9f-4fcb-b983-8dc08dd36bbc",\r\n"EOE" : ""\r\n}\r",  
"@version" =\> "1",  
"tags" =\> [  
[0] "multiline",  
[1] "\_jsonparsefailure"  
],  
"path" =\> "C:\Medseek\Logs\testfilter.log",  
}

## logstash configuration

input {  
file {  
path =\> "C:\Medseek\Logs\testfilter.log"  
start\_position =\> "beginning"  
codec =\> multiline {  
pattern =\> "^{"  
negate =\> true  
what =\> previous

```
    }
}

```

}

filter {  
json {  
source =\> "message"  
remove\_field =\> ["message", "tags"]  
}

}

## output for above logstash conf

{  
"@timestamp" =\> "2018-09-20T10:50:25.723Z",  
"@version" =\> "1",  
"path" =\> "C:\Medseek\Logs\testfilter.log",  
"host" =\> "AZ-AQA-COM-01",  
"SourceId" =\> "444444-cdd0-547e-babb-1212121",  
"EventId" =\> "1",  
"Keywords" =\> "1",  
"Level" =\> "Verbose",  
"Message" =\> "",  
"Opcode" =\> "Info",  
"Task" =\> "65533 WriteVerbose",  
"Version" =\> "0454",  
"Timestamp" =\> "2018-09-18T15:19:40.6927540Z",  
"Payload\_message" =\> "Message received this message.",  
"Payload\_MessageType" =\> "getting",  
"Payload\_MedseekPatientId" =\> "444-454545-4999-aade-45454",  
"Payload\_OriginatingMessageId" =\> "454-2a9f-454545454-b983-454545454",  
"EOE" =\> ""  
}

## expecting output:

{  
"@timestamp" =\> "2018-09-20T10:50:25.723Z",  
"@version" =\> "1",  
"path" =\> "C:\Medseek\Logs\testfilter.log",  
"host" =\> "AZ-AQA-COM-01",  
"SourceId" =\> "444444-cdd0-547e-babb-1212121",  
"EventId" =\> "1",  
"Keywords" =\> "1",  
"Level" =\> "Verbose",  
"Message" =\> "",  
"Opcode" =\> "Info",  
"Task" =\> "65533 WriteVerbose",  
"Version" =\> "0454",  
"Timestamp" =\> "2018-09-18T15:19:40.6927540Z",  
"Payload\_message" =\> "Message received this message.",  
"Payload\_MessageType" =\> "getting",  
"Payload\_MedseekPatientId" =\> "444-454545-4999-aade-45454",  
"Payload\_Exception" : "at System.Xml..6666666.exception exception exception exception exception  
at office.Xml.454544544.exception exception exception exception exception  
at ntegration.555555.exception exception exception exception exception  
"Payload\_OriginatingMessageId" =\> "454-2a9f-454545454-b983-454545454",  
"EOE" =\> ""  
}

multi line field (payload-exception field ) also should come one filed . i tried but no luck.

another issue is if the log is contain three logs files if i ran the logstash, last log is not processing .

Thanks in advance

Gopi

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 20, 2018, 11:27am UTC](https://discuss.elastic.co/t/logstash-km-filter/148857/16 "2018-09-20T11:27:00Z")

</div>

> multi line field (payload-exception field ) also should come one filed . i tried but no luck.

I can't tell off the top of my head what's going on and I don't have time to dig into it further.

> another issue is if the log is contain three logs files if i ran the logstash, last log is not processing .

Set the multiline codec's `auto_flush_interval` option to something reasonably low, like 5.

---

<div class="post-metadata">

**Author:** ![gopisa](https://avatars.discourse-cdn.com/v4/letter/g/7993a0/32.png) [@gopisa](https://discuss.elastic.co/u/gopisa)\
**Post date:** [September 20, 2018, 12:12pm UTC](https://discuss.elastic.co/t/logstash-km-filter/148857/17 "2018-09-20T12:12:36Z")

</div>

No Problem. thanks you very much for helping me on configuring logstash . grate support ..

---

<div class="post-metadata">

**Author:** ![gopisa](https://avatars.discourse-cdn.com/v4/letter/g/7993a0/32.png) [@gopisa](https://discuss.elastic.co/u/gopisa)\
**Post date:** [September 24, 2018, 8:52am UTC](https://discuss.elastic.co/t/logstash-km-filter/148857/18 "2018-09-24T08:52:13Z")

</div>

Hi ,

if you have time can you please help me on this issue .

Thanks in advance

Regards  
gopi

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 22, 2018, 8:52am UTC](https://discuss.elastic.co/t/logstash-km-filter/148857/19 "2018-10-22T08:52:18Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
