# Logstash kv plugin is not working **URL:** \ **Category:** Logstash\ **Created:** [March 24, 2022, 9:31am UTC](https://discuss.elastic.co/t/logstash-kv-plugin-is-not-working/300548 "2022-03-24T09:31:09Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![mmk1995](https://avatars.discourse-cdn.com/v4/letter/m/8797f3/32.png) [@mmk1995](https://discuss.elastic.co/u/mmk1995)\ **Post date:** [March 24, 2022, 9:31am UTC](https://discuss.elastic.co/t/logstash-kv-plugin-is-not-working/300548/1 "2022-03-24T09:31:09Z")
I am using ELK 7.6.2 The problem is when I remove the kv plugin, there is log loaded to Elasticsearch, when I add kv plugin, there is zero data can be loaded to Elasticsearch. Moreover, there is no error is logged in the logstash-plain.log I have Googled but no solution is out there. Please help. Thank you. The log sample is `Mar 24 16:51:40 2022 notice 111.2.333.44 111.22.333.44 time=16:51:39 devname="FG0000-NAT-B" devid="FG1K5Drfvv2345" logid="0000000013" type="traffic" subtype="forward" level="notice" vd="nat" eventtime=1648111900042439291 tz="+0800" srcip=11.222.333.144 srcport=523 srcintf="port34" srcintfrole="undefined" dstip=123.13.64.44 dstport=444 dstintf="port34" dstintfrole="undefined" sessionid=1355513311 proto=6 action="close" policyid=3 policytype="policy" poluuid="016d412dr0-72131298-51e7-3af0-d351asf32980" service="HTTPS" dstcountry="Singapore" srccountry="Reserved" trandisp="snat" transip=213.111.149.44 transport=2214 duration=2 sentbyte=4709 rcvdbyte=7217 sentpkt=23 rcvdpkt=17 shapingpolicyid=3 shaperperipname="ts-perip-web-dns" shaperperipdropbyte=0 appcat="unscanned"` In the filter plugin ```auto grok { match => { "message" => "^%{SYSLOGTIMESTAMP:timestamp}\s+%{YEAR:year}\s%{WORD:priority}\s%{IP:sourceIP}\s%{IP:host}\s%{GREEDYDATA:msg}$" } } kv { source => "msg" trim_value => "\"" value_split => "=" field_split_pattern => "\s+" #remove_field => ["msg"] } ``` ---
**Author:** ![mmk1995](https://avatars.discourse-cdn.com/v4/letter/m/8797f3/32.png) [@mmk1995](https://discuss.elastic.co/u/mmk1995)\ **Post date:** [March 25, 2022, 4:11pm UTC](https://discuss.elastic.co/t/logstash-kv-plugin-is-not-working/300548/2 "2022-03-25T16:11:00Z")
push ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [March 25, 2022, 4:26pm UTC](https://discuss.elastic.co/t/logstash-kv-plugin-is-not-working/300548/3 "2022-03-25T16:26:42Z")
If changing the filters results in the event not being indexed there is most likely a mapping exception in the logstash logs due to a type conflict in the Elasticsearch index. Check the logstash logs. ---
**Author:** ![mmk1995](https://avatars.discourse-cdn.com/v4/letter/m/8797f3/32.png) [@mmk1995](https://discuss.elastic.co/u/mmk1995)\ **Post date:** [March 25, 2022, 4:41pm UTC](https://discuss.elastic.co/t/logstash-kv-plugin-is-not-working/300548/4 "2022-03-25T16:41:04Z")
The log didn't have related error, I do not have set any mapping for the index, it is newly created ---
**Author:** ![mmk1995](https://avatars.discourse-cdn.com/v4/letter/m/8797f3/32.png) [@mmk1995](https://discuss.elastic.co/u/mmk1995)\ **Post date:** [April 2, 2022, 4:09pm UTC](https://discuss.elastic.co/t/logstash-kv-plugin-is-not-working/300548/5 "2022-04-02T16:09:52Z")
anyone experiencing this? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [April 2, 2022, 7:52pm UTC](https://discuss.elastic.co/t/logstash-kv-plugin-is-not-working/300548/6 "2022-04-02T19:52:16Z")
What is your output? Is your grok working? I tried to simulate it and your grok is not working, since your `grok` is not working you won't have the field to be used in the `kv` filter. From your sample message you are trying to parse the kv message from Fortigate devices, this is pretty simple. You do not need the extra configuration in your `kv` filter, just the source option is enough to parse this kind of log. And if your messages have always the same structure you also do not need `grok`, you could use a dissect filter. Try the following: ```auto filter { dissect { mapping => { "message" => "%{timestamp} %{+timestamp} %{+timestamp} %{year} %{priority} %{sourceIP} %{host} %{msg}" } remove_field => ["message"] } kv { source => "msg" remove_field => ["msg"] } } ``` The above filter configuration will parse your message and remove both the original `message` field and the sourcer kv `msg` field. Your result will be similar to this one: ```auto { "level" => "notice", "transip" => "213.111.149.44", "proto" => "6", "sentpkt" => "23", "timestamp" => "Mar 24 16:51:40", "type" => "traffic", "policytype" => "policy", "host" => "111.22.333.44", "srcport" => "523", "rcvdpkt" => "17", "@timestamp" => 2022-04-02T19:49:54.277Z, "dstintf" => "port34", "policyid" => "3", "dstport" => "444", "devid" => "FG1K5Drfvv2345", "action" => "close", "year" => "2022", "srcintfrole" => "undefined", "@version" => "1", "subtype" => "forward", "time" => "16:51:39", "logid" => "0000000013", "eventtime" => "1648111900042439291", "devname" => "FG0000-NAT-B", "shaperperipdropbyte" => "0", "shapingpolicyid" => "3", "srccountry" => "Reserved", "vd" => "nat", "appcat" => "unscanned", "priority" => "notice", "transport" => "2214", "duration" => "2", "srcintf" => "port34", "dstip" => "123.13.64.44", "dstcountry" => "Singapore", "poluuid" => "016d412dr0-72131298-51e7-3af0-d351asf32980", "sessionid" => "1355513311", "rcvdbyte" => "7217", "sourceIP" => "111.2.333.44", "trandisp" => "snat", "sentbyte" => "4709", "tz" => "+0800", "shaperperipname" => "ts-perip-web-dns", "dstintfrole" => "undefined", "service" => "HTTPS", "srcip" => "11.222.333.144" } ``` ---
**Author:** ![mmk1995](https://avatars.discourse-cdn.com/v4/letter/m/8797f3/32.png) [@mmk1995](https://discuss.elastic.co/u/mmk1995)\ **Post date:** [April 4, 2022, 9:24am UTC](https://discuss.elastic.co/t/logstash-kv-plugin-is-not-working/300548/7 "2022-04-04T09:24:20Z")
Thank you for your reply, Is there anything wrong in my filter config? Would you point me out? The grok plugin worked for me, btw how do I use kv filter without any configuration to parse the beginning log part? Mar 24 16:51:40 2022 notice 111.2.333.44 111.22.333.44 ---
**Author:** ![mmk1995](https://avatars.discourse-cdn.com/v4/letter/m/8797f3/32.png) [@mmk1995](https://discuss.elastic.co/u/mmk1995)\ **Post date:** [April 4, 2022, 9:25am UTC](https://discuss.elastic.co/t/logstash-kv-plugin-is-not-working/300548/8 "2022-04-04T09:25:10Z")
dissect may not work for me since the log structure is changing due to optional fields ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [April 4, 2022, 2:25pm UTC](https://discuss.elastic.co/t/logstash-kv-plugin-is-not-working/300548/9 "2022-04-04T14:25:46Z")
You are right, your grok is working, just checked it, the issue was that your sample message has a couple of invalid IPs, `111.2.333.44` is not a valid IP, which broke the pipeline when I tried to simulate, so I assumed that the grok was broke. > btw how do I use kv filter without any configuration to parse the beginning log part? > Mar 24 16:51:40 2022 notice 111.2.333.44 111.22.333.44 You don't, the `kv` filter only works for valid key-value messages, so since your message has a string part and a key-value part, you need to get the key-value part in a separated field, your grok is already doing this putting the key-value part in the `msg` field and creating the other fields from the string part. After your grok, you have the `msg` field with the following content: ```auto time=16:51:39 devname="FG0000-NAT-B" devid="FG1K5Drfvv2345" logid="0000000013" type="traffic" subtype="forward" level="notice" vd="nat" eventtime=1648111900042439291 tz="+0800" srcip=11.222.333.144 srcport=523 srcintf="port34" srcintfrole="undefined" dstip=123.13.64.44 dstport=444 dstintf="port34" dstintfrole="undefined" sessionid=1355513311 proto=6 action="close" policyid=3 policytype="policy" poluuid="016d412dr0-72131298-51e7-3af0-d351asf32980" service="HTTPS" dstcountry="Singapore" srccountry="Reserved" trandisp="snat" transip=213.111.149.44 transport=2214 duration=2 sentbyte=4709 rcvdbyte=7217 sentpkt=23 rcvdpkt=17 shapingpolicyid=3 shaperperipname="ts-perip-web-dns" shaperperipdropbyte=0 appcat="unscanned" ``` To parse this with `kv` you just need: ```auto kv { source => "msg" } ``` But again, it is not clear what is your issue since you didn't share any output you are getting and any log with errors. ---
**Author:** ![mmk1995](https://avatars.discourse-cdn.com/v4/letter/m/8797f3/32.png) [@mmk1995](https://discuss.elastic.co/u/mmk1995)\ **Post date:** [April 4, 2022, 3:28pm UTC](https://discuss.elastic.co/t/logstash-kv-plugin-is-not-working/300548/10 "2022-04-04T15:28:52Z")
anything I can get for you? there is no error log in logstash-plain.log ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [April 4, 2022, 3:36pm UTC](https://discuss.elastic.co/t/logstash-kv-plugin-is-not-working/300548/11 "2022-04-04T15:36:42Z")
You need to share what is the output you are getting, also share your full pipeline. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 2, 2022, 3:37pm UTC](https://discuss.elastic.co/t/logstash-kv-plugin-is-not-working/300548/12 "2022-05-02T15:37:28Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.