# Logstash kv filter plugin query

**URL:** <https://discuss.elastic.co/t/logstash-kv-filter-plugin-query/177894>\
**Category:** Logstash\
**Created:** [April 22, 2019, 7:38pm UTC](https://discuss.elastic.co/t/logstash-kv-filter-plugin-query/177894 "2019-04-22T19:38:38Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Neeraj\_Jain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/neeraj_jain/32/41289_2.png) [@Neeraj\_Jain](https://discuss.elastic.co/u/Neeraj_Jain)\
**Post date:** [April 22, 2019, 7:38pm UTC](https://discuss.elastic.co/t/logstash-kv-filter-plugin-query/177894/1 "2019-04-22T19:38:38Z")

</div>

Hi,  
I am trying to use logstash to read a file containing data as below. I want to push this data as key/value to elastic search.

pid: 2601 start: 0 stacksize: 0 breaksize: 25 command: app1 arguments: APP01  
pid: 2393 start: 0 stacksize: 0 breaksize: 21 command: app2 arguments: APP02  
pid: 2500 start: 0 stacksize: 0 breaksize: 27 command: app3 arguments: APP03

So i use kv filter plugin. Below is the filter.  
filter {  
kv {  
value\_split =\> ":"  
}  
}

But i see that due to new-lines, the key/value pair is not formed correctly. Below is the output.

{  
"@timestamp": "2019-04-22T19:36:52.349Z",  
"@version": "1",  
"arguments": [  
"APP01\npid:",  
"APP02\npid:",  
"APP03\n"  
],  
"breaksize": [  
"25",  
"21",  
"27"  
],  
"command": [  
"app1",  
"app2",  
"app3"  
],  
"host": "tb929cscf01",  
"message": "pid: 2601 start: 0 stacksize: 0 breaksize: 25 command: scscf arguments: IMS\_SCSCF0E\npid: 2393 start: 0 stacksize: 0 breaksize: 21 command: scscf arguments: IMS\_SCSCF03\npid: 2500 start: 0 stacksize: 0 breaksize: 27 command: scscf arguments: IMS\_SCSCF05\n",  
"pid": "2601",  
"stacksize": [  
"0",  
"0",  
"0"  
],  
"start": [  
"0",  
"0",  
"0"  
],  
"type": "proc\_heap"  
}

Why is '\n' followed by the first word in next line appended to value. please advice. Thanks

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 22, 2019, 8:01pm UTC](https://discuss.elastic.co/t/logstash-kv-filter-plugin-query/177894/2 "2019-04-22T20:01:23Z")

</div>

What does your input look like?

---

<div class="post-metadata">

**Author:** ![Neeraj\_Jain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/neeraj_jain/32/41289_2.png) [@Neeraj\_Jain](https://discuss.elastic.co/u/Neeraj_Jain)\
**Post date:** [April 23, 2019, 5:16am UTC](https://discuss.elastic.co/t/logstash-kv-filter-plugin-query/177894/3 "2019-04-23T05:16:13Z")

</div>

Hi Badger,  
Your question gave me a hint. my input was exec plugin with command as cat of the file. resulting in such an output. I changed to below, and now it works fine.  
input{  
file  
{  
path =\> "/etc/logstash/temp\_data"  
start\_position =\> "beginning"  
}  
}

However, my original problem statement was to run command every interval and capture the output of the command

input{  
exec{  
command =\> "my\_command"  
interval =\> 10  
type =\> 'proc\_heap'  
}

output of my command looks like the data in my previous question (as below). So i started debugging by adding into file and checking. So please help me how can i achieve the same output with exec command

pid: 2601 start: 0 stacksize: 0 breaksize: 25 command: app1 arguments: APP01  
pid: 2393 start: 0 stacksize: 0 breaksize: 21 command: app2 arguments: APP02  
pid: 2500 start: 0 stacksize: 0 breaksize: 27 command: app3 arguments: APP03

---

<div class="post-metadata">

**Author:** ![Neeraj\_Jain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/neeraj_jain/32/41289_2.png) [@Neeraj\_Jain](https://discuss.elastic.co/u/Neeraj_Jain)\
**Post date:** [April 23, 2019, 10:19am UTC](https://discuss.elastic.co/t/logstash-kv-filter-plugin-query/177894/4 "2019-04-23T10:19:20Z")

</div>

Hi,  
With below logstash conf, i am now able to get the job done, except that the drop filter isn't working (drop lines which do not contain "APP" string). Can you please help me with this

**output of my command:**  
pid: 32199 start: 0 stacksize: 0 breaksize: 16 command: app1 arguments: APP01  
pid: 32258 start: 0 stacksize: 0 breaksize: 0 command: app2 arguments: APP02  
pid: 32697 start: 0 stacksize: 0 breaksize: 86 command: java arguments: java  
pid: 32699 start: 0 stacksize: 0 breaksize: 140 command: java arguments: java

input{  
exec{  
command =\> "/home/app/my\_script"  
interval =\> 10  
type =\> 'proc\_heap'  
}  
}  
filter {  
mutate {  
gsub =\> [  
"message", "\n", " "  
]  
}  
if ([message] !~ "APP") {  
drop { }  
}  
kv {  
value\_split =\> ":"  
}  
}  
output {  
stdout {  
codec =\> json  
}  
}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 23, 2019, 11:48am UTC](https://discuss.elastic.co/t/logstash-kv-filter-plugin-query/177894/5 "2019-04-23T11:48:44Z")

</div>

If your command outputs multiple lines then the exec input will join them together into a single message. A [split](https://www.elastic.co/guide/en/logstash/current/plugins-filters-split.html) filter can be used to separate each line into its own event...

```
split {}
```

---

<div class="post-metadata">

**Author:** ![Neeraj\_Jain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/neeraj_jain/32/41289_2.png) [@Neeraj\_Jain](https://discuss.elastic.co/u/Neeraj_Jain)\
**Post date:** [April 23, 2019, 5:38pm UTC](https://discuss.elastic.co/t/logstash-kv-filter-plugin-query/177894/6 "2019-04-23T17:38:39Z")

</div>

Hi Badger,  
i see some difference when input reads from file and when we use exec command. with file input, each line is read in message and processed through the filter and so my output looks as expected. See below:

> {  
> "pid" =\> "1335",  
> "host" =\> "tb929cscf01",  
> "@version" =\> "1",  
> "path" =\> "/etc/logstash/full\_data",  
> "breaksize" =\> "25",  
> "@timestamp" =\> 2019-04-23T17:30:41.134Z,  
> "start" =\> "0",  
> "message" =\> "pid: 1335 start: 0 stacksize: 0 breaksize: 25 command: scscf arguments: IMS\_SCSCF0F",  
> "arguments" =\> "IMS\_SCSCF0F",  
> "command" =\> "scscf",  
> "stacksize" =\> "0"  
> }  
> {  
> "pid" =\> "1180",  
> "host" =\> "tb929cscf01",  
> "@version" =\> "1",  
> "path" =\> "/etc/logstash/full\_data",  
> "breaksize" =\> "25",  
> "@timestamp" =\> 2019-04-23T17:30:41.101Z,  
> "start" =\> "0",  
> "message" =\> "pid: 1180 start: 0 stacksize: 0 breaksize: 25 command: scscf arguments: IMS\_SCSCF06",  
> "arguments" =\> "IMS\_SCSCF06",  
> "command" =\> "scscf",  
> "stacksize" =\> "0"  
> }

But when i exec my command, which gives similar lines, the message has all the lines with '\n' character as one string. And so my output isn't as expected (see below). Is there a way i can have exec command output also be read line by line?

> {  
> "message" =\> "pid: 32199 start: 0 stacksize: 0 breaksize: 16 command: STMgr arguments: IMS\_STM01\npid: 32258 start: 0 stacksize: 0 breaksize: 0 command: nginxConfUpdato arguments: IMS\_NGINXCONF\_UPDATOR01\n",  
> "start" =\> [  
> [0] "0",  
> [1] "0"  
> ],  
> "host" =\> "tb929cscf01",  
> "pid" =\> "32199",  
> "breaksize" =\> [  
> [0] "16",  
> [1] "0"  
> ],  
> "@timestamp" =\> 2019-04-23T17:37:50.448Z,  
> "arguments" =\> [  
> [0] "IMS\_STM01\npid:",  
> [1] "IMS\_NGINXCONF\_UPDATOR01\n"  
> ],  
> "@version" =\> "1",  
> "command" =\> [  
> [0] "STMgr",  
> [1] "nginxConfUpdato"  
> ],  
> "type" =\> "proc\_heap",  
> "stacksize" =\> [  
> [0] "0",  
> [1] "0"  
> ]  
> }

~Neeraj

---

<div class="post-metadata">

**Author:** ![Neeraj\_Jain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/neeraj_jain/32/41289_2.png) [@Neeraj\_Jain](https://discuss.elastic.co/u/Neeraj_Jain)\
**Post date:** [April 23, 2019, 5:45pm UTC](https://discuss.elastic.co/t/logstash-kv-filter-plugin-query/177894/7 "2019-04-23T17:45:30Z")

</div>

Hola!!  
just when i posted this question i resolved the issue with simple split in the filter plugin at the start before kv plugin. initially i tried split but passing some arguments in it. but seems to work without any arguments.

filter {  
split {  
}  
kv {  
value\_split =\> ":"  
}  
}

Thank you.

~Neeraj

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 21, 2019, 5:45pm UTC](https://discuss.elastic.co/t/logstash-kv-filter-plugin-query/177894/8 "2019-05-21T17:45:35Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.