# How to pass array field to kv-filter
**URL:** https://discuss.elastic.co/t/how-to-pass-array-field-to-kv-filter/34496
**Category:** Logstash
**Created:** [November 13, 2015, 9:19am UTC](https://discuss.elastic.co/t/how-to-pass-array-field-to-kv-filter/34496 "2015-11-13T09:19:46Z")
**Posts on this page:** 16
**Page:** 1
### Author:  [@mne](https://discuss.elastic.co/u/mne)
#### Post date: [November 13, 2015, 9:19am UTC](https://discuss.elastic.co/t/how-to-pass-array-field-to-kv-filter/34496/1 "2015-11-13T09:19:46Z")
Hi all,
i'm looking for a way within logstash to pass an array to the kv-filter and use it as parameter "include\_fields".
For example in my json event i have an array named "keys":
"keys" =\> [
[0] "key1",
[1] "key2",
[2] "key3",
...
]
What I've tried so far is
kv {
include\_keys =\> ["%{keys}"]
}
But this doesn't work, only if i use
kv {
include\_keys =\> [%{[keys][0]}", "%{[keys][1]}" ]
}
it works, but i want the to be dynamic so whatever is in [keys] should be used in kv-filter as include\_keys. Any ideas?
Regards,
Marcus
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [November 14, 2015, 3:42pm UTC](https://discuss.elastic.co/t/how-to-pass-array-field-to-kv-filter/34496/2 "2015-11-14T15:42:18Z")
I don't think there's a way of doing what you want with `include_keys`. How about capturing all fields at first and removing them afterwards? The [prune filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-prune.html) would be ideal for that, except that it, too, doesn't support arrays (but that's fixable—feel free to file an issue). However, a [ruby filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html) would do the job. I'm pretty sure examples have been posted here in the past, and definitely on StackOverflow (see e.g. [http://stackoverflow.com/a/30343349/414355](http://stackoverflow.com/a/30343349/414355)).
---
### Author:  [@mne](https://discuss.elastic.co/u/mne)
#### Post date: [November 16, 2015, 6:17am UTC](https://discuss.elastic.co/t/how-to-pass-array-field-to-kv-filter/34496/3 "2015-11-16T06:17:15Z")
Hi Magnus,
thanks for your reply, but the problem is, that i don't know about the keys to be included. So other people configure keys which should used as kv-filter's include\_keys.
That's why i'm looking for a way to configure kv-filter's include\_keys in a dynamic way with values which are send to the logstash pipeline together with event data.
Regards,
Marcus
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [November 16, 2015, 6:38am UTC](https://discuss.elastic.co/t/how-to-pass-array-field-to-kv-filter/34496/4 "2015-11-16T06:38:53Z")
The StackOverflow link I posted shows how to delete all fields except those in a particular set, which would allow you to extract all fields with a kv filter but delete those you're not interested in. I believe this would solve your problem. If not, please explain why.
---
### Author:  [@mne](https://discuss.elastic.co/u/mne)
#### Post date: [November 16, 2015, 7:00am UTC](https://discuss.elastic.co/t/how-to-pass-array-field-to-kv-filter/34496/5 "2015-11-16T07:00:02Z")
Sorry i'll try to explain:
As i unterstand your ruby example (i'm not familiar) it shall remove all unwanted fields from an event and than you suggest do a kv-filter on the rest. But i don't want to remove any data from an event.
If an event (logline) is send to the logstash pipeline which includes an additional field "keys" the pipeline should use this field as input for kv-filter.
The contents of the field "keys" should be used to find a key-value match in the logline.
So we want to enrich our data with new fields based on key-value informations in the different loglines.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [November 16, 2015, 7:08am UTC](https://discuss.elastic.co/t/how-to-pass-array-field-to-kv-filter/34496/6 "2015-11-16T07:08:07Z")
> As i unterstand your ruby example (i'm not familiar) it shall remove all unwanted fields from an event and than you suggest do a kv-filter on the rest.
No! You run the kv filter to extract all fields from the input string, _then_ use a Ruby filter to remove the fields that you don't want, based on the contents of the `keys` field (delete the extracted fields that are _not_ included in `keys'). In the end you'll get exactly what you want except that it needs to be done with two or three filters instead of one.
---
### Author:  [@mne](https://discuss.elastic.co/u/mne)
#### Post date: [November 16, 2015, 7:33am UTC](https://discuss.elastic.co/t/how-to-pass-array-field-to-kv-filter/34496/7 "2015-11-16T07:33:34Z")
Ok, got it...
wanted\_fields = "%{keys}" or should it be wanted\_fields= event.keys?
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [November 16, 2015, 7:39am UTC](https://discuss.elastic.co/t/how-to-pass-array-field-to-kv-filter/34496/8 "2015-11-16T07:39:51Z")
```
wanted_fields = event['keys']
```
---
### Author:  [@mne](https://discuss.elastic.co/u/mne)
#### Post date: [November 16, 2015, 10:08am UTC](https://discuss.elastic.co/t/how-to-pass-array-field-to-kv-filter/34496/9 "2015-11-16T10:08:53Z")
OK tested it, but as i assumed all other content is removed from the event except the fields which are in keys.
If I would add a tag for each field added by kv-filter, is there a way to reference the tag in ruby?
If yes i could only to remove fields from an event which are added by kv-filter (has tag e.g. kvf) and not in keys...
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [November 16, 2015, 12:12pm UTC](https://discuss.elastic.co/t/how-to-pass-array-field-to-kv-filter/34496/10 "2015-11-16T12:12:40Z")
Fields can't have tags. I suggest you set the kv filter's `target` option to store those keys in a subfield. Then you know that all fields in there come from kv. Once you've purged the fields you don't want, move the rest to the top level. You can do that in the same ruby filter.
---
### Author:  [@mne](https://discuss.elastic.co/u/mne)
#### Post date: [November 17, 2015, 6:14am UTC](https://discuss.elastic.co/t/how-to-pass-array-field-to-kv-filter/34496/11 "2015-11-17T06:14:34Z")
Can you give me an example how to access subfields in ruby?
Unfortunatly i don't get it working...
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [November 17, 2015, 6:43am UTC](https://discuss.elastic.co/t/how-to-pass-array-field-to-kv-filter/34496/12 "2015-11-17T06:43:07Z")
`event['field']['subfield']` doesn't work?
---
### Author:  [@mne](https://discuss.elastic.co/u/mne)
#### Post date: [November 17, 2015, 6:53am UTC](https://discuss.elastic.co/t/how-to-pass-array-field-to-kv-filter/34496/13 "2015-11-17T06:53:32Z")
I've tried it like this (kv filter's target=keyval)
ruby {
code =\> "
wanted\_fields = event['keys']
event['keyval'].to\_hash.keys.each { |k|
event['keyval'].remove(k) unless wanted\_fields.include? k
}
"
But i get an error: Ruby exception occurred: undefined method `remove' for ...
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [November 17, 2015, 7:00am UTC](https://discuss.elastic.co/t/how-to-pass-array-field-to-kv-filter/34496/14 "2015-11-17T07:00:53Z")
Yeah, this is a bit awkward. `event` isn't really a Ruby hash object and the method for deleting fields is called `remove`. However, nested fields are implemented using regular hashes whose corresponding method is `delete`. So `event.remove('foo')` but `event['foo'].delete('bar')`.
---
### Author:  [@mne](https://discuss.elastic.co/u/mne)
#### Post date: [November 17, 2015, 8:24am UTC](https://discuss.elastic.co/t/how-to-pass-array-field-to-kv-filter/34496/15 "2015-11-17T08:24:26Z")
Thank's!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 5:22am UTC](https://discuss.elastic.co/t/how-to-pass-array-field-to-kv-filter/34496/16 "2017-07-06T05:22:37Z")