# Use of Split Filter for more than 1 fields, it is possible **URL:** \ **Category:** Logstash\ **Created:** [August 20, 2017, 1:26pm UTC](https://discuss.elastic.co/t/use-of-split-filter-for-more-than-1-fields-it-is-possible/97687 "2017-08-20T13:26:17Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Post date:** [August 20, 2017, 1:26pm UTC](https://discuss.elastic.co/t/use-of-split-filter-for-more-than-1-fields-it-is-possible/97687/1 "2017-08-20T13:26:18Z")
Hi All my output is like this ``` Field1 : A,B,C,D Field2: E,F,G,H Field3: W,X,Y,Z Field4: Q,R,S,T ``` Now i am using split filter in conf file ``` split {field => "[Field1]"} split {field => "[Field2]"} split {field => "[Field3]"} split {field => "[Field4]"} ``` My Expected result is Field1 Field2 Field3 Field4 A E W Q B F X R C G Y S D H Z T The conf file keeps running forever and doesn't create index ---
**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Post date:** [August 20, 2017, 4:09pm UTC](https://discuss.elastic.co/t/use-of-split-filter-for-more-than-1-fields-it-is-possible/97687/2 "2017-08-20T16:09:29Z")
anyone any idea on this? @wiibaa, any idea on this? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 20, 2017, 6:27pm UTC](https://discuss.elastic.co/t/use-of-split-filter-for-more-than-1-fields-it-is-possible/97687/3 "2017-08-20T18:27:54Z")
The split filter acts on a single field. You can use a ruby filter to join the elements of each array into another array so that you have this: ```nohighlight { ... "field_to_split": [ ["A", "E", "W", "Q"], ... ] } ``` Then feed `field_to_split` to a split filter, and use another filter or two to split the array back into discrete fields. ---
**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Post date:** [August 21, 2017, 9:00am UTC](https://discuss.elastic.co/t/use-of-split-filter-for-more-than-1-fields-it-is-possible/97687/4 "2017-08-21T09:00:08Z")
Hi Instead of ruby i used this ``` mutate { add_field => { "joiner" => "%{Field1},%{Field2},%{Field3},%{Field4}" } } split {field => "[joiner]"} ``` This is not working out, it is splitting the joiner fields but not other fields ---
**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Post date:** [August 22, 2017, 6:28pm UTC](https://discuss.elastic.co/t/use-of-split-filter-for-more-than-1-fields-it-is-possible/97687/5 "2017-08-22T18:28:17Z")
I think, this is general issue with JSON and XML dataset. How we can split the fields multiple times? Does nobody face this bigger issue ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 24, 2017, 11:45am UTC](https://discuss.elastic.co/t/use-of-split-filter-for-more-than-1-fields-it-is-possible/97687/6 "2017-08-24T11:45:21Z")
Did you try my suggestion? ---
**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Post date:** [August 24, 2017, 12:38pm UTC](https://discuss.elastic.co/t/use-of-split-filter-for-more-than-1-fields-it-is-possible/97687/7 "2017-08-24T12:38:23Z")
Hi I am not getting as array, as i described above, i am getting value as comma separated. I need to split this values in different events. This is working for only one column, i need to do for all the columns which split filter plugin doesn't support ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 24, 2017, 2:32pm UTC](https://discuss.elastic.co/t/use-of-split-filter-for-more-than-1-fields-it-is-possible/97687/8 "2017-08-24T14:32:44Z")
I asked if you tried my suggestion to "use a ruby filter to join the elements of each array into another array". What you just wrote does not answer that question. ---
**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Post date:** [August 24, 2017, 3:00pm UTC](https://discuss.elastic.co/t/use-of-split-filter-for-more-than-1-fields-it-is-possible/97687/9 "2017-08-24T15:00:10Z")
Hi Sorry for replying in different way. I wanted to communicate that i am not getting results in array after filtering through xml filter using XPATH. All i get is comma separated values I am not an expert of ruby filter. I know simple get and set method only to do calculation. But yes i have tried that you recommended me. But not with ruby code, i used mutate to join the values. I updated on this above My conf file, is like this input to get the data from API using http\_poller ``` input { http_poller } filter{ xml { source => "message" target => "message_parsed" add_tag => ["xml_parsed"] remove_namespaces => true store_xml => true force_array => true xpath => [ "/Envelope/Body/advancedSearchResponse/response/table/row/number/text()","number", "/Envelope/Body/advancedSearchResponse/response/table/row/type/selection/value/text()","type", "/Envelope/Body/advancedSearchResponse/response/table/row/description/text()","description", "/Envelope/Body/advancedSearchResponse/response/table/row/phase/selection/value/text()","phase" ] } } output{ elasticsearch } ``` Please let me know if i can try something else. Thanks ---
**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Post date:** [August 25, 2017, 8:20am UTC](https://discuss.elastic.co/t/use-of-split-filter-for-more-than-1-fields-it-is-possible/97687/10 "2017-08-25T08:20:25Z")
Hi I have tried this, but not much result. Here i am trying to convert string to array Then i will join all the arrays then split using split filter. Not sure if this is approach that you referred to Help me to split all the columns ``` ruby { code => " event.set('test', event.get('number').split(',')) " } ``` ---
**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Post date:** [August 26, 2017, 11:06am UTC](https://discuss.elastic.co/t/use-of-split-filter-for-more-than-1-fields-it-is-possible/97687/11 "2017-08-26T11:06:36Z")
Hi Do i am in right path?, please suggest as i am trying this from past couple of days and i am lacking in ruby ---
**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Post date:** [August 28, 2017, 6:03pm UTC](https://discuss.elastic.co/t/use-of-split-filter-for-more-than-1-fields-it-is-possible/97687/12 "2017-08-28T18:03:47Z")
Hi @magnusbaeck I have followed your suggestion I am able to achieve transpose of the array and able to get the record in this below format. But now i am not able to split the records in multiple events. After applying split filter i am getting one records which is the last row of record. Please help me to get multiple records ``` { "id" : "1", "author" : "author1", "desc" : "Description for 1", "date" : "Jun 18, 2017 1:48:43 PM" }, { "id" : "2", "author" : "author2", "desc" : "Description for 2", "date" : "Jun 21, 2017 1:48:43 PM" } ``` I have followed the concept given over here > [@Xml filter array](https://discuss.elastic.co/t/xml-filter-array/96055/9): > > I'm getting there step by step. ruby { code =\> " event.set('mksrevision', [event.get('ID'), event.get('author'), event.get('desc'), event.get('date')].transpose) array\_of\_hashes = event.get('mksrevision').collect { |i| {'id' =\> i[0], 'author' =\> i[1], 'desc' =\> i[2], 'date' =\> i[3]}} event.set('mks', array\_of\_hashes) "} this creates an array "mks" wich basically has the richt structure I need to parse each array elemnt as a seperate document to elasticsearch. edit: got it after … ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 29, 2017, 5:18am UTC](https://discuss.elastic.co/t/use-of-split-filter-for-more-than-1-fields-it-is-possible/97687/13 "2017-08-29T05:18:46Z")
> I am able to achieve transpose of the array and able to get the record in this below format. Okay, good. > But now i am not able to split the records in multiple events. After applying split filter i am getting one records which is the last row of record. That doesn't make sense, or I'm misunderstanding you. Show an example event without the split filter so that I can attempt to reproduce the problem. ---
**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Post date:** [August 29, 2017, 6:36am UTC](https://discuss.elastic.co/t/use-of-split-filter-for-more-than-1-fields-it-is-possible/97687/14 "2017-08-29T06:36:31Z")
Hi @magnusbaeck here is the details. I have 3 records : Delta1, Delta2, Delta3 I need to split the "mks" field into 3 records. after using this, below split stub i get only 1 record for DELTA3, i need all the three records split { field =\> "[mks]" } ``` { "_index": "test123", "_type": "logs", "_id": "DELTA1,DELTA2,DELTA3", "_score": 1, "_source": { "@timestamp": "2017-08-29T06:27:19.208Z", "number": [ "DELTA1", "DELTA2", "DELTA3" ], "mksrevision": [ [ "DELTA1", "Description for DELTA1", "asy", "try" ], [ "DELTA2", "Description for DELTA2", "BW", "try" ], [ "DELTA3", "Description for DELTA3", "MO", "try" ] ], "mks": [ { "numbernew": "DELTA1", "phasenew": "try", "descnew": "Description for DELTA1", "typenew": "asy" }, { "numbernew": "DELTA2", "phasenew": "try", "descnew": "Description for DELTA2", "typenew": "BW" }, { "numbernew": "DELTA3", "phasenew": "try", "descnew": "Description for DELTA3", "typenew": "MO" } ], "type": [ "asy", "BW", "MO" ], "phase": [ "try", "try", "try" ], "@version": "1", "message": ".......................................>", "desc": [ "Description for DELTA1", "Description for DELTA2", "Description for DELTA3" ], "tags": [ "xml_parsed" ] }, "fields": { "@timestamp": [ 1503988039208 ] } } ``` Basically i need this 4 columns with 3 records: numbernew,phasenew,descnew,typenew from the field "mks" ---
**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Post date:** [August 29, 2017, 7:23am UTC](https://discuss.elastic.co/t/use-of-split-filter-for-more-than-1-fields-it-is-possible/97687/15 "2017-08-29T07:23:39Z")
Its working for me i used this in output. We can close this now. Thanks for help `document_id => "%{[mks]}"` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 26, 2017, 7:23am UTC](https://discuss.elastic.co/t/use-of-split-filter-for-more-than-1-fields-it-is-possible/97687/16 "2017-09-26T07:23:42Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.