Use of Split Filter for more than 1 fields, it is possible

[rkhapre (RK)]

Hi All

my output is like this

Field1 : A,B,C,D
Field2: E,F,G,H
Field3: W,X,Y,Z
Field4: Q,R,S,T

Now i am using split filter in conf file

	 split {field => "[Field1 ]"}
	 split {field => "[Field2]"}
	 split {field => "[Field3]"}
	 split {field => "[Field4]"}

My Expected result is

Field1 Field2 Field3 Field4
A E W Q
B F X R
C G Y S
D H Z T

The conf file keeps running forever and doesn't create index

[rkhapre (RK)]

anyone any idea on this?

@wiibaa, any idea on this?

[magnusbaeck (Magnus Bäck)]

The split filter acts on a single field. You can use a ruby filter to join the elements of each array into another array so that you have this:

{
  ...
  "field_to_split": [
    ["A", "E", "W", "Q"],
    ...
  ]
}

Then feed field_to_split to a split filter, and use another filter or two to split the array back into discrete fields.

[rkhapre (RK)]

Hi

Instead of ruby i used this

mutate {
    add_field => { "joiner" => "%{Field1},%{Field2},%{Field3},%{Field4}" }
  }
  
 
		 split {field => "[joiner]"}

This is not working out, it is splitting the joiner fields but not other fields

[rkhapre (RK)]

I think, this is general issue with JSON and XML dataset.
How we can split the fields multiple times?

Does nobody face this bigger issue

[magnusbaeck (Magnus Bäck)]

Did you try my suggestion?

[rkhapre (RK)]

Hi

I am not getting as array, as i described above, i am getting value as comma separated.

I need to split this values in different events.

This is working for only one column, i need to do for all the columns which split filter plugin doesn't support

[magnusbaeck (Magnus Bäck)]

I asked if you tried my suggestion to "use a ruby filter to join the elements of each array into another array". What you just wrote does not answer that question.

[rkhapre (RK)]

Hi

Sorry for replying in different way. I wanted to communicate that i am not getting results in array after filtering through xml filter using XPATH. All i get is comma separated values

I am not an expert of ruby filter. I know simple get and set method only to do calculation.
But yes i have tried that you recommended me. But not with ruby code, i used mutate to join the values. I updated on this above

My conf file, is like this

input to get the data from API using http_poller

input {
http_poller
}

filter{
xml {
          source => "message"
          target => "message_parsed"
          add_tag => ["xml_parsed"]
          remove_namespaces => true
		  store_xml => true
		  force_array => true
          xpath => [
            "/Envelope/Body/advancedSearchResponse/response/table/row/number/text()","number",
			"/Envelope/Body/advancedSearchResponse/response/table/row/type/selection/value/text()","type",
			"/Envelope/Body/advancedSearchResponse/response/table/row/description/text()","description",
			"/Envelope/Body/advancedSearchResponse/response/table/row/phase/selection/value/text()","phase"
			
            ]
     }
}

output{

elasticsearch
}

Please let me know if i can try something else.

Thanks

[rkhapre (RK)]

Hi

I have tried this, but not much result. Here i am trying to convert string to array
Then i will join all the arrays then split using split filter.

Not sure if this is approach that you referred to

Help me to split all the columns

ruby {  code => "
        
		event.set('test', event.get('number').split(','))
	
		
		"
    }

[rkhapre (RK)]

Hi

Do i am in right path?, please suggest as i am trying this from past couple of days and i am lacking in ruby

[rkhapre (RK)]

Hi @magnusbaeck

I have followed your suggestion

I am able to achieve transpose of the array and able to get the record in this below format.
But now i am not able to split the records in multiple events. After applying split filter i am getting one records which is the last row of record.

Please help me to get multiple records

{
	"id" : "1",
	"author" : "author1",
	"desc" : "Description for 1",
	"date" : "Jun 18, 2017 1:48:43 PM"
}, {
	"id" : "2",
	"author" : "author2",
	"desc" : "Description for 2",
	"date" : "Jun 21, 2017 1:48:43 PM"
} 

I have followed the concept given over here

Xml filter array

I'm getting there step by step. ruby { code => " event.set('mksrevision', [event.get('ID'), event.get('author'), event.get('desc'), event.get('date')].transpose) array_of_hashes = event.get('mksrevision').collect { |i| {'id' => i[0], 'author' => i[1], 'desc' => i[2], 'date' => i[3]}} event.set('mks', array_of_hashes) "} this creates an array "mks" wich basically has the richt structure I need to parse each array elemnt as a seperate document to elasticsearch. edit: got it after …

[magnusbaeck (Magnus Bäck)]

I am able to achieve transpose of the array and able to get the record in this below format.

Okay, good.

But now i am not able to split the records in multiple events. After applying split filter i am getting one records which is the last row of record.

That doesn't make sense, or I'm misunderstanding you. Show an example event without the split filter so that I can attempt to reproduce the problem.

[rkhapre (RK)]

Hi @magnusbaeck

here is the details. I have 3 records : Delta1, Delta2, Delta3
I need to split the "mks" field into 3 records.

after using this, below split stub i get only 1 record for DELTA3, i need all the three records
split {
field => "[mks]"
}

{
  "_index": "test123",
  "_type": "logs",
  "_id": "DELTA1,DELTA2,DELTA3",
  "_score": 1,
  "_source": {
    "@timestamp": "2017-08-29T06:27:19.208Z",
    "number": [
      "DELTA1",
      "DELTA2",
      "DELTA3"
    ],
    "mksrevision": [
      [
        "DELTA1",
        "Description for DELTA1",
        "asy",
        "try"
      ],
      [
        "DELTA2",
        "Description for DELTA2",
        "BW",
        "try"
      ],
      [
        "DELTA3",
        "Description for DELTA3",
        "MO",
        "try"
      ]
    ],
    "mks": [
      {
        "numbernew": "DELTA1",
        "phasenew": "try",
        "descnew": "Description for DELTA1",
        "typenew": "asy"
      },
      {
        "numbernew": "DELTA2",
        "phasenew": "try",
        "descnew": "Description for DELTA2",
        "typenew": "BW"
      },
      {
        "numbernew": "DELTA3",
        "phasenew": "try",
        "descnew": "Description for DELTA3",
        "typenew": "MO"
      }
    ],
    "type": [
      "asy",
      "BW",
      "MO"
    ],
    "phase": [
      "try",
      "try",
      "try"
    ],
    "@version": "1",
    "message": "<?xml version='1.0' encoding='UTF-8'?>.......................................></S:Body></S:Envelope>",
    "desc": [
      "Description for DELTA1",
      "Description for DELTA2",
      "Description for DELTA3"
    ],
    "tags": [
      "xml_parsed"
    ]
  },
  "fields": {
    "@timestamp": [
      1503988039208
    ]
  }
}

Basically i need this 4 columns with 3 records: numbernew,phasenew,descnew,typenew
from the field "mks"

[rkhapre (RK)]

Its working for me

i used this in output. We can close this now.

Thanks for help

document_id => "%{[mks]}"