Csv Column -> Splitstring array and -> Nested Object Problem

Elastic Stack Logstash
1

Dear Community,

iam very new in the logstash and Ruby Environment and i have task that i have to solve.

Input is a CSV file and on Column has a String with the following content (content length is dynamic but structure (blocks size) is the same)

RCK-1-100-1-200-1-110| TML-1-100-1-300-1-100| APC-1-100-1-200-1-10
Modul-statusnr_IN-Statusmsgnr_IN-statusnr_WS-Statusmsgnr_WS-statusnr_TN-Statusmsgnr_TN | next block (amount of blocks can be dynamic) //explaination Line

So what is my Goal

i have a csv that will be read in logstash one coloumn of 12 i not want to upload the specific column to elastic search because it needs the following editing

the column lets call it "Modulstates" should be transformed into a nested field

for the nested field the input (string from column) shoud be string split | seperates the nested block and - seperates the attribute element.

Want this string RCK-1-100-1-200-1-110| TML-1-100-1-300-1-100| APC-1-100-1-200-1-10

into a nested input should look like this for each csv row

"Modulstates": / for eachcsv row
{
	"Modulname": "RCK" // first value from first block // for each block string split "|"
		{
	"Statusnr_IN": "1", // second value from first block // for each subinfo from split array "-"
	"Statusmsgnr_IN": "100", // third value from first block
	"Statusnr_WS": "1", // fourth value from first block
	"Statusmsgnr_WS": "200", // fith value from first block
	"Statusnr_TN": "1", // sixth value from first block
	"Statusmsgnr_TN": "110", // seventh value from first block
	} // the amount on blocks is dynamic there can be a second third block ... but contentstructure is fix
}

summarized problem: make 11/12 colums ready for elastic search (done),create a nested field like see above- This nested field should be filled from the column with the string array
1 entry for 1 row but 1 entry has multiple Blocks
1 entry value row 1 column 12 split array in multiple blocks Modulstatus->Block1(status,Numner), Block2

2 entry..(second row)

sry for my english and i hope you understand my problem

greetings Bender

2 
Dear Community,

we make a big step forward. We have push the data to elastic like below. But how can we define the type "nested" for elastig in the logstash config so that upcoming querys can work correctly?

ruby { code=> "
    	if event.get('STATUS_PROFIL') == nil
    		event.set('Modulstatus', nil)
    	else
    		event.set('Modulstatus', event.get('STATUS_PROFIL').split('|').collect { |t|
    		c=t.split ','
    				{
    				'Modul' => c[0],
    				'Statusnr_IM' => c[1],
    				'Statusmsgnr_IM' => c[2],
    				'Statusnr_IN' => c[3],
    				'Statusmsgnr_IN' => c[4],
    				'Statusnr_EX' => c[5],
    				'Statusmsgnr_EX' => c[6]
    			}
    		})
    	end
    		"
    		}
3

how can i define this as type "nested" in logstash but for elastic/kibana"

4

Hi Bender,

Not sure, which are the field you wish to be nested. But you could use a mutate filter and then use a config similar to this to achieve the result:

mutate {
  rename => {
    "deviceip" => "[IP][device]"
    "srcip" => "[IP][source]"
    "dstip" => "[IP][destination]"
  }
}

This will nest the device, source, and destination fields within the IP field.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.