Splitting a column in a csv

I have a csv file simular to this

1-01-2016|123|test@test.com|A test
2-01-2016|124|test@test.com, test123@test.com|another test

Sorry if this gets confusing, I have used | to show where each column is defined in my csv file But it is delimited with a ,

On my second line, there are two recipients is there anyway to either delete the second recipient before it is indexed or give it its own column

Sure, that's possible. You can process the extracted third field with grok, mutate, or any other filter put after the csv filter.

Would i be able to count how many times a domain appears in the field ?

With a ruby filter you can do pretty much what you want.

I've created a ruby scipt thatnow does what I want, Its basically just a bunch of counts, how do i go about adding the variables to the end of each line. So that they are inputed as well

To the end of each line? What output plugin(s) are you using? What do your processed events currently look like? Use a stdout { codec => rubydebug } output to see exactly what you've got.

I know the add field => doesnt work..

What i want to do is the input into elasticsearch will have the following fields

Date|ID|Recipient|Subject|Internal Count|Home Use Count|Buisness Partners Count|Managed Suppliers| All|Unknown

The last 6 getting there counts from the ruby variables.

Sorry orginally this was being inputted with the csv filter so using ruby is really new to me

		'path' => "/home/callum/test/raw.csv",
		@timestamp = > 2017-01-24T10:34:31,
		@version => "1",
		"host" => "ubuntu"
		"message" => "ZIP,2017-01-10 08:53 GMT,48784048,myhost (,This-is-the-sender@gmail.com,reciver@gmail.com,anotherreciver@yahoo.com,andanother@gamail.com,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,Email Subject\r",
		"tags" => {
		[0] "_rubyexception"

Your Ruby code is broken so you're getting a _rubyexception tag. Why? Look in the Logstash log for clues.

Taking a quick look at the code it seems like you're calling a lot of methods in the event object that don't exist.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.