Splitting a column in a csv

Elastic Stack Logstash
1

I have a csv file simular to this

Date|ID|Recipient|Subject
1-01-2016|123|test@test.com|A test
2-01-2016|124|test@test.com, test123@test.com|another test

Sorry if this gets confusing, I have used | to show where each column is defined in my csv file But it is delimited with a ,

On my second line, there are two recipients is there anyway to either delete the second recipient before it is indexed or give it its own column

2

Sure, that's possible. You can process the extracted third field with grok, mutate, or any other filter put after the csv filter.

3

Would i be able to count how many times a domain appears in the field ?

4

With a ruby filter you can do pretty much what you want.

5

I've created a ruby scipt thatnow does what I want, Its basically just a bunch of counts, how do i go about adding the variables to the end of each line. So that they are inputed as well

6

To the end of each line? What output plugin(s) are you using? What do your processed events currently look like? Use a stdout { codec => rubydebug } output to see exactly what you've got.

7

I know the add field => doesnt work..

What i want to do is the input into elasticsearch will have the following fields

Date|ID|Recipient|Subject|Internal Count|Home Use Count|Buisness Partners Count|Managed Suppliers| All|Unknown

The last 6 getting there counts from the ruby variables.

Sorry orginally this was being inputted with the csv filter so using ruby is really new to me

{
		'path' => "/home/callum/test/raw.csv",
		@timestamp = > 2017-01-24T10:34:31,
		@version => "1",
		"host" => "ubuntu"
		"message" => "ZIP,2017-01-10 08:53 GMT,48784048,myhost (10.15.193.196),This-is-the-sender@gmail.com,reciver@gmail.com,anotherreciver@yahoo.com,andanother@gamail.com,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,Email Subject\r",
		"tags" => {
		[0] "_rubyexception"
		]
	}

Capture.PNG1164×875 15.9 KB

8

Your Ruby code is broken so you're getting a _rubyexception tag. Why? Look in the Logstash log for clues.

Taking a quick look at the code it seems like you're calling a lot of methods in the event object that don't exist.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.