Splitting an event into multiple documents


(Matt Harvey) #1

I am having some trouble splitting fields and cloning events in logstash--this is the first time I am doing so.

Here's my logstash config (I've removed the filter split plugin configuration as it is incorrect):

input {
	file {
		path => "/data/input"
		start_position => "beginning"
		sincedb_path => "/dev/null"
	}
}
filter {
	grok {
		match => { "message" => "^(?<pin>[A-Za-z0-9 ]{15})(?<t1flg>[A-Za-z0-9 ]{1})(?<t1data>[A-Za-z0-9 ]{15})(?<t2flg>[A-Za-z0-9 ]{1})(?<t2data>[A-Za-z0-9 ]{15})(?<t3flg>[A-Za-z0-9 ]{1})(?<t3data>[A-Za-z0-9 ]{15})" }
	}
	mutate {
		strip   => ["pin", "t1flg", "t2flg", "t3flg", "t1data", "t2data", "t3data"]
	}
 }

output {
    elasticsearch {
      hosts => ["https://elastic_end_point"]
      user => elastic
      password => password
      document_type => document
      index => documents
      http_compression => true
    }
	}
	stdout {
		codec => rubydebug
	}
}

Expected data in ES:

[{
	pin: "abc",
	t1flg: "A",
	t1data: "t1data for abc"
},
{
	pin: "abc",
	t2flg: "A",
	t2data: "t2data for abc"
},
{
	pin: "abc",
	t3flg: "A",
	t3data: "t3data for abc"
}]

thank you!


(Magnus Bäck) #2

What's in /data/input? Also, you're talking about splitting and cloning but the expected output seems to be an array containing data from multiple input lines, which seems like the exact opposite of cloning and splitting.


(Matt Harvey) #3

Magnus,

the /dev/input is a file with a single line:

abc            At1data for abc At2data for abc At3data for abc 

Given this, I am trying to have LS create 3 documents to output to ES.

thank you!


(Magnus Bäck) #4

Okay, then the previous "expected data in ES" statement was a bit misleading but it actually describes the structure you need for the field you want to split. You can use a ruby filter to construct it.

event.set(
  'field-to-split', 
  [
    {'pin' => pin, 't1flg' => t1flg, 't1data' => t1data},
    {'pin' => pin, 't2flg' => t2flg, 't2data' => t2data},
    {'pin' => pin, 't3flg' => t3flg, 't3data' => t3data}
  ]
)

Then let the split filter work on the field-to-split field.

Are you sure you want to have different field names in each document (t1flg and t2flg etc)?


(Matt Harvey) #5

Thank you, Magnus. You're right, t1flg, t2flg etc field names in each doc are incorrect and was a typo as I was mocking up the documents. I would instead need only a generic field for flag tflg and another for data as tdata with an indicator tind to represent what the fields represent (t1, t2 or t3).

You've shown me the direction. I will follow on this and revert if I need any additional help.

thank you!


(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.