Splitting an event into multiple documents

Elastic Stack Logstash
1

I am having some trouble splitting fields and cloning events in logstash--this is the first time I am doing so.

Here's my logstash config (I've removed the filter split plugin configuration as it is incorrect):

input {
	file {
		path => "/data/input"
		start_position => "beginning"
		sincedb_path => "/dev/null"
	}
}
filter {
	grok {
		match => { "message" => "^(?<pin>[A-Za-z0-9 ]{15})(?<t1flg>[A-Za-z0-9 ]{1})(?<t1data>[A-Za-z0-9 ]{15})(?<t2flg>[A-Za-z0-9 ]{1})(?<t2data>[A-Za-z0-9 ]{15})(?<t3flg>[A-Za-z0-9 ]{1})(?<t3data>[A-Za-z0-9 ]{15})" }
	}
	mutate {
		strip   => ["pin", "t1flg", "t2flg", "t3flg", "t1data", "t2data", "t3data"]
	}
 }

output {
    elasticsearch {
      hosts => ["https://elastic_end_point"]
      user => elastic
      password => password
      document_type => document
      index => documents
      http_compression => true
    }
	}
	stdout {
		codec => rubydebug
	}
}

Expected data in ES:

[{
	pin: "abc",
	t1flg: "A",
	t1data: "t1data for abc"
},
{
	pin: "abc",
	t2flg: "A",
	t2data: "t2data for abc"
},
{
	pin: "abc",
	t3flg: "A",
	t3data: "t3data for abc"
}]

thank you!

2

What's in /data/input? Also, you're talking about splitting and cloning but the expected output seems to be an array containing data from multiple input lines, which seems like the exact opposite of cloning and splitting.

3

Magnus,

the /dev/input is a file with a single line:

abc            At1data for abc At2data for abc At3data for abc

Given this, I am trying to have LS create 3 documents to output to ES.

thank you!

4

Okay, then the previous "expected data in ES" statement was a bit misleading but it actually describes the structure you need for the field you want to split. You can use a ruby filter to construct it.

event.set(
  'field-to-split', 
  [
    {'pin' => pin, 't1flg' => t1flg, 't1data' => t1data},
    {'pin' => pin, 't2flg' => t2flg, 't2data' => t2data},
    {'pin' => pin, 't3flg' => t3flg, 't3data' => t3data}
  ]
)

Then let the split filter work on the field-to-split field.

Are you sure you want to have different field names in each document (t1flg and t2flg etc)?

1 Like
5

Thank you, Magnus. You're right, t1flg, t2flg etc field names in each doc are incorrect and was a typo as I was mocking up the documents. I would instead need only a generic field for flag tflg and another for data as tdata with an indicator tind to represent what the fields represent (t1, t2 or t3).

You've shown me the direction. I will follow on this and revert if I need any additional help.

thank you!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.