# Splitting an event into multiple documents

**URL:** <https://discuss.elastic.co/t/splitting-an-event-into-multiple-documents/113851>\
**Category:** Logstash\
**Created:** [January 3, 2018, 1:00am UTC](https://discuss.elastic.co/t/splitting-an-event-into-multiple-documents/113851 "2018-01-03T01:00:08Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![MattHarvey](https://avatars.discourse-cdn.com/v4/letter/m/a4c791/32.png) [@MattHarvey](https://discuss.elastic.co/u/MattHarvey)\
**Post date:** [January 3, 2018, 1:00am UTC](https://discuss.elastic.co/t/splitting-an-event-into-multiple-documents/113851/1 "2018-01-03T01:00:08Z")

</div>

I am having some trouble splitting fields and cloning events in logstash--this is the first time I am doing so.

Here's my logstash config (I've removed the filter split plugin configuration as it is incorrect):

```
input {
	file {
		path => "/data/input"
		start_position => "beginning"
		sincedb_path => "/dev/null"
	}
}
filter {
	grok {
		match => { "message" => "^(?<pin>[A-Za-z0-9]{15})(?<t1flg>[A-Za-z0-9]{1})(?<t1data>[A-Za-z0-9]{15})(?<t2flg>[A-Za-z0-9]{1})(?<t2data>[A-Za-z0-9]{15})(?<t3flg>[A-Za-z0-9]{1})(?<t3data>[A-Za-z0-9]{15})" }
	}
	mutate {
		strip => ["pin", "t1flg", "t2flg", "t3flg", "t1data", "t2data", "t3data"]
	}
 }

output {
    elasticsearch {
      hosts => ["https://elastic_end_point"]
      user => elastic
      password => password
      document_type => document
      index => documents
      http_compression => true
    }
	}
	stdout {
		codec => rubydebug
	}
}

```

Expected data in ES:

```
[{
	pin: "abc",
	t1flg: "A",
	t1data: "t1data for abc"
},
{
	pin: "abc",
	t2flg: "A",
	t2data: "t2data for abc"
},
{
	pin: "abc",
	t3flg: "A",
	t3data: "t3data for abc"
}]

```

thank you!

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 3, 2018, 9:32am UTC](https://discuss.elastic.co/t/splitting-an-event-into-multiple-documents/113851/2 "2018-01-03T09:32:57Z")

</div>

What's in /data/input? Also, you're talking about splitting and cloning but the expected output seems to be an array containing data from multiple input lines, which seems like the exact opposite of cloning and splitting.

---

<div class="post-metadata">

**Author:** ![MattHarvey](https://avatars.discourse-cdn.com/v4/letter/m/a4c791/32.png) [@MattHarvey](https://discuss.elastic.co/u/MattHarvey)\
**Post date:** [January 3, 2018, 2:22pm UTC](https://discuss.elastic.co/t/splitting-an-event-into-multiple-documents/113851/3 "2018-01-03T14:22:31Z")

</div>

Magnus,

the /dev/input is a file with a single line:

```
abc At1data for abc At2data for abc At3data for abc 

```

Given this, I am trying to have LS create 3 documents to output to ES.

thank you!

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 3, 2018, 7:04pm UTC](https://discuss.elastic.co/t/splitting-an-event-into-multiple-documents/113851/4 "2018-01-03T19:04:12Z")

</div>

Okay, then the previous "expected data in ES" statement was a bit misleading but it actually describes the structure you need for the field you want to split. You can use a ruby filter to construct it.

```nohighlight
event.set(
  'field-to-split', 
  [
    {'pin' => pin, 't1flg' => t1flg, 't1data' => t1data},
    {'pin' => pin, 't2flg' => t2flg, 't2data' => t2data},
    {'pin' => pin, 't3flg' => t3flg, 't3data' => t3data}
  ]
)

```

Then let the split filter work on the `field-to-split` field.

Are you sure you want to have different field names in each document (`t1flg` and `t2flg` etc)?

---

<div class="post-metadata">

**Author:** ![MattHarvey](https://avatars.discourse-cdn.com/v4/letter/m/a4c791/32.png) [@MattHarvey](https://discuss.elastic.co/u/MattHarvey)\
**Post date:** [January 3, 2018, 7:29pm UTC](https://discuss.elastic.co/t/splitting-an-event-into-multiple-documents/113851/5 "2018-01-03T19:29:17Z")

</div>

Thank you, Magnus. You're right, `t1flg`, `t2flg` etc field names in each doc are incorrect and was a typo as I was mocking up the documents. I would instead need only a generic field for flag `tflg` and another for data as `tdata` with an indicator `tind` to represent what the fields represent (t1, t2 or t3).

You've shown me the direction. I will follow on this and revert if I need any additional help.

thank you!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 31, 2018, 7:29pm UTC](https://discuss.elastic.co/t/splitting-an-event-into-multiple-documents/113851/6 "2018-01-31T19:29:20Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.