# Creating separate documents from log

**URL:** <https://discuss.elastic.co/t/creating-separate-documents-from-log/94521>\
**Category:** Logstash\
**Created:** [July 25, 2017, 5:47pm UTC](https://discuss.elastic.co/t/creating-separate-documents-from-log/94521 "2017-07-25T17:47:55Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![jukeboxhero85](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jukeboxhero85/32/20391_2.png) [@jukeboxhero85](https://discuss.elastic.co/u/jukeboxhero85)\
**Post date:** [July 25, 2017, 5:47pm UTC](https://discuss.elastic.co/t/creating-separate-documents-from-log/94521/1 "2017-07-25T17:47:55Z")

</div>

Hi, is it possible to create separate documents from logstash? what I mean is, I have this log that's one single string, I want to get the first part of the string since that's where the identifier is and the rest are patterns and other IDs. This logs are huge and while the the filter I'm using in logstash makes it more readable I want to logstash to create individual documents with a template like format for every .log file, example off of a single log file:

Log in logstash:

MainID-Version | Pattern{1234 version 1234} | Pattern{1234 version 1234}| Pattern{1234 version 1234}| Pattern{1234 version 1234}| Pattern{1234 version 1234}| Pattern{1234 version 1234}| Pattern{1234 version 1234}| Pattern{1234 version 1234}| Pattern{1234 version 1234}| Pattern{1234 version 1234}...

Elastic Index{

doc 1 [MainID, Version, Pattern]  
doc 2 [MainID, Version, Pattern]

}

Is this possible? and if it is, can you provide direction?

Thanks

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 26, 2017, 7:13am UTC](https://discuss.elastic.co/t/creating-separate-documents-from-log/94521/2 "2017-07-26T07:13:52Z")

</div>

The split filter can splice a single event into multiple events based on an array. You could for example

- use a grok filter to extract MainID-Version into a field of its own and the remains of the string to another field (say, `patterns`),
- use a mutate filter and its split option (not to be confused with the split filter) to split the `patterns` field on " | ", and then
- use the split filter on the `patterns` field which should now be an array.

---

<div class="post-metadata">

**Author:** ![jukeboxhero85](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jukeboxhero85/32/20391_2.png) [@jukeboxhero85](https://discuss.elastic.co/u/jukeboxhero85)\
**Post date:** [July 26, 2017, 9:12am UTC](https://discuss.elastic.co/t/creating-separate-documents-from-log/94521/3 "2017-07-26T09:12:03Z")

</div>

Thank you so much! I'll give it a try tomorrow morning, sounds like it's  
going to work though. Thanks again.

---

<div class="post-metadata">

**Author:** ![jukeboxhero85](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jukeboxhero85/32/20391_2.png) [@jukeboxhero85](https://discuss.elastic.co/u/jukeboxhero85)\
**Post date:** [July 26, 2017, 11:03pm UTC](https://discuss.elastic.co/t/creating-separate-documents-from-log/94521/4 "2017-07-26T23:03:43Z")

</div>

Indeed it worked, thank you so much sir, I really appreciated. For future reference if any other ELK fellow comes across something similar here is a sample code (this makes dealing with MSSQL logs so easy and clean, I love ELK):

```
input {
  beats{
    host => "localhost"
    port => 5044
  }
}

filter {
    grok {
        match => { "message" => "(?<complaint_id>[^|]*)(?<patterns>[^.]*)" }
        remove_field => ["message"]
    }
    
    mutate {
        split => { "patterns" => "|"}
    }
    
    split{
        field => "patterns"
    }
}

output {

  stdout {codec => rubydebug }
  
}
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 23, 2017, 11:04pm UTC](https://discuss.elastic.co/t/creating-separate-documents-from-log/94521/5 "2017-08-23T23:04:16Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.