Creating separate documents from log

[jukeboxhero85 (Juke Box Hero)]

Hi, is it possible to create separate documents from logstash? what I mean is, I have this log that's one single string, I want to get the first part of the string since that's where the identifier is and the rest are patterns and other IDs. This logs are huge and while the the filter I'm using in logstash makes it more readable I want to logstash to create individual documents with a template like format for every .log file, example off of a single log file:

Log in logstash:

MainID-Version | Pattern{1234 version 1234} | Pattern{1234 version 1234}| Pattern{1234 version 1234}| Pattern{1234 version 1234}| Pattern{1234 version 1234}| Pattern{1234 version 1234}| Pattern{1234 version 1234}| Pattern{1234 version 1234}| Pattern{1234 version 1234}| Pattern{1234 version 1234}...

Elastic Index{

doc 1 [MainID, Version, Pattern]
doc 2 [MainID, Version, Pattern]

}

Is this possible? and if it is, can you provide direction?

Thanks

[magnusbaeck (Magnus Bäck)]

The split filter can splice a single event into multiple events based on an array. You could for example

• use a grok filter to extract MainID-Version into a field of its own and the remains of the string to another field (say, patterns),
• use a mutate filter and its split option (not to be confused with the split filter) to split the patterns field on " | ", and then
• use the split filter on the patterns field which should now be an array.

[jukeboxhero85 (Juke Box Hero)]

Thank you so much! I'll give it a try tomorrow morning, sounds like it's
going to work though. Thanks again.

[jukeboxhero85 (Juke Box Hero)]

Indeed it worked, thank you so much sir, I really appreciated. For future reference if any other ELK fellow comes across something similar here is a sample code (this makes dealing with MSSQL logs so easy and clean, I love ELK):

input {
  beats{
    host => "localhost"
    port => 5044
  }
}

filter {
    grok {
        match => { "message" => "(?<complaint_id>[^|]*)(?<patterns>[^.]*)" }
        remove_field => ["message"]
    }
    
    mutate {
        split => { "patterns" => "|"}
    }
    
    split{
        field => "patterns"
    }
}

output {

  stdout {codec => rubydebug }
  
}